cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: libcurl and async I/O

From: Cory Nelson <phrosty_at_gmail.com>
Date: Tue, 19 Aug 2008 19:26:31 -0700

On Tue, Aug 19, 2008 at 10:50 AM, Jamie Lokier <jamie_at_shareable.org> wrote:
> Cory Nelson wrote:
>> > And neither method is optimal!
>> >
>> > Ideally, you don't want to commit 10000 large receive buffers in
>> > overlapped reads which may take a long time to receive anything. You
>> > want to allocate buffers from a smaller pool as data becomes
>> > available. For writes, similar but less clear cut.
>>
>> Windows allows you to perform a WSARecv() with a null pointer. It
>> lowers your overall throughput, but it does let you minimize the
>> amount of locked pages which can be useful on 32-bit.
>
> Thank you, that's very interesting!
>
> I've just re-read the WSARecv() documentation :
>
> http://msdn.microsoft.com/en-us/library/ms741688(VS.85).aspx
>
> http://msdn.microsoft.com/en-us/library/ms741542(VS.85).aspx
>
> and I don't see this anywhere. Can you point me at something which
> describes this feature and exactly what it does (and if it's and
> extension specific to certain types of socket provider) - or where I
> missed it in the documentation?

It's been a long time so I can't be certain, but I believe it is
covered in the book "Network Programming for Microsoft Windows". The
idea is to perform a 0-byte async receive to get notified, then
non-blocking receives to pull the data.

1000 concurrent operations = only 4MiB of locked pages, assuming you
use aligned 4KiB buffers. It does affect performance slightly, so
it's really only useful if you'll have many thousands of concurrent
operations and are on 32-bit or have very low memory.

> Does WSASend() support similar for writes - so that you can use these
> operations with null buffers as another way to build an epoll-style
> event collector?

Nope, only receives.

-- 
Cory Nelson
Received on 2008-08-20