On Mon, Aug 18, 2008 at 2:32 PM, Daniel Stenberg <daniel_at_haxx.se> wrote:
> On Sat, 16 Aug 2008, Cory Nelson wrote:
>
>>>> the existing buffer instead of reading or writing, but is otherwise the
>>>> same.
>>>
>>> I don't quite understand this brief description. Can you add some more
>>> psuedo code for a client using this suggested API?
>>
>> The good thing is that curl's public API doesn't need to change one bit to
>> use IOCP - curl_multi_perform() is the equivalent of the d.run() above. How
>> much the internal code would need to change is another question though!
>
> Ok, I think I start to understand things. In order to get really high
> performance on Windows for C10k (10000+ connections), we need this IOCP
> approach.

Yes.

> For an ordinary application with < 100 connections, why would an app
> particularly insist on such an asynch API for libcurl? Also, given the
> portable nature of libcurl, I don't see how we could even dream of providing
> an "asynch API" (unless we do threads of course).

Indeed, if you have < 100 connections it doesn't matter as much.

As I said before, you don't need to worry about threading at all with IOCP.

>>> In my view, asynchronous is mostly just another word for running the
>>> stuff in another thread until it has something, and then have a means of
>>> telling the first thread when it is done. And you can use libcurl fine
>>> already for doing exactly that.
>>
>> Well, the current way of doing it (select) supports that model too -- it's
>> just not anything close to IOCP's efficiency which scales to tens of
>> thousands of concurrent operations.
>
> I wouldn't call libcurl's current way to be select-based. libcurl can be
> select-based indeed, but libcurl has successfully been used with several
> tens of thousands of connections just fine using its multi_socket API that
> is portable and functional on a large amount of operating systems. Including
> windows.

Well, I meant libcurl on Windows. It definitely won't handle
thousands of concurrent connections there.

> What possibly is lacking on Windows, as Jamie Lokier possibly suggests, is
> an event-based system that is good enough or suitable to do the job in a
> convenient manner!

IIRC the primary difference between IOCP and epoll is that IOCP gives
notification when an operation finishes, and epoll gives notification
when an operation is ok to start. So while there is some difference,
it's not really a super inconvenient one.

>> asio does some magic to make a truly async OpenSSL socket (see
>> asio::ssl::stream), but OpenSSL's API, much like almost every other protocol
>> library, is not really designed well for this kind of async.
>
> Right, but SSL (with OpenSSL or the other SSL libs) and SSH based protocols
> are supported by libcurl's high performance API...
>
> So, the notion that we "need an asynch API" is thus simply based on the fact
> that you need to use it on Windows to get high performance when using large
> amounts of connections?

Yes. But again, I don't think the external API of libcurl would have
to change at all. It would all be internal changes.

-- 
Cory Nelson