Since my use case is more about sharing a network engine (and thread
pool and so on) between asio and libcurl for a few hundred connections
than about scaling to 10K, I think I can get by with no changes to
libcurl and the new null-buffer stuff in boost.asio.

Is anyone interested in further followup on async-izing libcurl, or
should I just drop it?

Meanwhile:

19 Aug 2008 10:35, Jamie Lokier:
> > By the way, I'm not sure if I mentioned this before, but faking
> > asio on top of a poll-type API is trivial and actually pretty
> > efficient, while the other way around (basically what you suggested)
> > is more complicated and measurably slow. If you're really going to
> > force one or the other, forcing asio makes more sense.
>
> I don't agree entirely. It's true that faking epoll-style on top of
> async I/O is slow because of extra copying, and more complicated.
>
> But faking async I/O on top of epoll-style wastes a lot of memory, in
> reserved buffers, and they are in cold cache. Neither method is
> optimal for all scalability issues. For memory use, roughly speaking
> epoll-style is better for reading and async I/O is better for writing,
> but it depends on details.

True. I was implicitly assuming that someone worrying about the C10K
problem was building a server, and that a typical server does a lot
more writing than reading. But that's obviously not universally true.
An IM server does just as much in each direction, and you could be
building a spider or load testing tool that does much more reading
than writing.

So, one little change. Or one of two. Either the caller can provide
both callbacks, in which case curl uses async for writes but
poll/sync for reads, or the caller can still only provide one but
has an option that can be set to specify null-buffer async reads.
(You don't always want sync reads, after all. If you have 10
publishers and 10K subscribers, you want to poll the subscribers'
reads but async the publishers'.)

Implementing this in the app wouldn't be that hard. On boost.asio,
it's trivial (as of 1.36). On Windows IOCP, as long as you don't
need to handle null-buffer writes (which there doesn't seem to be
any need for), it's just as easy. If you're already faking asio on
select/epoll/etc., just don't do the read on null-buffer calls. The
only real issue might be Solaris IOCP, but I'm guessing they can
either do POLLIN and aio_write on the same socket, or handle
null-buffer aio_read calls.
Received on 2008-08-23