>>Are you sure about that? Calling curl_multi_socket_action() repeatedly is
not meant to have any other effect (assuming nothing changed) so if if you
need to do it and if it helps, I'd say we have a bug to fix.

I did a bit more testing today, and my confidence in the diagnosis has gone
up. However, I would like to explain what I'm observing. I realize that a
lot of you probably know this stuff already, but besides giving smart
people a chance to double-check my logic, putting it down in black and
white will force me to be precise in my own thinking, and it might help
future people who google. So here goes:

(All of this applies to libcurl used in event-oriented multi mode...)

Libcurl issues a callback to CURLMOPT_SOCKETFUNCTION each time it wants to
change the state of a socket (for example, libcurl wants to read from the
socket [POLL_IN]). The application is monitoring sockets for state changes
using an OS mechanism (epoll on linux, for example). Soon after the
CURLMOPT_SOCKETFUNCTION callback, the socket state changes, and the OS
notifies the app. The app then calls curl_multi_socket_action() to do the
actual work of reading (which eventually causes libcurl to invoke the
CURLOPT_WRITEFUNCTION callback).

So far, so good. But here's the rub. Suppose that when curl does its read,
8k of data is available. It slurps it all (see readwrite_data() in
transfer.c), but that 8k is not enough to complete the response (e.g., it's
only headers + half of an html doc over http). Curl stops reading when the
non-blocking socket returns EAGAIN.

An instant later, more data arrives. How does libcurl find out?

If the application is using level-triggered notifications (e.g., poll() or
select(), or an old version of asio), then the app will get a notification
about the new data from the OS, and it will call libcurl again, and all
will be well. But if the application is using edge-triggered notifications
(e.g., epoll() with EPOLLET, or recent asio), the OS doesn't tell it that
new data has arrived, because the state of the socket that would be
reported is identical to what it was a moment ago. In other words, no
"edge" has been crossed.

Fix 1 is for the app to use level-triggered notifications. I might be able
to do this with an ugly kludge to asio, but I don't see any direct support
for that configuration. I'm still experimenting.

Fix 2 is to modify the application code that calls
curl_multi_socket_action(). Instead of calling it once, I call it in a loop
until libcurl decides it's done reading because the response is complete,
and invokes the CURMLOPT_SOCKETFUNCTION callback to tell me so. This loop
is problematic for a number of reasons, but most importantly, it amounts to
a frantic read()->EAGAIN cycle that keeps the CPU busy, unless I either
wait or poll. However, as an experiment I wrote such a loop today, and
confirmed that it solves both the problems I've been seeing:

    a) my app was hanging partway through a read
    b) the asiohiper.cpp sample was taking ridiculously long to finish a
simple download.

Fix 3 might be to abandon asio--not just for me, but for all consumers of
libcurl--due to fundamental incompatibility. As mm.w pointed out, writing
raw epoll code is certainly doable. Or there are other notification
libraries.

Anybody want to suggest other fixes? I notice that Nathan R asked a related
question about edge-triggered notifications on this mailing list on Jan 27,
and received no response. He's using libevent, which happily lets the app
choose either trigger approach.

Bottom line is that libcurl's working fine, but the asio sample may be
unsalvageable. I also think we ought to update the documentation for
curl_multi_socket_action() to discuss the edge-triggered notification
scenario; I'll do that if I submit a patch for the sample.

--Daniel

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2015-04-23