Excellent! Thanks so much, Svante. I will see what I can learn from that
code.

--daniel

On Wed, Apr 22, 2015 at 11:50 PM, Svante Karlsson <svante.karlsson_at_csi.se>
wrote:

> I just noticed your post on the asio list and wanted to mention that I use
> libcurl and asio in a httpclient. Just compiled 7.42 & boost 1.57. I know I
> looked at the asio/libcurl sample when I wrote the httpclient and I
> remember that something was not working. I don't have a small sample but
> the httplibrary is on github and the http_client.cpp is small...
>
> https://github.com/bitbouncer/csi-http
>
>
> https://github.com/bitbouncer/csi-http/blob/master/csi_http/client/http_client.cpp
>
> best regards
> svante
>
>
>
>
>
>
>
>
>
> 2015-04-23 2:14 GMT+02:00 Daniel Hardman <daniel.hardman_at_gmail.com>:
>
>> >>Are you sure about that? Calling curl_multi_socket_action() repeatedly
>> is not meant to have any other effect (assuming nothing changed) so if if
>> you need to do it and if it helps, I'd say we have a bug to fix.
>>
>> I did a bit more testing today, and my confidence in the diagnosis has
>> gone up. However, I would like to explain what I'm observing. I realize
>> that a lot of you probably know this stuff already, but besides giving
>> smart people a chance to double-check my logic, putting it down in black
>> and white will force me to be precise in my own thinking, and it might help
>> future people who google. So here goes:
>>
>> (All of this applies to libcurl used in event-oriented multi mode...)
>>
>> Libcurl issues a callback to CURLMOPT_SOCKETFUNCTION each time it wants
>> to change the state of a socket (for example, libcurl wants to read from
>> the socket [POLL_IN]). The application is monitoring sockets for state
>> changes using an OS mechanism (epoll on linux, for example). Soon after the
>> CURLMOPT_SOCKETFUNCTION callback, the socket state changes, and the OS
>> notifies the app. The app then calls curl_multi_socket_action() to do the
>> actual work of reading (which eventually causes libcurl to invoke the
>> CURLOPT_WRITEFUNCTION callback).
>>
>> So far, so good. But here's the rub. Suppose that when curl does its
>> read, 8k of data is available. It slurps it all (see readwrite_data() in
>> transfer.c), but that 8k is not enough to complete the response (e.g., it's
>> only headers + half of an html doc over http). Curl stops reading when the
>> non-blocking socket returns EAGAIN.
>>
>> An instant later, more data arrives. How does libcurl find out?
>>
>> If the application is using level-triggered notifications (e.g., poll()
>> or select(), or an old version of asio), then the app will get a
>> notification about the new data from the OS, and it will call libcurl
>> again, and all will be well. But if the application is using edge-triggered
>> notifications (e.g., epoll() with EPOLLET, or recent asio), the OS doesn't
>> tell it that new data has arrived, because the state of the socket that
>> would be reported is identical to what it was a moment ago. In other words,
>> no "edge" has been crossed.
>>
>> Fix 1 is for the app to use level-triggered notifications. I might be
>> able to do this with an ugly kludge to asio, but I don't see any direct
>> support for that configuration. I'm still experimenting.
>>
>> Fix 2 is to modify the application code that calls
>> curl_multi_socket_action(). Instead of calling it once, I call it in a loop
>> until libcurl decides it's done reading because the response is complete,
>> and invokes the CURMLOPT_SOCKETFUNCTION callback to tell me so. This loop
>> is problematic for a number of reasons, but most importantly, it amounts to
>> a frantic read()->EAGAIN cycle that keeps the CPU busy, unless I either
>> wait or poll. However, as an experiment I wrote such a loop today, and
>> confirmed that it solves both the problems I've been seeing:
>>
>> a) my app was hanging partway through a read
>> b) the asiohiper.cpp sample was taking ridiculously long to finish a
>> simple download.
>>
>> Fix 3 might be to abandon asio--not just for me, but for all consumers of
>> libcurl--due to fundamental incompatibility. As mm.w pointed out, writing
>> raw epoll code is certainly doable. Or there are other notification
>> libraries.
>>
>> Anybody want to suggest other fixes? I notice that Nathan R asked a
>> related question about edge-triggered notifications on this mailing list on
>> Jan 27, and received no response. He's using libevent, which happily lets
>> the app choose either trigger approach.
>>
>> Bottom line is that libcurl's working fine, but the asio sample may be
>> unsalvageable. I also think we ought to update the documentation for
>> curl_multi_socket_action() to discuss the edge-triggered notification
>> scenario; I'll do that if I submit a patch for the sample.
>>
>> --Daniel
>>
>>
>> -------------------------------------------------------------------
>> List admin: http://cool.haxx.se/list/listinfo/curl-library
>> Etiquette: http://curl.haxx.se/mail/etiquette.html
>>
>
>

-- 
"We must remember that those mortals we meet in parking lots, offices,
elevators, and elsewhere are that portion of mankind God has given us to
love and to serve. It will do us little good to speak of the general
brotherhood of mankind if we cannot regard those who are all around us as
our brothers and sisters.”
--Spencer W. Kimball

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2015-04-23