2008/5/16 Daniel Stenberg <daniel_at_haxx.se>:
> On Fri, 16 May 2008, Emil Romanus wrote:
>> My second question, how can I solve this?
>
> Since it seems likely this is a bug, the procedure would include figuring
> out why it happens and where, and then adjust the code to make it work
> better. I'm not sure exactly what you're asking or what else I can respond?
>

Ah, please excuse, my question was pretty misleading. What I meant was
if you could think of any solutions to work around this problem, i.e.
by calling curl_multi_socket_all() once a DNS-lookup completes to
signal waiting easy handles (and how that would be done). However,
currently I think that this would involve too much code outside of
libcurl (to determine if the socket was for a DNS-lookup, if there was
any waiting easy handles, etc), to a bug that will most likely be
fixed soon anyway. Furthermore, as long as I use a timeout in my event
loop, everything "works", but is dependent on the timeout when it
comes to multiple handles and one DNS-lookup.

>
> You should probably upgrade c-ares since there have been a few flaws fixed
> in there since that release.
>

Thanks, I did that.

2008/5/19 Daniel Stenberg <daniel_at_haxx.se>:
> On Sat, 17 May 2008, Daniel Stenberg wrote:
>
>
> I'm now quite convinced that I'm seeing at least two problems.
>
> First a problem that was introduced with the fixed pipelining code, as the
> regular case now goes into the WAITDO state which it really has no reason to
> do and in that state it has no socket to check for and then the app gets
> told to remove the check for that socket.
>
> A related bug seems to be that when I fix the code to go straight to the DO
> state when not pipelining, the multi_getsock() code will not return a socket
> to wait for there and apparently not in the DOING state either for HTTP and
> only partly for HTTPS. I'm a bit in shock over this as I don't understand
> why we haven't seen downsides from this in the past, but I'll continue to
> dig around and see if this is valid conclusions or if I maybe just have been
> too tired lately when I've worked on this.
>

I dug around in the code today looking for some place where we could
introduce a wake up call for waiting easy handles. So far, the fault
when it comes to my problem has the following path:
* 5 handles are added, each pointing to the same host
* DNS lookup is started by the first one
* The 4 others are put in a WAITDO state (not sure, but so it seems)
* DNS lookup completes, first handle continues and completes transfer

Presumably, we are only waiting for one single socket during the DNS
lookup. Once this socket needs attention, curl_multi_socket() is
called on that socket. Here, the problem arises, since the 4 other
handles are never notified of the DNS lookup being done. The only
workaround I can find is to replace *every* curl_multi_socket() or
curl_multi_socket_action() with curl_multi_socket_all(), which will
make the 4 other handles in WAITDO state notified of the DNS lookup
being complete. If curl_multi_socket_all() is never called, or
curl_multi_socket() isn't called due to timeout, then attention will
*never* be given to the 4 waiting handles. However, the bottleneck of
calling curl_multi_socket_all() instead of curl_multi_socket() on
every time attention is needed by a socket is notably big.

If libcurl could somehow change the state and update the sockets
(through the callback, I mean) to every (host-wise) related easy
handle in a multi handle when DNS lookup completes. Though I'm not
able to patch this myself, since I lack enough knowledge about the
multi stack, especially how I would find out which easy handles that
are waiting for the host lookup to complete.

In 7.18.1 the timeout "workaround" doesn't have much effect when
pipelining is enabled. The timeout with pipelining enabled is set by
libcurl to as high as 300 000 ms instead of 4 000 ms when running
without pipelining. This causes everything to stall for five minutes
instead of four seconds. However, this wouldn't be a problem if the
discussed real problem of the DNS stalling was solved, as we wouldn't
have to rely on the timeout expiring.

I'll continue experimenting and perhaps report back with something useful later.

Emil
Received on 2008-05-19