Hello libcurl hackers,

I'm having a little bit of a trouble with the multi interface. Now,
don't let the topic of this thread fool you, what I mean when I say
the DNS lookup freezes other easy handles is that they are are stalled
even after the DNS lookup is done, until a timeout is reached and I
call curl_multi_socket().

I'm writing a module for one of my projects, a module that will be fed
hundreds of URLs to perform HTTP HEAD on simultaneously. The module
uses libev for events on sockets.

The problem shows up if I have, for example, 5 URLs pointing to the
same host name. The adding of the URLs is not very prioritized, and
when the second URL is added I get this:
* Connection #0 hasn't finished name resolve, can't reuse

Doesn't seem too wrong does it? What happens next is that the first
transfer completes, and then everything stalls. Nothing happens, no
events on the sockets. In my current list of testing URLs, I have
three URLs, and each is added, as i can tell by my socket callback
printing their sockets:

socket_cb: h=0x62abd0, s=6, action=IN
* Connection #0 hasn't finished name resolve, can't reuse
socket_cb: h=0x638740, s=7, action=IN
* Connection #0 hasn't finished name resolve, can't reuse
* Connection #1 hasn't finished name resolve, can't reuse
socket_cb: h=0x6460f0, s=8, action=IN

(h is the easy handle, s is the file descriptors as given by libcurl,
both are probably uninteresting though)

If I disable http pipelining, the same freezing can be noticed, but
without the "hasn't finished name resolve" message, of course.

My first question, the first socket given by libcurl is obviously a
CURL_POLL_IN socket, is it somehow associated with the DNS response?
If so, then perhaps I should call curl_multi_socket_all() once the
FIRST socket is removed, to "signal" other easy handles waiting for
the DNS lookup to finish.

I have libcurl linked with c-ares, and have tested both 7.17.1 and
7.18.1. The problem happens when I try the hiperfifo.c example too.
Note that if I do this once, and then ctrl+c to cancel it, followed by
running the program again with the same URLs, the host name is cached
and the problem doesn't seem to arise anymore. Hence, every time I
debug the program I need to clear the DNS cache (or change the URLs),
to prevent everything from running without problems (we don't want
that, do we?).

Any way, while this stalled state is happening, if I wait 1-3 seconds,
my timeout event occurs (time given by libcurl), and after calling
curl_multi_socket(), everything works again. And while the DNS lookup
takes ~50 ms, the following curl easy handles are forced to wait 1-3
seconds before continuing... Another option is to add yet another URL
to the multi stack, which will (in my case) invoke
curl_multi_socket_all(), and the easy handles will wake up.

My second question, how can I solve this? I need to somehow know when
the DNS lookup is done, and "wake up" the curl handles when this
happens. Obviously, this problem was missed when hiperfifo.c was
implemented by Jeff Pohlmeyer, since he presumably tested it with lots
of URLs, and not 4-5 to the same host. Hence in his case too, each
added URL would invoke curl_multi_socket_all(), and thereby wake up
"sleeping" easy handles.

uname -a:
Linux slyna 2.6.24-gentoo-r8 #1 Tue May 13 17:19:21 CEST 2008 x86_64
AMD Turion(tm) 64 Mobile Technology ML-37 AuthenticAMD GNU/Linux

curl --version:
curl 7.17.1 (x86_64-pc-linux-gnu) libcurl/7.17.1 OpenSSL/0.9.8g
zlib/1.2.3 c-ares/1.4.0
Protocols: tftp ftp telnet dict http file https ftps
Features: AsynchDNS Largefile NTLM SSL libz

Tested fd "polling" methods:
select(), poll(), epoll()

Tested libcurl versions:
7.18.1, 7.17.1

I'm not able to test this on Windows until tuesday, if it should be
necessary to test on another platform. Would be nice if someone else
could try the hiperfifo.c example with 5-6 URLs pointing to the same
host and report back.

Thanks in advance,
Emil Romanus
Received on 2008-05-16