Hi, Daniel:

I am working on a research project to developer a large-scale http
fetcher. The goal is to download a million web pages as fast as possible.
Libcurl is chosen because of its great functionality and support for
asynchronous transfer and dns resolves.

In my prototype system, I use the python interface, pycurl. I have
upgraded to the latest 7.11.1 version. Multi interface is used and Ares
support is enabled. The development is quite easy. However, I am facing a
strange problem: some handles get stuck in name resolve and never come
out. According to debugging info, the connection time and total time of
those dead handles are all zero. If I disable Ares support, the problem
seems to be gone.

After checking pycurl code, I don't think it's a problem in the python
wrapper. So I wonder if it's a bug in libcurl. Because I am doing a lot of
dns query, the dns server can be busy and drop some queries. In that case,
what does libcurl do? Can it time-out? I looked into libcurl code and
didn't see anything wrong so far.

Thanks a lot for your advice.

Roy
Received on 2004-03-29