>>> pycurl.version
'libcurl/7.18.0 zlib/1.2.3 c-ares/1.4.0'
>>> pycurl.version_info()
(3, '7.18.0', 463360, 'i686-pc-linux-gnu', 648, None, 0, '1.2.3',
('tftp', 'ftp', 'telnet', 'dict',
'http', 'file'), '1.4.0', 66560, None)

I've been working on a project using pycurl and lots of Curl handles
(200-500) inside a CurlMulti. I've found that when I increase from
200 to 500 handles I start getting lots of E_COULDNT_RESOLVE_HOST
errors. The error message that comes with it is the regular "couldn't
resolve host", but also includes an unexpected "Error Reading File"
(which I believe is coming from c-ares). If I immediately do a
socket.gethostbyname for the same hostname it will resolve just fine.
In fact if I record all the failures and run socket.gethostbyname
serially for all of them, they'll succeed.

What's strange is that I can do the same workload with 100 to 200
handles and I don't have a problem. But if I double that number to,
say 400 handles on one CurlMulti, everything runs fine for a while
(with better throughput), but after some time I start getting "Error
Reading File" for my requests and I have to kill the app.

I've also tried splitting my work across different threads each using
their own CurlMulti objects and a fraction of the Curl handles, but I
still get the error.

Is this amount (>200) more handles than is reasonably expected?
Is this a known limitation (max handles or simultaneous connections)?
Is this my DNS server not responding to so many simultaneous requests?
(I doubt this since once I start getting the error all subsequent
requests start failing with the same message)
Anyone know what "Error Reading File" means from c-ares?

Thanks in advance,
Nick Gerner
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
Received on 2008-02-11