curl-and-python
CURLE_OPERATION_TIMEDOUT while connecting.
Date: Thu, 26 Apr 2012 14:10:59 +0400
Hello.
I am trying to make use of pycurl with multi interface + pyev and
there are some issues i dont understand from docs:
I am not planning to use curl timeout system but to use libev timers
and manually remove_handle from multi when timeout happens. But i have
tried curl timeouts and have issues described in 2)
1) I create curl and and handle to multi with multi.add_handle(c) but
if dont call multi.socket_action(pycurl.SOCKET_TIMEOUT, 0) nothing
fetches. But there is also multi.socket_all() witch is seems to be
doing the same? Is there any difference in calling
multi.socket_action(pycurl.SOCKET_TIMEOUT, 0) or multi.socket_all()
What is cheaper?
2) the main question: when a keepalive socket timeout outed and curl
thinks connection is dead and tells so via error list of
multi.info_read() do i have to manually start another Curl() instance
in order to fetch url finally? (i thought curl will just ask for
another socket) Does creating pool of curls will somehow help in such
situation?
CURLE_OPERATION_TIMEDOUT means connect timeout or fetch timeout or
both? Can it be related to any other?
Here is the log lines related to curl:
2012-04-26 10:48:02,010 INFO 3404 start curl
2012-04-26 10:48:02,011 DEBUG 3404 127.0.0.1 debug: 0 'Connection #0
seems to be dead!\n'
2012-04-26 10:48:02,011 DEBUG 3404 127.0.0.1 debug: 0 'Closing connection #0\n'
2012-04-26 10:48:02,011 DEBUG 3404 127.0.0.1 debug: 0 'Connection #1
seems to be dead!\n'
2012-04-26 10:48:02,011 DEBUG 3404 127.0.0.1 debug: 0 'Closing connection #1\n'
2012-04-26 10:48:02,012 DEBUG 3404 127.0.0.1 debug: 0 'Connection #3
seems to be dead!\n'
2012-04-26 10:48:02,012 DEBUG 3404 127.0.0.1 debug: 0 'Closing connection #3\n'
2012-04-26 10:48:02,012 DEBUG 3404 127.0.0.1 debug: 0 'Connection #4
seems to be dead!\n'
2012-04-26 10:48:02,012 DEBUG 3404 127.0.0.1 debug: 0 'Closing connection #4\n'
2012-04-26 10:48:02,013 DEBUG 3404 127.0.0.1 debug: 0 'Closing connection #0\n'
2012-04-26 10:48:02,014 DEBUG 3404 baz.com debug: 0 'About to
connect() to baz.com port 80 (#0)\n'
2012-04-26 10:48:02,014 DEBUG 3404 baz.com debug: 0 ' Trying <baz.com_ip>... '
2012-04-26 10:48:02,015 DEBUG 3404 baz.com debug: 0 '0x6281450 is at
send pipe head!\n'
2012-04-26 10:48:02,015 ERROR 3404 curl_socket_cb: event: 2, fd:15,
multi:<pycurl.CurlMulti object at 0x17f8240> data:None
2012-04-26 10:48:02,015 ERROR 3404 self.watchers: {}
2012-04-26 10:48:02,015 ERROR 3404 creating new watcher
2012-04-26 10:48:02,016 DEBUG 3404 foo.com debug: 0 'Re-using existing
connection! (#2) with host foo.com\n'
2012-04-26 10:48:02,016 DEBUG 3404 foo.com debug: 0 'Connected to
foo.com (<foo.com_ip>) port 80 (#2)\n'
2012-04-26 10:48:02,016 DEBUG 3404 foo.com debug: 0 '0x6274b80 is at
send pipe head!\n'
2012-04-26 10:48:02,016 DEBUG 3404 foo.com debug: 2 'GET /<long line here>'
2012-04-26 10:48:02,017 ERROR 3404 curl_socket_cb: event: 1, fd:17,
multi:<pycurl.CurlMulti object at 0x17f8240> data:None
2012-04-26 10:48:02,017 ERROR 3404 self.watchers: {15: <pyev.Io object
at 0x5d8ef30>}
2012-04-26 10:48:02,017 ERROR 3404 creating new watcher
2012-04-26 10:48:02,018 DEBUG 3404 bar.com debug: 0 'About to
connect() to bar.com port 80 (#1)\n'
2012-04-26 10:48:02,019 DEBUG 3404 bar.com debug: 0 ' Trying <bar.com_ip>... '
2012-04-26 10:48:02,019 DEBUG 3404 bar.com debug: 0 '0x6262bc0 is at
send pipe head!\n'
2012-04-26 10:48:02,019 ERROR 3404 curl_socket_cb: event: 2, fd:16,
multi:<pycurl.CurlMulti object at 0x17f8240> data:None
2012-04-26 10:48:02,019 ERROR 3404 self.watchers: {17: <pyev.Io object
at 0x60ae8d0>, 15: <pyev.Io object at 0x5d8ef30>}
2012-04-26 10:48:02,020 ERROR 3404 creating new watcher
2012-04-26 10:48:02,020 DEBUG 3404 process_curl_complete_handles: 0,
[], [(<pycurl.Curl object at 0x62554d0>, 28, '')]
2012-04-26 10:48:02,021 ERROR 3404 c.fetcher.fail
2012-04-26 10:48:02,022 ERROR 3404 XXX curl fail: 28,
Why Connection #0 gets closed 2 times? Does error list [(<pycurl.Curl
object at 0x62554d0>, 28, '')] returned by multi.info_read() related
to dead connection?
the curl was opened with
setopt(pycurl.CONNECTTIMEOUT, 1)
setopt(pycurl.TIMEOUT, 5)
version info:
version = 'libcurl/7.21.3 GnuTLS/2.8.6 zlib/1.2.3.4 libidn/1.18'
(3, '7.21.3', 464131, 'x86_64-pc-linux-gnu', 1597, 'GnuTLS/2.8.6', 0,
'1.2.3.4', ('dict', 'file', 'ftp', 'ftps', 'gopher', 'http', 'https',
'imap', 'imaps', 'ldap', 'ldaps', 'pop3', 'pop3s', 'rtsp', 'smtp',
'smtps', 'telnet', 'tftp'), None, 0, '1.18')
-- -------------------------------------------- Турнаев Евгений Викторович +7 906 875 09 43 -------------------------------------------- _______________________________________________ http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-pythonReceived on 2012-04-26