Hi

I'm working with curl via pycurl via tornado, so I might be barking up the
wrong tree; let me know if I am.

I'm having an issue where my tornado app seems to work fine for a while,
but over time degrades until it basically isn't working at all (e.g. lots
of timeouts on processing requests). This makes me think the issue has
something to do with resource usage. I've watched memory, there doesn't
seem to be a leak, so now I'm looking at the number of network connections.

I've noticed that when my app starts performing poorly, the number of
connections reported by:

$ lsof -i

is larger than 'normal'.

I'm worried that I'm doing something wrong when it comes to handling curl
connections. Here's where I think I might be doing something wrong:

I'm using tornado's curl_httpclient, and basically just plugging in
customized header and body callbacks.

Sometimes I decide that I don't want to process an HTTP response's body by
looking at the headers. So, in the curl header callback (set with setopt
HEADERFUNCTION), I look for certain signs, and set a flag to ignore the
body of the request. If the flag is set, then in the curl body callback
(set with setopt WRITEFUNCTION) I return a number different than the number
of bytes passed in. From reading the libcurl documentation, it seems to me
that this should abort that transfer and (I'm hoping) free up the easy
handle for another connection.

So, is this a reasonable thing to do to abort a curl connection? Or am I
shooting myself in the foot? Any input is appreciated.

Thanks for your time.

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
Received on 2012-04-13