> Date: Fri, 13 Apr 2012 13:25:05 +0400
> Subject: Re: Possible issue with aborting curl connections
> From: valievkarim_at_gmail.com
> To: curl-and-python_at_cool.haxx.se
>
> I'm not sure about using pycurl with tornado, may be you need to
> return 0 from WRITEFUNCTION ?
>
> On Fri, Apr 13, 2012 at 10:05 AM, Andy Pellett <anrope_at_gmail.com> wrote:
> > Hi
> >
> > I'm working with curl via pycurl via tornado, so I might be barking up the
> > wrong tree; let me know if I am.
> >
> > I'm having an issue where my tornado app seems to work fine for a while, but
> > over time degrades until it basically isn't working at all (e.g. lots of
> > timeouts on processing requests). This makes me think the issue has
> > something to do with resource usage. I've watched memory, there doesn't seem
> > to be a leak, so now I'm looking at the number of network connections.
> >
> > I've noticed that when my app starts performing poorly, the number of
> > connections reported by:
> >
> > $ lsof -i
> >
> > is larger than 'normal'.
> >
> > I'm worried that I'm doing something wrong when it comes to handling curl
> > connections. Here's where I think I might be doing something wrong:
> >
> > I'm using tornado's curl_httpclient, and basically just plugging in
> > customized header and body callbacks.
> >
> > Sometimes I decide that I don't want to process an HTTP response's body by
> > looking at the headers. So, in the curl header callback (set with setopt
> > HEADERFUNCTION), I look for certain signs, and set a flag to ignore the body
> > of the request. If the flag is set, then in the curl body callback (set with
> > setopt WRITEFUNCTION) I return a number different than the number of bytes
> > passed in. From reading the libcurl documentation, it seems to me that this
> > should abort that transfer and (I'm hoping) free up the easy handle for
> > another connection.
> >
> > So, is this a reasonable thing to do to abort a curl connection? Or am I
> > shooting myself in the foot? Any input is appreciated.
> >
> > Thanks for your time.
> >
> > _______________________________________________
> > http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
> >
> _______________________________________________
> http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python

@Andy: Are you hitting the same domain? 99% servers block your request when they receive too many request per unit time and a similar situation is faced. Can you share an example of the situation? I am sure we shall be able to guide you better then :)

..
d_r_a_G_o_s
Are you following pycurl on twitter? https://twitter.com/#!/pycURL

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
Received on 2012-04-13