stop sending email to me.

> From: curl-and-python-request_at_cool.haxx.se
> Subject: curl-and-python Digest, Vol 40, Issue 5
> To: curl-and-python_at_cool.haxx.se
> Date: Thu, 24 Sep 2009 12:00:02 +0200
>
> Send curl-and-python mailing list submissions to
> curl-and-python_at_cool.haxx.se
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
> or, via email, send a message with subject or body 'help' to
> curl-and-python-request_at_cool.haxx.se
>
> You can reach the person managing the list at
> curl-and-python-owner_at_cool.haxx.se
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of curl-and-python digest..."
>
>
> Today's Topics:
>
> 1. Re: aborting a transaction (johansen_at_sun.com)
> 2. Re: aborting a transaction (Seth Vidal)
> 3. Re: aborting a transaction (Daniel Stenberg)
> 4. Re: aborting a transaction (johansen_at_sun.com)
> 5. Re: aborting a transaction (Daniel Stenberg)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 23 Sep 2009 14:07:52 -0700
> From: johansen_at_sun.com
> To: curl with python <curl-and-python_at_cool.haxx.se>
> Subject: Re: aborting a transaction
> Message-ID: <20090923210752.GC581479_at_eng.sun.com>
> Content-Type: text/plain; charset=us-ascii
>
> On Tue, Sep 22, 2009 at 10:15:17PM -0400, Seth Vidal wrote:
> > arguably you could be handed an endless header and obliterate all memory.
> > I put a configurable max header size (default to 2mb) in place and work
> > from there.
>
> Is this actually a case that we can hit if the client hasn't installed a
> HEADERFUNCTION? I'm a bit surprised that curl would hold onto all of
> the headers, instead of just extracting the ones that it cares about.
>
> >> I used the multi interface, so my approach may not map exactly to what
> >> you're doing. However, I created an object that contained a bunch of
> >> information about the transfer. This object also contained the
> >> progress_callback(), so progress_callback(self, ...) always referred to
> >> the transfer that was in progress. I made sure that the easy handle had
> >> a reference to this object. When you're in your error loop, processing
> >> the easy handles that have failed, look at the object that you assigned
> >> to the handle. If the reason for the transfer's abort was saved in that
> >> object, then all you would have to do is check the code (transfer too
> >> large, aborted by user, whatever) and then raise the error that you
> >> actually wanted to raise when you aborted the callback.
> >>
> >
> > That does make sense and it is what I ended up doing - but it felt a
> > little hacky.
>
> I have to admit I thought the same thing the first time I wrote code
> this way. However, having the callback attached to an object that knows
> something about its own state seemed like a reasonable approach in an OO
> language.
>
> > I also wish there was a nice way to go from errorcode->string in the
> > python library.
>
> There's easy.errstr(), but it only returns the error string from the
> current handle. There's no curl_easy_strerror() equivalent in pycurl,
> AFAICT.
>
> -j
>
>
>
> ------------------------------
>
> Message: 2
> Date: Wed, 23 Sep 2009 17:12:27 -0400 (EDT)
> From: Seth Vidal <skvidal_at_fedoraproject.org>
> To: curl with python <curl-and-python_at_cool.haxx.se>
> Subject: Re: aborting a transaction
> Message-ID:
> <alpine.LFD.2.00.0909231710370.20158_at_localhost.localdomain>
> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
>
>
>
> On Wed, 23 Sep 2009, johansen_at_sun.com wrote:
>
> > On Tue, Sep 22, 2009 at 10:15:17PM -0400, Seth Vidal wrote:
> >> arguably you could be handed an endless header and obliterate all memory.
> >> I put a configurable max header size (default to 2mb) in place and work
> >> from there.
> >
> > Is this actually a case that we can hit if the client hasn't installed a
> > HEADERFUNCTION? I'm a bit surprised that curl would hold onto all of
> > the headers, instead of just extracting the ones that it cares about.
>
> I don't know - but I needed a separate headerfunction b/c of how I need to
> extract the content-length before continuing on.
>
> So in my case, it was necessary.
>
>
>
> > There's easy.errstr(), but it only returns the error string from the
> > current handle. There's no curl_easy_strerror() equivalent in pycurl,
> > AFAICT.
>
> the latter is what I want. It'd be nice to turn pycurl.E_FILESIZE_EXCEEDED
> into a slightly nicer output.
>
> -sv
>
>
>
> ------------------------------
>
> Message: 3
> Date: Wed, 23 Sep 2009 23:22:09 +0200 (CEST)
> From: Daniel Stenberg <daniel_at_haxx.se>
> To: curl with python <curl-and-python_at_cool.haxx.se>
> Subject: Re: aborting a transaction
> Message-ID: <alpine.DEB.2.00.0909232320230.27072_at_tvnag.unkk.fr>
> Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
>
> On Wed, 23 Sep 2009, johansen_at_sun.com wrote:
>
> >> arguably you could be handed an endless header and obliterate all memory. I
> >> put a configurable max header size (default to 2mb) in place and work from
> >> there.
> >
> > Is this actually a case that we can hit if the client hasn't installed a
> > HEADERFUNCTION? I'm a bit surprised that curl would hold onto all of the
> > headers, instead of just extracting the ones that it cares about.
>
> It doesn't. It does create a buffer large enough to hold a single header line,
> so if you send an enourmously long header line libcurl will allocate such a
> buffer. I guess we could consider a max length for sensible headers...
>
> --
>
> / daniel.haxx.se
>
>
> ------------------------------
>
> Message: 4
> Date: Wed, 23 Sep 2009 16:03:30 -0700
> From: johansen_at_sun.com
> To: curl with python <curl-and-python_at_cool.haxx.se>
> Subject: Re: aborting a transaction
> Message-ID: <20090923230330.GI581479_at_eng.sun.com>
> Content-Type: text/plain; charset=us-ascii
>
> Daniel,
>
> On Wed, Sep 23, 2009 at 11:22:09PM +0200, Daniel Stenberg wrote:
> > On Wed, 23 Sep 2009, johansen_at_sun.com wrote:
> >
> >>> arguably you could be handed an endless header and obliterate all
> >>> memory. I put a configurable max header size (default to 2mb) in
> >>> place and work from there.
> >>
> >> Is this actually a case that we can hit if the client hasn't installed
> >> a HEADERFUNCTION? I'm a bit surprised that curl would hold onto all of
> >> the headers, instead of just extracting the ones that it cares about.
> >
> > It doesn't. It does create a buffer large enough to hold a single header
> > line, so if you send an enourmously long header line libcurl will
> > allocate such a buffer. I guess we could consider a max length for
> > sensible headers...
>
> Thanks for clarifying this point.
>
> Perhaps I have misunderstood, but based upon your description, doesn't
> this limit what the HEADERFUNCTION can do to defend against such an
> attack? If the remote host sends libcurl an arbitrarily long string and
> the library allocates space to hold it, then the client will have
> allocated a large amount of memory before the HEADERFUNCTION has a
> chance to return a write error and abort the transaction. Is the idea
> that if the HEADERFUNCTION returns a write error, then we'll free the
> header buffer before we get into too much trouble?
>
> Thanks,
>
> -j
>
>
> ------------------------------
>
> Message: 5
> Date: Thu, 24 Sep 2009 05:55:58 +0200 (CEST)
> From: Daniel Stenberg <daniel_at_haxx.se>
> To: curl with python <curl-and-python_at_cool.haxx.se>
> Subject: Re: aborting a transaction
> Message-ID: <alpine.DEB.2.00.0909240547170.26828_at_tvnag.unkk.fr>
> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
>
> On Wed, 23 Sep 2009, johansen_at_sun.com wrote:
>
> > Perhaps I have misunderstood, but based upon your description, doesn't this
> > limit what the HEADERFUNCTION can do to defend against such an attack?
>
> Yes. That would be an attack that libcurl itself needs counter-measures for,
> as the app won't be the one that can decide when to stop.
>
> libcurl does the service of putting each complete header in a buffer before it
> calls the app simply to make life easier for the apps, since they can thus
> assume that they always get full headers and not have to support getting tiny
> fragments or pieces with embedded newlines etc.
>
> > If the remote host sends libcurl an arbitrarily long string and the library
> > allocates space to hold it, then the client will have allocated a large
> > amount of memory before the HEADERFUNCTION has a chance to return a write
> > error and abort the transaction. Is the idea that if the HEADERFUNCTION
> > returns a write error, then we'll free the header buffer before we get into
> > too much trouble?
>
> It's simply so that libcurl hasn't been done to safe-guard against an "attack"
> of this kind. Responding with a HTTP stream that sends an "infinite" length
> HTTP header will cause libcurl to attempt to allocate a buffer for that header
> and thus it will realloc() away until it gets no further memory.
>
> lib/transfer.c:readwrite_http_headers() is the responsible function. I guess
> at least some kind of fixed maximum header length (like a 100KB or 1MB or so)
> is suitable to use there. Wouldn't you agree on that?
>
> --
>
> / daniel.haxx.se
>
>
> ------------------------------
>
> _______________________________________________
> curl-and-python mailing list
> curl-and-python_at_cool.haxx.se
> http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
>
>
> End of curl-and-python Digest, Vol 40, Issue 5
> **********************************************
                                                _________________________________________________________________ Lauren found her dream laptop. Find the PC that’s right for you. http://www.microsoft.com/windows/choosepc/?ocid=ftp_val_wl_290

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
Received on 2009-09-24