On Tue, Sep 22, 2009 at 05:48:26PM -0400, Seth Vidal wrote:
> Hi all,
> confused by a problem.
>
>> From the libcurl programming guide:
>
> "A malicious server could cause libcurl to download an infinite amount of
> data, potentially causing all of memory or disk to be filled. Setting the
> CURLOPT_MAXFILESIZE_LARGE option is not sufficient to guard against this.
> Instead, the app should monitor the amount of data received within the
> write or progress callback and abort once the limit is reached. "

It might be worthwhile to submit this question to the curl-library list
too. I would be interested in the particulars about just how this might
happen. The documentation for CURLOPT_MAXFILESIZE_LARGE seems to imply
that libcurl only checks the content-length header if it exists. One
would need to perform a transfer where no content-length header is
present in order to escape the MAXFILESIZE check.

I have to admit, that wasn't entirely what I expected. I assumed that
curl would be able to monitor the size of the transfer as long as it was
using its default write callback. I guessed, incorrectly, that if you
defined your own write callback, then you were responsible for the
check.

> From pycurl, I have my progress function checking against a set size
> and/or the content-length to make sure we've not being given an endless
> stream of data. I also do this in the header read function.
>
> How do I promote it up the stack so perform() gets the right error
> message? Can I just raise pycurl.error(63, 'msg here')? B/c that doesn't
> seem to do what I'd expect.

I used the multi interface, so my approach may not map exactly to what
you're doing. However, I created an object that contained a bunch of
information about the transfer. This object also contained the
progress_callback(), so progress_callback(self, ...) always referred to
the transfer that was in progress. I made sure that the easy handle had
a reference to this object. When you're in your error loop, processing
the easy handles that have failed, look at the object that you assigned
to the handle. If the reason for the transfer's abort was saved in that
object, then all you would have to do is check the code (transfer too
large, aborted by user, whatever) and then raise the error that you
actually wanted to raise when you aborted the callback.

I hope that explanation makes some amount of sense.

-j
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
Received on 2009-09-23