On Wed, 2004-04-21 at 02:49, Daniel Stenberg wrote:
> On Tue, 20 Apr 2004, Jesse Noller wrote:
>
> > We have a 3 GB file, we want to upload via a form POST, for instance:
> >
> > curl -i -F path=/home/foo/tmp/3GB.dat -F
> > uploadedFile=@/home/foo/tmp/3GB.dat
> >
> > The process runs for awhile after execution, and then linux throws:
> > Killed
>
> The formpost code in libcurl will create the whole post in memory before
> attempting to post it. This is of course rendering the -F feature useless if
> you want to post something that is bigger than your amount of available
> memory.
>
> (Yes I want that fixed, but my work load is high already.)
>

More than understandable. I'll look at how hard it will be for us to
retrofit fixes to our libcurl apps versus simply waiting for a fix. I'm
not yet advanced enough to hack on the libcurl code and make a patch
myself.

> > We're assuming that the URL encoding required for a form POST
>
> This is a multipart formpost, it doesn't need any URL encoding. It just needs
> RFC1867-formatting.
>

Ah, thanks!

> > The question is this - does curl itself have a method to url encode on disk,
> > and then pass a pre-encoded file to the server, and secondarily, how do we
> > avoid this with libcurl?
>
> You can avoid this by producing a full pre-formatted multipart chunk on disk
> and then using the "standard" POST function with (lib)curl as that can be done
> from a file/using the plain read callback functionality.
>
> There is however no support in libcurl to produce such a pre-formatted
> multipart post file, you would need to read up on rfc1867 details and do it
> yourself. But personally, I consider the fix to make it not produce the full
> post in memory before sending to be the most sensible action to circumvent
> this problem.
>

Ok, that makes sense. I'll just need to devise a way to do this. As I
said - in python I should be able to do it fairly easily.

> > In python/pycurl's case, it would be a matter of importing httplib and url
> > encoding the file, but how do you pass libcurl the handle to say "this file
> > is already encoded?"
>
> That's easy! libcurl doesn't encode anything you tell it to post! :-)

Makes sense! Thanks.

-jesse
Received on 2004-04-21