Howdy,

I've put together my first version (well, first public version) of a patch
to handle ftp server REST inconsistencies. The essential rundown is that
if the user has requested an offset above 2 GB, we check the response of
the server to see if its offset (if we can find it) matches that of the
request. If so, everything runs like normal; if not, we send another REST
request to the server with an offset of 2^31 - 1. The response is again
checked, and our offset is halved until we get the same offset from the
server as we've requested. (I'm not too terribly fond of this halving
retry, and might remove it later.)

Once curl and the server have agreed on an offset, things proceed like
normal except that curl notes what the difference between the offsets is,
and thus ignores the first X bytes of the server's data, skipping until we
hit the offset we've requested. Then writes to the body data proceed like
normal.

There is a new option, CURLOPT_CHECK_REST_RESPONSE, which can be used to
turn on or off this additional checking. By default, it is turned on, as
is listed in the documentation for this option. In addition, a new curl
command-line option has been added, --disable-rest-check, which sets the
option to false (zero). Thus, if someone discovers that this is hampering
their ability to get work done with the library or the command-line tool,
they can disable it.

This patch currently only handles ftp download REST failures. It was
built against my large file support patch, although I've modified the
patch file to work correctly on the base 7.10.8 code. I compiled with
-Wall in gcc, and didn't get any warning messages, although since the
unmodified 7.10.8 code doesn't handle offsets above 2 GB appropriately,
I'm not positive that the patch really works on that. It certainly works
on the large file offset version, though...

> > Of course, the assumption I'm making here is that anytime data is received
> > from the server it eventually passes through Curl_client_write. If that's
> > not the case, then I'm not sure the solution I'm working on will actually
> > work in all cases...
>
> That's a correct assumption for downloads.

Good! That means the patch is probably vaguely useful... :)

> For uploads, the app will tell libcurl "resume upload at pos X" and then it'll
> probably seek to that position in a file or something, then when the read
> callback gets called, the app reads from the file at that position.

Erk. Yeah, that makes sense. What a pain.

> With this new REST stuff, the library can find out that X is no good, we need
> to restart at Y instead and thus the app needs to be told to upload from that
> position instead...

Over the holidays I'll look into dealing with upload REST failures. I
don't know what the general flow of an app dealing with libcurl for
uploads would look like yet. Perhaps there is some final step done before
uploading data which could inform the app of the rest failure, or maybe
there's some form of callback...?

Incidentally, is it more helpful if my patches are directly on the CVS
development version rather than the latest release? When I'm working on
things at home, I can really do it either way.

Thanks,

Dave

-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click