On Tue, Aug 12, 2003 at 06:42:16PM -0700, Rob Braun wrote:
> I had a brief conversation today in SourceForge's bug tracking
> system with Daniel Stenberg about curl and >2GB file support
> in libcurl. I was pointed at this list.
>
> So, I have libcurl downloading >2GB files with the patch I'm
> attaching to this mail. This is a preliminary patch that has
> several caveats:
> 1) I have only tested http/https downloads. I haven't tested ftp,
> although the changes should work for ftp as well.
> 2) This adds a new type "filesize_t", which I've simply #defined
> to int64_t. This should be autoconf'd to something more reasonable.
> 3) This change is binary incompatible with the existing libcurl.
> This changes the size of several libcurl structures, and the parameters
> to several functions.
> 4) One problem with autoconf'ing the type of filesize_t, is that there
> are many printf() and one sscanf() strings that need to be updated
> depending on the size of filesize_t. This could be done with macros
> or internal functions to generate the correct printf()/scanf() strings
> depending on the size of filesize_t. I'm open to suggestions on how
> best to implement that, for portability.
> 4) I haven't done extensive testing, but "works for me" =)
>
> Anyway, I'd love to get comments/flames/suggestions on this patch.

This is great news! The patch somehow seems shorter than I thought it would
be solve this problem.

A solution to #4 is something like this in a header file:

#ifdef LARGE_FILE_SUPPORT
#define FILESIZE_FMT "%lld"
#else
#define FILESIZE_FMT "%ld"
#endif

used like this:

printf("transfer closed with " FILESIZE_FMT " bytes remaining to read", sz);

which sounds like what you were getting at in your note. This approach
breaks gettext, but that's not an issue with curl.

It looks like the API is going to have to change to properly support
large files. If the move is made to 64 bit ints and the API changed,
I'd like to see the new progress callbacks also changed to use long longs
instead of floats. It's useful sometimes, especially on some embedded
processors, to eliminate use of floats entirely and an API change is a
good excuse to do it. That float usage has always seemed a bit odd to
me since you can't depend on the numbers in the callback to be exact.

In the short term, perhaps the internals could be changed to use filesize_t
to add partial large file support, while the API changes are sorted out.
That ought to at least allow straightforward large downloads to succeed
with most clients.

>>> Dan

-- 
http://www.MoveAnnouncer.com              The web change of address service
          Let webmasters know that your web site has moved
-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01