On Thu, 26 Aug 2004 a.j.vandenberg02_at_kpn.com wrote:

> ---------------------------------
> In the trace-file:
> <= Recv header, 22 bytes (0x16)
> 0000: content-length: 4334

...

> <= Recv data, 4341 bytes (0x10f5)

...

> Part of the header-file (obtained with the -D option):

...

> content-length: 4334

> When I download the file manually wit Internet Explorer the size is 4441

... and what does the content-length: say when IE downloads it? I guess that
is hard for you to figure out since it is encrypted. Have you tried to set the
user-agent to look like IE?

> In all testcases (test with 34 files) the Recv data in the trace-file is 100
> less then the size of the manually downloaded file.

With "manually downloaded" you mean using IE?

> But the size of the with curl obtained file lies is exactly (4334 bytes) the
> content-length in the header-file

curl respects the content-length rather aggressively and thus stops
downloading when it reaches the amount given in that header.

> Has anyone an idea why the size in the trace-file differs from the size in
> the header and the actual size?

Obviously, the server claims the file is of a certain size and then delivers a
different (somewhat larger) size. This is major bad and a server flaw if you
ask me.

Let me quote a little paragraph from RFC1945 (the HTTP 1.0 spect) that is
highly relevant:

       Note: Some older servers supply an invalid Content-Length when
       sending a document that contains server-side includes dynamically
       inserted into the data stream. It must be emphasized that this
       will not be tolerated by future versions of HTTP. Unless the
       client knows that it is receiving a response from a compliant
       server, it should not depend on the Content-Length value being
       correct.

... if a Content-Length is provided, that value is believed to indicate the
actual and true size of the delivered content.

To work around that, you'll have to do one of these:

1. modify curl to not respect the content-length this blindly

2. prevent the content-length from arriving to curl by using a filtering proxy
    or similar

3. fix the server to send back proper headers

4. convince a curl developer that curl is somehow wrong and he/she can do #1

> In the trace there are 3 Recv header's With the -D option I only get one
> header-file.

-D writes all received headers to the same file.

> Is there a way to see what's in the other two headers?

Those headers are all in the same file. Read that file!

-- 
      Daniel Stenberg -- http://curl.haxx.se -- http://daniel.haxx.se
       Dedicated custom curl help for hire: http://haxx.se/curl.html