On 4/8/2013 00:37, curl-library-request_at_cool.haxx.se wrote:
> From: Daniel Stenberg <daniel_at_haxx.se> On Sun, 7 Apr 2013, Giuseppe
> Attardi wrote:
>> >The settings for the easy handle are:
> Which libcurl version on what platform?
Libcurl 7.28.0 on both Ubuntu 12 and CentOS 6.
>
>> >Besides I ask for compressed transfer:
>> > curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, "gzip");
>> >I configured curl --without-zlib, because I want to save the file zipped.
>> >The transfer callback (CURLOPT_WRITEFUNCTION) just copies the received bytes
>> >to a local file.
> You could still have a zlib enabled libcurl if you just add the correct
> Accept-Encoding header manually instead.
OK.
Does this affect the way libcurl determines transfer completion (you
implicitely answered no)?
Is compressed transfer always performed using chunked encoding?
Shall I disable HTTP_TRANSFER_DECODING?
>
>> >Everything works fine, except that occasionally some files are received
>> >truncated. curl does not report any error on those files.
> So what does such a transfer look like, in regards to request headers and
> response headers + body?
>
> What makes you suspect this is a libcurl problem to begin with and not a
> server-side problem?
I don't think it is a libcurl problem.
I suspect it is the way it is used in combination with libevent,
multithreading and gzip.
I am asking the list just for advice on where to look for the cause.
I exclude a server side problem, since getting the same data from the
server in different runs or with the command line tool does not have any
problem.

For example, to me it sounds unlikely that a call to
curl_multi_socket_action does not trigger the WRITEFUNCTION callback and
that when asked for curl_multi_info_read it returns CURLMSG_DONE,
without raising an error.

If the server had interrupted the transfer, libcurl would have noticed,
right?

How does gzip compression interfere with detecting transfer completion?
Since the Content-Length that servers send in this case is missing or
unreliable, libcurl must use some other mean to decide that the transfer
completed.
I am also using KeepAlive, does this matter?

If it were a server-side problem I would be even more in trouble, since
it would mean I could not use libcurl with libevent, having no way to
detect that something was going wrong, in order to repeat the download.
At the moment the only way to check that the transfer was interrupted it
to unzip the file and to get an error unzipping it.
But if I have to unzip any single file I receive, I would miss part of
the benefits to use compressed transfer.
>
>> >The problem only occurrs when a large number (~ 1000) of files are
>> >requested, and it happens occasionally for different files.
> So if you only add 500 the problem doesn't happen? If so, where's the limit?
I have no idea. I have spent 5 days trying to find a reproducible situation.
Sometimes it works with 1000 and I have to go to 5000 to see the problem.
If I just select those servers that had problems, the problem disappears.
It must be an issue due to timing or race conditions and every run is
different.
This is why I could not submit code showing the problem.

I also tested the program with valgrind for memory leaks, but there are
none.
>> >Could you suggest what could cause missing an event?
> I'm far from convinced that is actually what's happening.
>
> libcurl just can't "forget" to call the write callback for data it receives,
> so if the data was truly received it should've been delivered in a callback as
> well.
Right. This is what puzzles me.

So if you see nothing wrong in my use of events and callbacks, I will
continue investigating other parts of the program.

Thanks for the advice.

-- Beppe Attardi
> -- / daniel.haxx.se ------------------------------

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2013-04-08