I found the cause of the truncated files.
It is due to libcurl aborting some transfers because of reaching the low
speed limit.
Individually, the files get transferred perfectly, but when they are
transfered in bunches of 1000, occasionally for a few of them,
unpredicatabily, the transfer slows down and aborts.

I am using a fast 1Gb Internet connection and monitoring the bandwidth
usage shows that I am well below saturation.

Could it be due to other OS issues, like the number of connections
becoming too large?
I did increase though the limit on open files to 200000.
The machine has 24 cores, 28 GB of memory and is largely idle.

Any advice will be appreciated.

I enclose a reduced version of the program I am using, to replicate the
situation.

You can compile it on linux with

     c++ -o evstress evstress.cpp -lcurl -levent_core

You run it by supplying a file with a list of URLs to donwload:

     evstress url-list

and it will create a directory /tmp/Cache, where it will store the
downloaded files.

After running it, you can check which files were truncated by doing:

zgrep WHATEVER /tmp/Cache/*

I get from 0 to 20 truncated files out of a list of 3000.
The number is random and sometimes occurs also with a list of less
than1000 URLs.

I am not posting the list of URLs I am using, but I think any list of
2000 or more URLs will do.

Thanks again.

-- Beppe Attardi

On 4/8/2013 10:42, Giuseppe Attardi wrote:
> On 4/8/2013 00:37, curl-library-request_at_cool.haxx.se wrote:
>> From: Daniel Stenberg <daniel_at_haxx.se> On Sun, 7 Apr 2013, Giuseppe
>> Attardi wrote:
>>> >The settings for the easy handle are:
>> Which libcurl version on what platform?
> Libcurl 7.28.0 on both Ubuntu 12 and CentOS 6.
> ...
>>
>>> >The problem only occurrs when a large number (~ 1000) of files are
>>> >requested, and it happens occasionally for different files.
>> So if you only add 500 the problem doesn't happen? If so, where's the
>> limit?
> I have no idea. I have spent 5 days trying to find a reproducible
> situation.
> Sometimes it works with 1000 and I have to go to 5000 to see the problem.
> If I just select those servers that had problems, the problem disappears.
> It must be an issue due to timing or race conditions and every run is
> different.
> This is why I could not submit code showing the problem.
>
> I also tested the program with valgrind for memory leaks, but there
> are none.
>>> >Could you suggest what could cause missing an event?
>> I'm far from convinced that is actually what's happening.
>>
>> libcurl just can't "forget" to call the write callback for data it
>> receives,
>> so if the data was truly received it should've been delivered in a
>> callback as
>> well.
> Right. This is what puzzles me.
>
> So if you see nothing wrong in my use of events and callbacks, I will
> continue investigating other parts of the program.
>
> Thanks for the advice.
>
> -- Beppe Attardi
>> -- / daniel.haxx.se ------------------------------
>

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html