On Sun, 6 Nov 2005, Daniel Stenberg wrote:

> On Thu, 3 Nov 2005, Allen Zhao wrote:
>
> > The name resoluation is fine and the transfer actually began, but not
> > finished. The downloaded file basically is not complete.
> >
> > At some point of time, I suspect maybe connection is too slow (this is a
> > Chinese RSS site, connection from US to it is slow for whatever reason).
>
> It certainly sounds very strange. Any chance you can debug this
> further to see what libcurl does or thinks when it stalls in the
> middle of a transfer?

I think I found the problem. It is not the libcurl, but a race condition I
created in the data download (well, not exactly a race condition, see
later).

Basically, this is a multithreaded RSS reader and the RSS feed is scanned
in a fixed number of threads controlled by a semaphore count.
(http://www.bitscast.com/software), and the scan of a list of RSS feed
will repeat at some intervals (as set in preference or RSS feed TTL).

What happened was that I used the same file name (calculated from MD5 of
the download URL) for each download URL. In download one thread at a time,
this is ok.

But when I enable the multithreading, there could be a special situation
that one URL is stalled in a very slow speed, and the next run of RSS URL
scan started to launched. Now, with the same MD5 for this particular URL,
I opened a new file for write and passed a new file handler to libcurl
during the download --- this thread is then locked. When the thread lock
exceed the semaphore count, everything is then stalled.

Obviously, this is a programming error. Once I start to consider a
duplicated file during url download, the problem seems to be gone for the
last several hours testing. Apparently, the assumption that a URL download
of a Chinese RSS feed can finish/time-out within 5 min is just not
realistic :-)

I hope this was all of the problem, as I am still testing the
multithreaded scan now and will know for certain by tomorrow.

Thanks for input. I really appreciate it.

Best regards,

Allen Zhao

>
> --
> Commercial curl and libcurl Technical Support: http://haxx.se/curl.html
>
Received on 2005-11-06