cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: Unnecessary delay downloading multiple files (via SFTP)?

From: Patrik Thunström <patrik.thunstrom_at_bassetlabs.com>
Date: Tue, 17 Feb 2009 17:27:11 +0100

Daniel Stenberg wrote:
> On Tue, 17 Feb 2009, Patrik Thunström wrote:
>
>> For now, the only protocol I'm using for transfers is SFTP, together
>> with the curl_easy-interface, so I cannot say if this "issue" is
>> affecting other protocols too. I've got the picture that this issue
>> is not affecting the multi-interface by reading the mailing list
>> archive.
>
> We've recently done some SFTP related changes and improvements to
> libcurl and libssh2 so it is very important what specific versions of
> these libs you're using. Anything before libcurl 7.19.3 with libssh2
> 1.0 is known to have flaws. Of course I'm not claiming that pair
> doesn't have it selection of flaws too, but earlier versions had even
> more! ;-)
Yes, I've understood that there's been quite some progress with both
Libssh2 and how libcurl uses it. I'm currently using the latest stable
releases of the libraries, meaning libcurl 7.19.3, Libssh2 1.0 and also
OpenSSL 0.9.8j.
>
>> What I'm suspicious about is the performance when downloading
>> multiple files from the same SFTP connection. The transfer itself is
>> nice and fast, but for each of the transfers I noticed there seemed
>> to be a fixed delay in between transferring the next file. Tracking
>> it down I found out that there is a constant timeout value given at
>> line 1856 of Transfer.c, set at 1000ms.
>
> That sounds like a bug. If nothing happens for 1000ms then it
> shouldn't make much difference if it tries to continue or not. If it
> _does_ continue after just having waited for 1000ms then surely the
> wait was pointless.
Yes, and as I also mentioned the value of the constant timeout does not
seem to effect the execution, besides how long the delay is in between
transfers. Could it just be that there is action on the sockets just as
libcurl is about to issue the call where it performs the wait, or is
this a atomic operation where libcurl does the evaluation of the socket
state and starts its wait before anything can happen on the sockets?

Best Regards
Patrik Thunström
Received on 2009-02-17