cURL / Mailing Lists / curl-library / Single Mail

curl-library

RE: Performance issue with libcurl library for SFTP upload

From: Daniel Stenberg <daniel_at_haxx.se>
Date: Thu, 16 Jun 2011 14:49:34 +0200 (CEST)

On Thu, 16 Jun 2011, Srikanth Ramaprasad08 wrote:

> Que: what's the RTT (ping time) from the client to the server?
>
> Ans: I tried pinging the server and got a "Request timed out" message.
...
> Please let me know if there is any other way I can find out the ping time
> for a SFTP connection.

The only other way I can think of then is to liten on network traffic between
the machines and figure it out. Not as easy though.

The reason I asked about ping time is the back-and-forth nature of SFTP[*]. We
(I am the leading developer and maintainer of libssh2) improved the situation
performance wise significantly in libssh2 1.2.8. But in order to really make
SFTP fly, the application using libssh2 should be modified accordingly and use
larger buffers and optimally also a "sliding window" kind of buffer technique.
libcurl has not been modified or changed even the slightest to deal with SFTP
better since 1.2.8.

The larger the RTT is, the worse the problem gets.

Even without any changes done to libcurl it should be faster with 1.2.8 than
with older libssh2 versions, althought it isn't necassarily a very noticable
difference. I've not actually done any particular tests with that.

> Ans: curl-7.21.6. libssh2-1.2.8,

A first quick test is to bump CURL_MAX_WRITE_SIZE (in include/curl/curl.h)
significantly (perhaps to 100K or even 1M) and rebuild libcurl and see what
difference you'll see.

Depending on what results you see, we should really consider doing some kind
of changes to libcurl to improve SFTP performance.

I'll appreciate help to get it done.

[*] = http://daniel.haxx.se/blog/2010/12/08/making-sftp-transfers-fast/

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Received on 2011-06-16