cURL / Mailing Lists / curl-library / Single Mail

curl-library

sftp performance issue

From: cesare pietra <cesare.pietra_at_valueteam.com>
Date: Fri, 11 Jul 2008 11:47:33 +0200

Hi All.

We're in charge to develop a simple file transfer component of a large
system, managing huge amounts of data per day.

We've been requested using sftp transfer protocol.

 

Compiling curl-7.18.2-1.src.rpm on a 2 processor Linux redhat (x86_64
GNU/Linux) box, we added support for libssh2, by configuring -with-libssh2.

Upon successful installation, we tried running a sample code snippet I'm
catting herebye, in order to test the basic sftp get functionality.

 

//****************************CODE START*********************************

  #include <stdio.h>

  #include <curl/curl.h>

  #include <curl/types.h>

  #include <curl/easy.h>

  #include <errno.h>

  

  int MAX_LINE=1024;

 

  struct FtpFile {

    const char *filename;

    FILE *stream;

  };

 

  int main(void)

  {

    CURL *curl;

    CURLcode res;

    struct FtpFile ftpfile={

      "got.txt", /* name to store the file as if succesful */

      NULL

  };

  

    curl_global_init(CURL_GLOBAL_DEFAULT);

  

    curl = curl_easy_init();

    if(curl) {

      res=curl_easy_setopt(curl,
CURLOPT_URL,"sftp://gordian/~/druFileTransfer/sourcedir/filediprova6.txt");

      if(CURLE_OK != res) {

        /* we failed */

        fprintf(stderr, "curl told us %d\n", res);

            perror("errore: ");

 

 

      res= curl_easy_setopt(curl,CURLOPT_USERPWD, "cesare:XXX");

      if(CURLE_OK != res) {

        /* we failed */

        fprintf(stderr, "curl told us %d\n", res);

            perror("errore: ");

      }

 

      ftpfile.stream=fopen(ftpfile.filename, "wb");

    

      res=curl_easy_setopt(curl, CURLOPT_WRITEDATA, ftpfile.stream);

  

      /* Switch on full protocol/debug output */

      curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

  

      res = curl_easy_perform(curl);

  

      /* always cleanup */

      curl_easy_cleanup(curl);

  

      if(CURLE_OK != res) {

        /* we failed */

        fprintf(stderr, "dopo l'esecuzione curl told us %d\n", res);

            perror("errore: ");

            //exit (1);

      }

     

    }

  

    if(ftpfile.stream)

      fclose(ftpfile.stream); /* close the local file */

  

    curl_global_cleanup();

  

    return 0;

  }

//*************************CODE END***********************************

 

Running this code against the time utility, that's what we measured:

* About to connect() to gordian port 22 (#0)

* Trying 10.96.4.31... * connected

* Connected to gordian (10.96.4.31) port 22 (#0)

* SSH authentication methods available: publickey,password

* Using ssh public key file /home/grmdati/.ssh/id_dsa.pub

* Using ssh private key file /home/grmdati/.ssh/id_dsa

* SSH public key authentication failed: Username/PublicKey combination
invalid

* Initialized password authentication

* Authentication complete

 

* Connection #0 to host gordian left intact

* Closing connection #0

 

real 38m30.764s

user 0m21.288s

sys 0m7.542s

 

(Don't think the key authorization problem can be an issue.)

 

 

On the other hand, getting the same remote file by the /usr/bin/sftp
command, configured with 3des as cipher algorithm, that's what we got
(difference between 2m9.482s and 2m05s is interactive password):

 

time sftp cesare_at_gordian:druFileTransfer/sourcedir/fil

Connecting to gordian...

cesare_at_gordian's password:

Fetching /export/home/cesare/druFileTransfer/sourcedir/filediprova6.txt to
filediprova6.txt

/export/home/cesare/druFileTransfer/sourcedir/fi 100% 557MB 4.5MB/s
02:05

real 2m9.482s

user 0m10.829s

 

Isn't it weird?

 

We are getting the file from another box on the same 100Mb LAN, with no
proxy in the middle. Same tests have been carried out onto the machine the
code itself is running into, via the loopback interface, resulting indeed in
similar behavior.

 

We didn't figure out how to set the Cipher algorithm using libcurl, we're
therefore assuming the 3des algorithm being used in our code too for the
comparison to make sense.

It turns out there might be some configuration problem in the libcurl or in
the libssh2.

 

Can someone please help?

 

Thanks in advance,

\c

 

 

 

Cesare Pietra

Value Team s.p.a.

Via del Brennero Km. 4,

Loc. La Figuretta

56123

Pisa

tel. +39 050 8009526

email: cesare.pietra_at_valueteam.com

http: www.valueteam.com

 
Received on 2008-07-11