[Quipment Logo]

Hello All,

I am facing issue while getting file list using curl when the file list is very large (around 4000+ files).

Title: Facing issue while fetching large amount of file list using curl.
Code :
// initialize curl parameters
int curlInit()
{
    curl = NULL;
    curl = curl_easy_init();
    logfileNameBuffer = "";
    if(curl)
    {
        // Set option to get error string in case of error
        curl_easy_setopt(curl,CURLOPT_ERRORBUFFER,errorBuffer);

        // curl_easy_setopt(curl, CURLOPT_SSH_AUTH_TYPES, CURLSSH_AUTH_AGENT);
        curl_easy_setopt(curl, CURLOPT_SSH_AUTH_TYPES,
                         CURLSSH_AUTH_PUBLICKEY);

        // Set Public, private keys for authentication
        curl_easy_setopt(curl, CURLOPT_SSH_PUBLIC_KEYFILE,
                         publicKeyPath.data());
        curl_easy_setopt(curl, CURLOPT_SSH_PRIVATE_KEYFILE,
                         privateKeyPath.data());

        /* Switch on full protocol/debug output */
        curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

        return 0;
    }
    return -1;
}

// Call back function for get file list
static size_t getFileListCallBackFunction (void *buffer, size_t size, size_t nmemb,
                            void *stream)
{
     printf(" buffer: %s",(char*)buffer);
     return size*nmemb;

// if(size*nmemb)
// {
// fileList.append(QString((char*)buffer));
// return size*nmemb;
// }
// else
// return 0;
}

// function to get file list
int getFileList()
{
    // Init curl library
    if(curlInit() != 0)
    {
        /* we failed */
       perror("Connection to SFTP server failed: %s",errorBuffer);
        // Close curl
        closeCurl();
        return -1;
    }

    // set sftp url of file path
    res = curl_easy_setopt(curl, CURLOPT_URL,
                           (url+remoteLogPath).data());

    /* Define our callback to get called when there's data to be written */
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, getFileListCallBackFunction);

    // set option to get only file names and not details
    res = curl_easy_setopt(curl, CURLOPT_DIRLISTONLY,1);

    // Perform the Log files fetch operation
    res = curl_easy_perform(curl);

    if(CURLE_OK != res)
    {
        /* we failed */
       perror("Connection to SFTP server failed: %s",errorBuffer);
        // Close curl
        closeCurl();
        return res;
    }

    // Reset directory view
    res = curl_easy_setopt(curl, CURLOPT_DIRLISTONLY,0);

    // Close curl
    closeCurl();

   return 0;
}

Issue:
This code working perfectly fine with less number of file list (around less than 1000 files).
But if I am trying to get file list from the particular folder with having around 4000+ numbers of files, it is getting file names up to around 1199 files but after that "get file list call back function" get blocked for some amount of time and then it gives error, "Could not open remote file for reading: Unknown error in libssh2 :: -30".

Please give me some hint where things are going wrong or any solution for this issue.

Thanks in advance,

Heet Kansagra
Software Engineer

R&D Embedded Department | Project Qoos | BSP

T / +91 (0) 74056 40695
heet.kansagra@<mailto:%20heet.kansagra@quipment.in> / www.quipment.in<http://www.quipment.in>

Quipment Group BV
KvK Arnhem 9208806
BTW NL.8214.81.897.B01
mail@<mailto:mail@quipment.nl> / www.quipment.nl<http://www.quipment.nl>

Hoofdkantoor Nijmegen
Kerkenbos 10-111A
6546 BJ Nijmegen
Postbus 6859
6503 GJ Nijmegen
Tel.

+31 (0)24 372 47 00

Fax.

+31 (0)24 372 47 07

Servicepunt Amsterdam
Schinkeldijkje 16K
1432 CE Aalsmeer
Tel.

+31 (0)20 441 07 55

Fax.

+31 (0)20 203 14 65

Quipment Deutschland
Tiergartenstraße 64
47533 Kleve - Deutschland
Tel.

+49 (0)2821 711 56 11

Fax.

+49 (0)2821 711 56 39

Quipment India
A-212 Shivalik Corporate Park
B/H IOC Petrol Pump
Satellite, Ahmedabad
Gujarat - 380015 - India,
Tel: +91 (0) 79 4009 7847

[Quipment Logo]

Disclaimer

This e-mail message (including any attachment) is intended only for the personal use of the recipient(s) named above. This message is confidential and may be legally privileged. If you are not an intended recipient, you may not review, copy or distribute this message. If you have received this communication in error, please notify us immediately by e-mail and delete the original message.

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html