cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: Fetching More than 500 files from Server

From: Swamy Mudhbasalar <swamydbit_at_gmail.com>
Date: Tue, 19 Jun 2012 09:30:14 +0530

Hi,

We have added one more line to our code to reduce the connection established.

curl_easy_setopt(ptCurlHandle, CURLOPT_COOKIE, pstrSessionCookie)

After adding the CURLOPT_COOKIE, the connection issue was solved but
data fetched is zero bytes

Please let me know what may be causing this issue.

Regards,
SAM

> ------------------------------
>
> Message: 3
> Date: Thu, 14 Jun 2012 10:33:08 +0200 (CEST)
> From: Daniel Stenberg <daniel_at_haxx.se>
> To: libcurl development <curl-library_at_cool.haxx.se>
> Subject: Re: Fetching More than 500 files from Server
> Message-ID: <alpine.DEB.2.00.1206141023270.19610_at_tvnag.unkk.fr>
> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
>
> On Thu, 14 Jun 2012, Swamy Mudhbasalar wrote:
>
> Please stop top-posting and full-quoting. If you intend to reply to emails
> here, switch off the digest-mode!
>
>> LibCurl Version : 7.21.0
>
> We have made 13 releases since that, so it's not unthinkable that a newer
> version works better or at least different. It would be very interesting to
>
> learn if you see anything similar with a recent version.
>
>> curl_easy_setopt(ptCurlHandle, CURLOPT_ERRORBUFFER, acErrorBuffer);
>> curl_easy_setopt(ptCurlHandle, CURLOPT_URL, pstrURL);
>> curl_easy_setopt(ptCurlHandle, CURLOPT_WRITEFUNCTION, VmwGetCB);
>> curl_easy_setopt(ptCurlHandle, CURLOPT_WRITEDATA, (void *)&tContext);
>> curl_easy_setopt(ptCurlHandle, CURLOPT_USERAGENT, pstrUserAgent);
>> curl_easy_setopt(ptCurlHandle, CURLOPT_SSL_VERIFYPEER, 0L);
>> curl_easy_setopt(ptCurlHandle, CURLOPT_SSL_VERIFYHOST, 0L);
>> curl_easy_setopt(ptCurlHandle, CURLOPT_USERPWD, pstrUserPass);
>
> If these are truly the only options you use then the connection: header in
> the
> request is truly a mystery to me...
>
> You could consider using CURLOPT_VERBOSE to get to see about libcurl's
> choices
> to keep connections open or close them etc.
>
>>> In which sense is this connection then still "alive" ? How do you see it
>>>
>>> being kept in 30 minutes?
>>
>> When I run the netstat command from the command prompt, i can see many
>> connection established. For every file there is one connection
>> established.
>
> Even ESTABLISHED! That sounds so completely broken. When libcurl concludes
> that a connection can't be kept for further re-use it will _close_ that
> socket. If it doesn't conclude that, it will keep the connection in its
> connection pool (and possibly close an older one to make room) but that pool
>
> will only fit a finite number of connections and that will be much less than
>
> 500 by default...
>
>> All this connection will get closed after some time approximately 30
>> mins.
>
> So who or what is closing them after 30 minutes?
>
> --
>
> / daniel.haxx.se
>
>
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2012-06-19