cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: Fetching More than 500 files from Server

From: Daniel Stenberg <daniel_at_haxx.se>
Date: Thu, 14 Jun 2012 10:33:08 +0200 (CEST)

On Thu, 14 Jun 2012, Swamy Mudhbasalar wrote:

Please stop top-posting and full-quoting. If you intend to reply to emails
here, switch off the digest-mode!

> LibCurl Version : 7.21.0

We have made 13 releases since that, so it's not unthinkable that a newer
version works better or at least different. It would be very interesting to
learn if you see anything similar with a recent version.

> curl_easy_setopt(ptCurlHandle, CURLOPT_ERRORBUFFER, acErrorBuffer);
> curl_easy_setopt(ptCurlHandle, CURLOPT_URL, pstrURL);
> curl_easy_setopt(ptCurlHandle, CURLOPT_WRITEFUNCTION, VmwGetCB);
> curl_easy_setopt(ptCurlHandle, CURLOPT_WRITEDATA, (void *)&tContext);
> curl_easy_setopt(ptCurlHandle, CURLOPT_USERAGENT, pstrUserAgent);
> curl_easy_setopt(ptCurlHandle, CURLOPT_SSL_VERIFYPEER, 0L);
> curl_easy_setopt(ptCurlHandle, CURLOPT_SSL_VERIFYHOST, 0L);
> curl_easy_setopt(ptCurlHandle, CURLOPT_USERPWD, pstrUserPass);

If these are truly the only options you use then the connection: header in the
request is truly a mystery to me...

You could consider using CURLOPT_VERBOSE to get to see about libcurl's choices
to keep connections open or close them etc.

>> In which sense is this connection then still "alive" ? How do you see it
>> being kept in 30 minutes?
>
> When I run the netstat command from the command prompt, i can see many
> connection established. For every file there is one connection established.

Even ESTABLISHED! That sounds so completely broken. When libcurl concludes
that a connection can't be kept for further re-use it will _close_ that
socket. If it doesn't conclude that, it will keep the connection in its
connection pool (and possibly close an older one to make room) but that pool
will only fit a finite number of connections and that will be much less than
500 by default...

> All this connection will get closed after some time approximately 30 mins.

So who or what is closing them after 30 minutes?

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Received on 2012-06-14