If what you are saying is what it's really doing, and no one else suggests
anything, you might also try going multithreaded.

Libcurl is thread safe with a few caveats (check out this page for more info
http://curl.haxx.se/libcurl/features.html)

On Mon, Jun 6, 2011 at 11:30 AM, Jimish Shah <jimishrshah_at_gmail.com> wrote:

> Hi,
>
> I am trying to use libcurl to scale to several thousand https connections
> to an external web server. I have downloaded and investigated the source
> code, so i have a fair understanding of the function calls.
>
> If i use multi-handle and add say 1000 easy handles to it. If each of the
> thousand easy handles need to connect simultaneously , i noticed that there
> is a blocking "select ()" during the connect phase as explained below.
>
> Curl_multi_perform() --> multi_runsingle() --> Curl_connect() ....... -->
> Singleipconnect --> waitconnect() --> Curl_socket_ready()* --> select ()* every
> one second until error or success or timeout
>
> In other words, although the socket itself is non-blocking, this function
> call waits (multi_runsingle() loops until error or success) until the single
> connect on one easy handle either succeeds or fails completely or times-out,
> which makes it somewhat pipelined if the responding web-server is slow (in
> which case we will repeatedly get EINPROGRESS on the socket). Also the other
> 999 easy handle connect are waiting till this one is done. Hence its
> serialized in some manner.
>
> Is there a way wherein libcurl allows to open all the 1000 connections
> using some type of a event driven mechanism or some alternate to change the
> connection state as and when they are successfull / fail.
>
> Pls correct me if any of my understanding is incorrect. Also i went through
> the mailer, but couldnt find something that answered my questions.
>
> Pls let me know if i can clarify further. Thanks in advance for helping me
> out.
>
> ~ Jimish
>
> “Perseverance is not a long race; it is many short races one after the
> other” - Walter Elliot
>
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/list/listinfo/curl-library
> Etiquette: http://curl.haxx.se/mail/etiquette.html
>

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2011-06-06