On Thu, 10 Oct 2002, Paul Koch wrote:

> I am building an application which requires to perform many (eg. 100+)
> simultaneous HTTP requests to many remote servers to retrieve data. Client
> and servers all run FreeBSD.

Are you sure you can do this using threads on FreeBSD?

I'm certainly not an expert on FreeBSD or anything, but I've seen people
experience problems with multi-threading and curl under *BSD one time too
many so thus I'm asking.

libcurl works fine in multi-threaded use, as long as the thread
implementation is good and solid. The *BSD threads have a history of not
being that good and solid.

> Is 100+ simultaneous requests/connections an issue for libcurl ?
> Does it have a limit ?

The issues that you might face are not imposed by libcurl itself (at least I
can't think of any particular ones), but by the system itself. I know people
have faced problems with the maximum amount of open file descriptors, or if
you direct too many of those simmultanous operations against one single
server, that server might start rejecting connections or generally not feel
good.

> The manual page defines "CURLOPT_MAXCONNECTS". Is this relevent ? The
> manual says not to change it unless you are aware of how it works. Should I
> and can I set it to 150 ?

Well it might be relevant, but not in the way you suggest here. MAXCONNECTS
is the maximum amount of live connections a single curl handle might keep.
Thus, if you have 100 handles and MAXCONNECTS is 5, there's a theoretical
maximum of 500 simultaneous connections kept alive.

Each curl handle deals with its own "persistency" and "connection re-use" and
that's what the MAXCONNECTS is for. The handle-specific connection cache.

If you plan to use such a massive amount of threads, I figure it would make
sense to *lower* MAXCONNECTS so that you'll get a smaller max amount of open
connections.

A future version of libcurl might allow the different threads to share a
connection pool for this (using the so called 'share system').

-- 
 Daniel Stenberg -- curl related mails on curl related mailing lists please
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf