Well let's take this a little deeper.
I was just testing two versions, one with gnutls bound libcurl ssl calls in
each thread.

It gives me some vector resource nonsense around 450 threads with 768mb of
ram.
Keep in mind that error is certainly an out of memory situation and nothing
to do with my data sets.
Further to this it only gets those 450 threads initialized and can not
continuously operate them.

Now test number two consisted of a nearly identical application however with
out major difference.
I dropped out the ssl calls in favor of a standard http url to do the exact
same workload upon.
I was able to built up to 800 threads and maintain them consistently without
an issue.
This makes me highly suspect of atleast libcurl + gnutls's ssl memory
management and that
when used in a high thread count environment memory is being sacraficed at
some stage.

The 8mb default stack is a crash at 350-450 or stability at 800, when the
only difference is a well built ssl lib.
Tough call, any thoughts?

John

On Mon, Jun 29, 2009 at 12:54 PM, Phil Blundell <pb_at_reciva.com> wrote:

> On Mon, 2009-06-29 at 12:01 +0300, John D wrote:
> > The issue is how much ram curl it is requiring. I literally must
> > leave the default stack size within debian at 8mb.
> > So with 16gb of ram I am getting 1800-2000 threads. Any more and I
> > will encounter a seg fault in relation to thread resources.
> > A lower stack and it is curl's chance to seg fault.
>
> This certainly doesn't sound like the normal curl behaviour. I have a
> libcurl-based application running quite happily on a machine which has
> only 32MB RAM in total (and no swap). Admittedly this is only with a
> single thread, and maybe 10-20 active easy handles at any one time, but
> I'm fairly sure curl is not using 8MB of stack space.
>
> Whereabouts in curl does it segfault if you run it with a smaller stack?
>
> p.
>
>
>
Received on 2009-06-29