cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: libcurl 7.9.8: Bottleneck on multhithread HTTP fetches?

From: Andrew Fuller <andrewfuller2k2_at_yahoo.co.uk>
Date: Wed, 15 Oct 2003 11:50:54 +0100 (BST)

Wow,

I tried your "somewhat ugly" workaround for windows -
ie I took out the socketerror() test if connect()
returns zero and it runs perfectly. Very quick, and
no CPU maxout, even with 400 simultaneous hits.

Do you think this is a reasonable workaround? The
MSDN link you sent seems quite clear-cut : a 0
returned from connect() indicates the connection was
made.

FYI here's the libcurl code I modified for this test
(starts line 685 in connect.c, libcurl 7.10.7) :-

    if(0 == rc) {
#ifdef WIN32
                // win32 : presume 0 means truly connected
                *connected = TRUE;
                break;
#else
      int err = socketerror(sockfd);
      if ((0 == err) || (EISCONN == err)) {
        /* we are connected, awesome! */
        *connected = TRUE; /* this is a true connect
*/
        break;
      }
      /* nope, not connected for real */
      rc = -1;
#endif // WIN32
    }

Any problems with this?

Thanks lots for your help so far!

rgds,
Andrew Fuller

--- Daniel Stenberg <daniel_at_haxx.se> wrote: > On Tue,
14 Oct 2003, Andrew Fuller wrote:
>
> > By the way, if it's any use, this time the longest
> path (timewise) according
> > to Rational Quantify was :
> >
> >
>
curl_easy_perform->Curl_perform->Curl_connect->SetupConnection->ConnectPlease->Curl_connecthost->socketerror->getsockopt->EnterCriticalSection
> >
> > Next longest was the same to above except ended :
> >
> > ...->getsockopt->NtRemoveIoCompletion
>
> Thanks for the provided info. Here's my take on
> analyzing it:
>
> The VERBOSE output log (from your previous mail) is
> a mess to read and really
> doesn't help much since all the different threads'
> output mix in a seemingly
> random way. You would need to prefix all output
> lines with a thread id/number
> or something for it to start becoming useful. (Thus
> you'd need to use
> DEBUGFUNCTION.)
>
> How many threads do you need to get this problem?
> You used 80 here, but what's
> the minimum amount? What's your network connection
> to the test server?
>
> We could try to learn from history. See what Chicco
> wrote back in 2002:
> http://curl.haxx.se/mail/lib-2002-02/0057.html
>
> Bjorn Reese filled in some more info on why libcurl
> does this:
> http://curl.haxx.se/mail/lib-2002-02/0058.html
>
> The relevant question here and now is of course: can
> we modify this behavior
> on Windows to make it run better? Chicco said that
> he didn't need to invoke
> 'socketerror' if connect() returns 0. Perhaps this
> is always true on windows?
> This page on microsoft.com seems to claim this:
>
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winsock/winsock/connect_2.asp
>
> I don't use windows enough myself to try this out.
>
> --
> Daniel Stenberg -- curl: been grokking URLs since
> 1998
>
>
>
-------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback
> Program.
> SourceForge.net hosts over 70,000 Open Source
> Projects.
> See the people who have HELPED US provide better
> services:
> Click here: http://sourceforge.net/supporters.php
>

________________________________________________________________________
Want to chat instantly with your online friends? Get the FREE Yahoo!
Messenger http://mail.messenger.yahoo.co.uk

-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
Received on 2003-10-15