On Fri, Feb 21, 2003 at 12:21:11AM -0600, Ralph Mitchell wrote:
> For what it's worth, I tried it a bunch of times too, and always got back around
> 76Kb, with the /html close tag as the last line in every case. Download "current
> speed" at close varied from 142K to 355K... (based in Tulsa, probably going out
> via multiple T3's in Dallas :-)
>
> I can't help wondering why you're bothering to close the connection? When curl
> exits, your OS will shutdown the connection anyway... As far as I know, keeping
> the connection open is only meaningful if you're fetching more than one url from
> the same server during a single invocation of curl.
>
> A *browser* will benefit from keeping the connection open, because it will
> immediately turn around and grab images, javascript, style sheets, etc from the
> same server, after parsing the web page. Curl isn't even attempting to do that
> (and that's just fine with me, Daniel :-).
>

Hi Ralph,

Thanks for trying and confirming this problem.

Let me state my purpose here. I first encountered this problem using perl's
LWP package. Our fetching tool based on that always gives sporadic
timeouts for various urls. After some analysis I isolated the problem to
that by default lwp does not set keep-alive. Once that is set, all is well.

I am trying to understand what keep-alive does differently compared
to the case when it is not set. I suspected that there may be some
bugs in lwp package. So I fired up my favorite curl tool
and found out that, disabling keep-alive in curl will also
give sporadic timeouts.

So my purpose is to understand the cause of timeouts when keep-alive
is disabled. I have to say that curl is working perfectly fine by default.
I am just using it to understand a puzzling situation because curl
is a very mature tool without too much bugs.

However, my understanding of keep-alive is that it will make
multiple downloading more efficient. It should not affect the
downloading of a single page in any way. But now the result
I have seen squarely contradict my understanding.

I have used ethereal and snoop and suggested by Daniel. However,
the output is too long to be given here. The bottom line is that
when the client hangs, it is either due to client/server issuing
RST (rest) or FIN packets. What is odd is that when hanging happens,
'netstat -a' on client shows tcp still as 'ESTABLISHED' but
the server side shows nothing, meaning the server side already
closed the connection.

So for some reason, the client program does not know that the
server side had already closed the connection. This looks more
like UDP instead of TCP in that, some packets may get lost,
the client still expect more, but the server has already closed
the connection. The client is waiting in vain.

I know this is a little bit off topic in a curl mailinglist, but the
fact that it happens to curl means that the big user pool of curl
may offer me some ideas about what is going on. Perhaps there is
some implementation defect of the socket connections?

Thanks again for your interest and help.

Richard

-------------------------------------------------------
This SF.net email is sponsored by: SlickEdit Inc. Develop an edge.
The most comprehensive and flexible code editor you can use.
Code faster. C/C++, C#, Java, HTML, XML, many more. FREE 30-Day Trial.
www.slickedit.com/sourceforge
Received on 2003-02-21