Daniel Stenberg wrote:
>>There is one complete POST request, with its corresponding post data.
>>Then the server returned two responses... one '100-continue' as it would be
>>expecting post data (already sent) and then a '200-ok' with a 'Connection:
>>close' header.
>>
>>It seems an odd behavior from this server (and correct behavior for
>>libcurl.)
>>
>
> The only thing that isn't very nice in this scenario, is that libcurl sends
> away the POST data immediately, without waiting for the 100-continue. But
> then again, it doesn't set the Expect: header either and thus it is allowed
> to perform like this. (It just isn't very wise when POSTing *huge* data that
> isn't accepted by the server end.)
>

What about a new option to tell libcurl to defer transfer for "POST"
data? Maybe with 3 alternate behaviors:

        a) HTTP/1.0-compatible, send data within the http POST request
        b) HTTP/1.1 with "Expect:", defer sending until the
           100-continue response comes
        c) HTTP/1.1 "autosensing", send data with the request if
           the POST data fits a small packet, and otherwise
           use deferred sends.

I will try to work a patch if people tell me this makes sense.

>

[snip]

>
> "Connection: Keep-Alive" shouldn't mean anything when talking HTTP 1.1.
> Connections are always kept alive unless specificly set to "Connection:
> close".
>
>
>>It seems the native LWP http client ignores the "Connection: close"
>>header, and keep using the connection while it is open. The IIS server does
>>not care about its own response and keep the connection open.
>>
>
> Ugha.

Maybe it is not look as bad as I stated. Inspecting LWP code seems it
handle the "Connection: close" response correctly. However it is still
way faster than libcurl handling subsequent requests.

My next tests will sniff the ssl handshake to check whether ssl sessions
  are kept (by accident? on purpose?) by LWP.

[snip]

>>
>
> First, I'd like to see you report success that this fix actually makes your
> application do good. Then, I'd prefer to figure out why this happens. If this
> is a IIS flaw, then we might *have to* address this problem in libcurl. Or
> even if it is't a flaw, it might be frequent enough to force us to allow this
> work-around for libcurl users/developers.
>
> It would also help to hear how the LWP guys have solved this or how come they
> don't close the connection even though they get a connection: close?
>
> So, yes, we would be interested in that patch of yours! ;-) Please just post
> it here and we can all digest it as we see fit!

It will be available soon (I am still fighting with the ssl traffic
sniffer.) I also hope it will br a good improvement to the library.

Just a head-up on the test case: I tested 24 content extraction robots
both with LWP and libcurl. Most of them are home banking services, and
the robot crawl for some 4, 8 pages to gather all the requested information.

Changes in total time for a given robot

more than 50% less 4 sites
30% to 50% less 6 sites
10% a 30% less 5 sites
up to 10% less 4 sites
10% increase 2 sites
10% to 30% increase 2 sites
more than 30% increase 1 site

So 5 in 24 robots presented a worse performance with libcurl. Now we are
checking every significant factor to explain this, and later I will post
here our findings.

*Saran
Received on 2002-04-04