On Wed, 3 Apr 2002, Rogerio Saran wrote:

> > The only thing that isn't very nice in this scenario, is that libcurl
> > sends away the POST data immediately, without waiting for the
> > 100-continue. But then again, it doesn't set the Expect: header either
> > and thus it is allowed to perform like this. (It just isn't very wise
> > when POSTing *huge* data that isn't accepted by the server end.)
> >
>
> What about a new option to tell libcurl to defer transfer for "POST"
> data? Maybe with 3 alternate behaviors:
>
> a) HTTP/1.0-compatible, send data within the http POST request

Using the Expect: header as RFC2616 describes it _is_ HTTP/1.0-compatible,
since it times out the waiting. Of course that makes the POST slower.

But since we have an option to set HTTP/1.0 operations, it wouldn't be that
hard to just switch off the header during that condition.

> b) HTTP/1.1 with "Expect:", defer sending until the
> 100-continue response comes

> c) HTTP/1.1 "autosensing", send data with the request if
> the POST data fits a small packet, and otherwise
> use deferred sends.

This could make sense, yes. But I don't think the "autosensing" should try to
figure out if fits in a small package or not, but instead ask "what is the
maximum POST size to send when risking a reject after the send isn't such a
big deal".

> I will try to work a patch if people tell me this makes sense.

Please do it against the CVS version, if you decide to do this. I've modified
the actual sending of the HTTP request since the 7.9.5 release...

> Maybe it is not look as bad as I stated. Inspecting LWP code seems it
> handle the "Connection: close" response correctly. However it is still way
> faster than libcurl handling subsequent requests.
>
> My next tests will sniff the ssl handshake to check whether ssl sessions
> are kept (by accident? on purpose?) by LWP.

Well, libcurl also keeps SSL sessions and re-uses them on subsequent connects
so they should be equally fast on those terms.

I'm puzzled. It'll be interesting to see what more you can dig up here!

> Just a head-up on the test case: I tested 24 content extraction robots both
> with LWP and libcurl. Most of them are home banking services, and the robot
> crawl for some 4, 8 pages to gather all the requested information.
>
> Changes in total time for a given robot
>
> more than 50% less 4 sites
> 30% to 50% less 6 sites
> 10% a 30% less 5 sites
> up to 10% less 4 sites
> 10% increase 2 sites
> 10% to 30% increase 2 sites
> more than 30% increase 1 site
>
> So 5 in 24 robots presented a worse performance with libcurl. Now we are
> checking every significant factor to explain this, and later I will post
> here our findings.

Just let us know if there's anything we can do to help out.

I'm still thrilled that 15 tests (62%) showed more than 10% speed
improvement.

Have you found any particular things/factors that the five performance hogs
have in common?

-- 
    Daniel Stenberg -- curl groks URLs -- http://curl.haxx.se/