On Wed, 14 Sep 2011, Martin J. Evans wrote:

> I am using libcurl under Perl's WWW::Curl::Easy module because it works out
> a lot faster than the mostly pure Perl LWP module. For HTTP GETs Curl works
> out a lot faster but with small POSTs Curl works out a lot slower (in real
> time although faster in CPU time).

Thanks a lot for these interesting results. Let's try to figure out why they
look the way they do!

I think you need to do some wiresharking of the TCP traffic during the tests
to properly understand what's going on.

> Through experimentation I have discovered setting either CURLOPT_TCP_NODELAY
> or CURLOPT_FORBID_REUSE speeds up the POSTs so Curl is faster than LWP.

Have you tried with _only_ CURLOPT_TCP_NODELAY ? And in the other end, do you
know if LWP disables the Nagle algorithm and does it re-use the same
connection or not?

> I was a little surprised setting CURLOPT_TCP_NOLDELAY made a difference but
> as it does it sort of suggested to me that maybe the POST headers are sent
> first then followed up by the form data (i.e., at least 2 writes to the
> socket).

Without NODELAY I would rather suspect that you'll see a short wait when a
single request has been sent and the Nagle waits a short while to see if you
send more data to merge into the same packet. When it doesn't come, it sends
off the request anyway.

> Also, if CURLOPT_FORBID_REUSE is set I suppose it is possible libcurl does a
> shutdown(write) on the socket after the last write which would also expedite
> any unset data.

Did you try with _only_ CURLOPT_FORBID_REUSE set and not NODELAY then? I can't
recall any shutdown(write) done by libcurl...

> When I strace the code I do in fact see 2 separate writes to the socket for
> the header and the form data.

Oh right, because you use multipart formpost. If you'd change your test to
instead use a "normal" small POST you'll see that libcurl won't do that. It
might make sense to change the test to use that. It should make the speed very
similar to a plain GET test.

The reason multipart formpost causes two (or more) writes instead of a single
one, even when the post is very small, is mostly due to the layout of the
internal APIs and convenience/laziness. It could very well be optimized to do
better...

> Ideally I'd like to reuse the socket but I am wary of setting
> CURLOPT_TCP_NOLDELAY.

For this kind of use case, I'd say using CURLOPT_TCP_NODELAY makes sense. Of
course, in a real life application you might not send so small posts and you
might not fire them off that rapidly immediately next after each other I
guess...

> If libcurl breaks the post header and the form data into 2 writes why?

You'll have to see the lib/formdata.c and how the data is presented to the
libcurl uploader for that to be clear. I wouldn't mind having libcurl
optimized to prefer a single write() for small multipart formposts, but I'm
not sure I'm willing to waste a lot of time or code readability for it. It's
really a corner case.

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html