> > Setting CURLMOPT_PIPELINING on my CURLM handle causes problems for me when I
> > use digest authentication (7.26.0, C++, Win64).
>
> I bet this is due to https://sourceforge.net/tracker/?func=detail&
> aid=3545398&group_id=976&atid=100976, which I just found today.
>
> The description is incomplete. There's ALL SORTS of auth-related state
> that isn't cleared between requests. (I'm going to gather more data and
> update the bug with test cases tomorrow.) It wouldn't surprise me at all

Excellent!

> You could use CURLOPT_FRESH_CONNECT on each curl handle that uses digest auth.

Unfortunatelly all content coming from this server is auth'd so re-connecting & re-authing every time would be a huge step backwards for performance I would expect.

Thanks for the pointer, though -- I wasn't aware of that option.

Incidentally, my tests with (unathenticated) pipelining were pretty disappointing. My test case is 7,400 small files totalling 675MB (some ~1MB, many ~ 1KB). My benchmarks were on a gigabit LAN (ie: the most useless case for pipelining) but even there the difference was striking.

Rough data for posterity:

60.6 seconds, max 8 pipelined requests at once, one connection
13.9 seconds, max 8 simultaneous requests in single CURLM

If I forked 8 threads myself and used pipelining I got comperable timings to 8/CURLM but it's kind of ham-fisted. I'd
really hoped that "one pipe to rule them all" would be killer but maybe I'm doing it wrong.

I did some WAN tests with a 60ms ping but pipelining was clearly worse so I didn't sit around to run the whole suite to get exact numbers -- it wouldn't be practical for me right now over WAN because of the authentication problem anyway.

-g
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2012-07-19