curl-library
RE: Massive HTTP/2 parallel requests
Date: Wed, 2 Mar 2016 15:21:36 +0000
> The results of curl —version are:
>
> curl 7.47.1 (x86_64-apple-darwin15.3.0) libcurl/7.47.1 OpenSSL/1.0.2f
> zlib/1.2.5 nghttp2/1.7.0
> Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3
> pop3s rtsp smb smbs smtp smtps telnet tftp
> Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets
>
> Concerning the server, I’m running an apache with mod_http2 enabled and
> the following HTTP/2-related configuration:
>
> LoadModule http2_module modules/mod_http2.so <IfModule
> http2_module>
> Protocols h2c http/1.1
> LogLevel http2:info
> </IfModule>
>
> I performed this very morning another test adding the following line:
> H2MaxSessionStreams 1000000
> I was just to make sure that the number of streams is higher than the
> number of parallel downloads for my test (1024). I realised looking at the logs
> that all my requests fit in the pipe, but still files are downloaded in 64-stream
> blocks! If I try with normal HTPP/1.1 connections the total test is around 20
> times faster than this way.
>
> On a second test I set the following parameters:
> curl_multi_setopt(multi_handle, CURLMOPT_MAX_TOTAL_CONNECTIONS,
> 10);
> curl_multi_setopt(multi_handle, CURLMOPT_MAX_HOST_CONNECTIONS,
> 10);
>
> And what happens is quite annoying: it opened the connection 0 for the file
> 1, and consecutive connections 1-9 for files 2-10. The rest 11-1024 where
> attached to connection 0, so that when connections 1-9 finished (with only
> one request) they were not reused.
>
> Any clues?
> José
>
Opening more than one HTTP/2 connection is a SHOULD NOT according to Section 9.1 of the HTTP/2 specification, so your client behaviour seems a bit out of keeping with what is expected.
Have you enabled HTTP/2 multiplexing in the single connection as described by CURLOPT_PIPELINING (https://curl.haxx.se/libcurl/c/CURLMOPT_PIPELINING.html). Doing so should limit the number of connections to 1, ignoring any value of CURLMOPT_MAX_HOST_CONNECTIONS.
To eliminate a server issue I would suggest testing the sever with h2load (https://nghttp2.org/documentation/h2load.1.html), an example of 1 client making 1024 concurrent requests is:
h2load -c1 -n1024 -m1024 http://{server}/{file}
Does it also show the 64-stream behaviour?
Lucas
-------------------------------------------------------------------
List admin: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2016-03-02