>When libcurl starts a new transfer and you let it do HTTP/2, it *will*
reuse
>existing HTTP/2 connections and multiplex over that - if one is available
at
>the time and it has room for a new stream.

>If it doesn't have any existing HTTP/2 to add itself to, it will create a
new
>connection instead *unless* you have PIPEWAIT set for it as then it will
wait
>for other connections to the same host to first check if those can be
>multiplexed over and then it can use that connection instead.

I was verifying this with local echo server (120.0.0.1) so the response for
pretty quick.
Here I noticed that same connection was reuse. Later I added some(2 second)
delay in responding to requests
at server and I noticed the same behavior.

Changes done at http2-download.c
1. Just after multi_handle is initialized
     curl_multi_setopt(multi_handle, CURLMOPT_MAX_HOST_CONNECTIONS, 50);
     curl_multi_setopt(multi_handle, CURLMOPT_MAX_TOTAL_CONNECTIONS, 100);
2. In setup function
     curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION,
CURL_HTTP_VERSION_2_PRIOR_KNOWLEDGE);
       if (num == 1) {
         curl_easy_setopt(hnd, CURLOPT_STREAM_WEIGHT, 20);
         curl_easy_setopt(hnd, CURLOPT_STREAM_DEPENDS, curl_hnd[0]);
      }
      if (num == 2) {
         curl_easy_setopt(hnd, CURLOPT_STREAM_WEIGHT, 80);
         curl_easy_setopt(hnd, CURLOPT_STREAM_DEPENDS, curl_hnd[0]);
      }
     #if (CURLPIPE_MULTIPLEX > 0) //CURLPIPE_MULTIPLEX is set by default
to value 2
     /* wait for pipe connection to confirm */
        curl_easy_setopt(hnd, CURLOPT_PIPEWAIT, 1L);
    #endif

Noticed that new connection was created for each new HTTP/2 transfer here
with following print:
== 0 Info: Expire in 0 ms for 6 (transfer 0x16a1ce0)
== 1 Info: Expire in 0 ms for 6 (transfer 0x16a7520)
== 2 Info: Expire in 0 ms for 6 (transfer 0x16acd60)
CURLPIPE_MULTIPLEX is :[2]
== 0 Info: Trying 127.0.0.1...
== 0 Info: TCP_NODELAY set
== 0 Info: Expire in 200 ms for 4 (transfer 0x16a1ce0)
== 0 Info: - Conn 0 (0x16b25a0) send_pipe: 1, recv_pipe: 0
== 0 Info: Connected to 127.0.0.1 (127.0.0.1) port 9001 (#0)
== 0 Info: Using HTTP2, server supports multi-use
== 0 Info: Connection state changed (HTTP/2 confirmed)
== 0 Info: Copying HTTP/2 data in stream buffer to connection buffer after
upgrade: len=0
== 0 Info: Using Stream ID: 1 (easy handle 0x16a1ce0)
0 => Send header, 57 bytes (0x39)
0000: GET /hello2 HTTP/2
0014: Host: 127.0.0.1:9001
002a: Accept: */*
0037:
== 0 Info: Connection state changed (MAX_CONCURRENT_STREAMS == 200)!
== 1 Info: Found bundle for host 127.0.0.1: 0x169c240 [can multiplex]
== 1 Info: Conn: 0 (0x16b25a0) Receive pipe weight: (-1/0), penalized: FALSE
== 1 Info: Multiplexed connection found!
== 1 Info: Found connection 0, with requests in the pipe (1)
== 1 Info: We can reuse, but we want a new connection anyway
== 1 Info: Hostname 127.0.0.1 was found in DNS cache
== 1 Info: Trying 127.0.0.1...
== 1 Info: TCP_NODELAY set
== 1 Info: Expire in 200 ms for 4 (transfer 0x16a7520)
== 1 Info: - Conn 0 (0x16b25a0) send_pipe: 0, recv_pipe: 1
== 1 Info: - Conn 1 (0x16bff60) send_pipe: 1, recv_pipe: 0
== 1 Info: Connected to 127.0.0.1 (127.0.0.1) port 9001 (#1)
== 1 Info: Using HTTP2, server supports multi-use
== 1 Info: Connection state changed (HTTP/2 confirmed)
== 1 Info: Copying HTTP/2 data in stream buffer to connection buffer after
upgrade: len=0
== 1 Info: Failed sending HTTP request
== 1 Info: Connection #1 to host 127.0.0.1 left intact
== 2 Info: Found bundle for host 127.0.0.1: 0x169c240 [can multiplex]
== 2 Info: Conn: 0 (0x16b25a0) Receive pipe weight: (-1/0), penalized: FALSE
== 2 Info: Multiplexed connection found!
== 2 Info: Found connection 0, with requests in the pipe (1)
== 2 Info: We can reuse, but we want a new connection anyway
== 2 Info: Hostname 127.0.0.1 was found in DNS cache
== 2 Info: Trying 127.0.0.1...
== 2 Info: TCP_NODELAY set
== 2 Info: Expire in 200 ms for 4 (transfer 0x16acd60)
== 2 Info: - Conn 0 (0x16b25a0) send_pipe: 0, recv_pipe: 1
== 2 Info: - Conn 1 (0x16bff60) send_pipe: 0, recv_pipe: 0
== 2 Info: - Conn 2 (0x16cdfe0) send_pipe: 1, recv_pipe: 0
== 2 Info: Connected to 127.0.0.1 (127.0.0.1) port 9001 (#2)

Later I just made following change at client(server still same with 2
second delay):
1. curl_multi_setopt(multi_handle, CURLMOPT_MAX_HOST_CONNECTIONS, 1);

Observed that connection was being re-used.
== 0 Info: Expire in 0 ms for 6 (transfer 0x170cce0)
== 1 Info: Expire in 0 ms for 6 (transfer 0x1712520)
== 2 Info: Expire in 0 ms for 6 (transfer 0x1717d60)
CURLPIPE_MULTIPLEX is :[2]
== 0 Info: Trying 127.0.0.1...
== 0 Info: TCP_NODELAY set
== 0 Info: Expire in 200 ms for 4 (transfer 0x170cce0)
== 0 Info: - Conn 0 (0x171d5a0) send_pipe: 1, recv_pipe: 0
== 0 Info: Connected to 127.0.0.1 (127.0.0.1) port 9001 (#0)
== 0 Info: Using HTTP2, server supports multi-use
== 0 Info: Connection state changed (HTTP/2 confirmed)
== 0 Info: Copying HTTP/2 data in stream buffer to connection buffer after
upgrade: len=0
== 0 Info: Using Stream ID: 1 (easy handle 0x170cce0)
0 => Send header, 57 bytes (0x39)
0000: GET /hello2 HTTP/2
0014: Host: 127.0.0.1:9001
002a: Accept: */*
0037:
== 0 Info: Connection state changed (MAX_CONCURRENT_STREAMS == 200)!
== 1 Info: Found bundle for host 127.0.0.1: 0x1707240 [can multiplex]
== 1 Info: Conn: 0 (0x171d5a0) Receive pipe weight: (-1/0), penalized: FALSE
== 1 Info: Multiplexed connection found!
== 1 Info: Found connection 0, with requests in the pipe (1)
== 1 Info: Re-using existing connection! (#0) with host 127.0.0.1
== 1 Info: - Conn 0 (0x171d5a0) send_pipe: 1, recv_pipe: 1
== 1 Info: Using Stream ID: 3 (easy handle 0x1712520)
1 => Send header, 57 bytes (0x39)
0000: GET /hello2 HTTP/2
0014: Host: 127.0.0.1:9001
002a: Accept: */*
0037:
== 2 Info: Found bundle for host 127.0.0.1: 0x1707240 [can multiplex]
== 2 Info: Conn: 0 (0x171d5a0) Receive pipe weight: (-1/0), penalized: FALSE
== 2 Info: Multiplexed connection found!
== 2 Info: Found connection 0, with requests in the pipe (2)
== 2 Info: Re-using existing connection! (#0) with host 127.0.0.1
== 2 Info: - Conn 0 (0x171d5a0) send_pipe: 1, recv_pipe: 2
== 2 Info: Using Stream ID: 5 (easy handle 0x1717d60)
2 => Send header, 57 bytes (0x39)
0000: GET /hello2 HTTP/2
0014: Host: 127.0.0.1:9001
002a: Accept: */*
0037:

Not sure, what I'm still missing here w.r.t observation.

> The HEADER frame seems to send the dependency properly, but later
PRIORITY
> is sent for same stream id but the dependency is 0.

>A bug?

Not sure if this can be treated as bug as per HTTP/2 specs but I could
trace when this happens.
As mentioned earlier my server was local so the response was quick. So when
libcurl sent out
stream 1 it got response immediately. Later when it sent stream 3, as
stream 1 was already processed,
it put dependent stream as 0 instead of 1. When I tried with delay at
server, noticed that stream dependency was
set correctly to 1. I was also noticing that libcurl is always sending
HEADER streams in sequential manner.
So, if 3 requests were initiated, HEADER(1) would be sent in one request (1
transfer as seen in pcap), then HEADER(3)
and then HEADER(7). Whereas if I try nghttp2 client it shows in single
transfer - like HEADER(1), HEADER(3), HEADER(7).
May be this is related on when & how socket is flushed out. The only
concern with this is that if the server is fast enough,
it will process HEADER(1) fast and reply by the time client sends HEADER(3)
and dependency information getting reset.

Thanks

On Thu, Mar 21, 2019 at 7:48 PM Daniel Stenberg <daniel_at_haxx.se> wrote:

> On Wed, 20 Mar 2019, Kunal Ekawde wrote:
>
> > Then in case of HTTP/1.1 transfer, we want to have
> > CURLMOPT_MAX_TOTAL_CONNECTIONS = 1000, CURLMOPT_MAX_HOST_CONNECTIONS =
> 100.
> > If we run the HTTP/2 transfer with these set, libcurl opens a new
> connection
> > instead of multiplexing the next stream over earlier connection
>
> As I've explained to Arnaud, it is not that simple and this statement is
> simply wrong or at best incomplete.
>
> When libcurl starts a new transfer and you let it do HTTP/2, it *will*
> reuse
> existing HTTP/2 connections and multiplex over that - if one is available
> at
> the time and it has room for a new stream.
>
> If it doesn't have any existing HTTP/2 to add itself to, it will create a
> new
> connection instead *unless* you have PIPEWAIT set for it as then it will
> wait
> for other connections to the same host to first check if those can be
> multiplexed over and then it can use that connection instead.
>
> So, it all depends. libcurl is not designed to maximize multiplexing, it
> is
> made to perform the transfers you ask it to perform - when you ask it.
>
> > Even with normal dependency (not as in case#1) wherein for 3
> requests(stream
> > 1, 3, 5) -- 3(weight 20) depends on 1, 5(weight=80) depends on 1, the
> server
> > is not showing consistent behavior of processing the requests.
>
> ...
>
> > The HEADER frame seems to send the dependency properly, but later
> PRIORITY
> > is sent for same stream id but the dependency is 0.
>
> A bug?
>
> --
>
> / daniel.haxx.se
>

-- 
~Kunal

-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2019-03-25