Re: Connection re-use w.r.t HTTP/1.1 and HTTP/2.0

On Mon, Dec 16, 2019 at 6:17 PM Daniel Stenberg <daniel_at_haxx.se> wrote:
>
> On Mon, 16 Dec 2019, Kunal Ekawde via curl-library wrote:
>
> > Connection re-use has been discussed earlier in some context but I wanted to
> > know more with respect to recent modifications.
>
> I don't think there have been any recent modifications that changed connection
> reuse.
>
> > curl_multi_setopt(multi_handle_m, CURLMOPT_MAXCONNECTS, maxConnects);
>
> This is the size of the multi handle's connection cache. Basically that is the
> max number of connections that can be kept alive after a request has completed
> - but it should be noted that also running connections are part of the cache
> so if MAXCONNECTS is smaller than the number of active transfers, there will
> be no idle connections left in the cache after a transfer (thus effectively
> killing connection reuse).
>
> > curl_multi_setopt(multi_handle_m, CURLMOPT_MAX_HOST_CONNECTIONS,
> > maxConnectsPerHost);
>
> This limits the number of connections this handle will use to the same host
> name.
>
> > curl_multi_setopt(multi_handle_m, CURLMOPT_MAX_TOTAL_CONNECTIONS,
> > maxParallelConnects);
>
> This limits the total number of *active* connections the multi handle will
> use. Attempting to create more, with additional transfers, will put those
> transfer in an internal waiting queue, waiting for a connection "slot" to
> become available.
>
> > curl_multi_setopt(multi_handle_m, CURLMOPT_PIPELINING, CURLPIPE_MULTIPLEX);
>
> This is the default since 7.62.0, it enables multiplexing if curl finds that
> it can. Like when doing HTTP/2 to the same host with two concurrent requests.
>
> > For each transfer:
> > curl_easy_setopt(easy_handle, CURLOPT_PIPEWAIT, 1L);
>
> This makes transfers prefer to (wait and) multiplex on an existing connection
> rather than immediately start a separate new connection, if that situation
> arises.
>
> > Desired behavior in HTTP/2 for me would be to re-use the same connection
> > until some limit on requests on a connection,
>
> Why would you limit this to something else than the underlying maximum number
> of streams allowed? The limit we provide for that is the one *you* implemented
> and is provided since 7.67.0: CURLMOPT_MAX_CONCURRENT_STREAMS.
>
> > something like may be max concurrent streams and only then create new
> > connection based on 'CURLMOPT_MAX_HOST_CONNECTIONS'.
>
> Right: CURLMOPT_MAX_CONCURRENT_STREAMS.
>

Yes, this is desired behavior, basically that the connection is fully
utilized to its multiplexing limit
before new one is created towards that host, but I don't see it,
rather, it creates new connection to
host for a new stream if 'CURLMOPT_MAX_HOST_CONNECTIONS' limit not
reached yet. e.g.

Given that: CURLMOPT_MAX_HOST_CONNECTIONS = 100
Request 1 -> Stream1 -- New connection #0
Request 2 -> Stream2 -- New connection #1

Desired:
Request 1 -> Stream1 -- New connection #0
Request 2 -> Stream2 -- Re-use connection #0

> > Current observation is that each new request/transfer starts a new
> > connection upto 'CURLMOPT_MAX_HOST_CONNECTIONS' and then re-uses
> > those.
>
> Yes exactly. Isn't that also what the documentation says?
>
> > For HTTP/1.1 desired would be to pipe the requests upto certain limits and
> > then create another connection.
>
> libcurl doesn't support Pipelining (anymore) so it will never pipeline, but it
> will reuse connections whenever it can.
>
> --
>
> / daniel.haxx.se | Get the best commercial curl support there is - from me
> | Private help, bug fixes, support, ports, new features
> | https://www.wolfssl.com/contact/

-- 
~Kunal
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html