Re: Connection re-use w.r.t HTTP/1.1 and HTTP/2.0

On Mon, 16 Dec 2019, Kunal Ekawde via curl-library wrote:

> Connection re-use has been discussed earlier in some context but I wanted to
> know more with respect to recent modifications.

I don't think there have been any recent modifications that changed connection
reuse.

> curl_multi_setopt(multi_handle_m, CURLMOPT_MAXCONNECTS, maxConnects);

This is the size of the multi handle's connection cache. Basically that is the
max number of connections that can be kept alive after a request has completed
- but it should be noted that also running connections are part of the cache
so if MAXCONNECTS is smaller than the number of active transfers, there will
be no idle connections left in the cache after a transfer (thus effectively
killing connection reuse).

> curl_multi_setopt(multi_handle_m, CURLMOPT_MAX_HOST_CONNECTIONS,
> maxConnectsPerHost);

This limits the number of connections this handle will use to the same host
name.

> curl_multi_setopt(multi_handle_m, CURLMOPT_MAX_TOTAL_CONNECTIONS,
> maxParallelConnects);

This limits the total number of *active* connections the multi handle will
use. Attempting to create more, with additional transfers, will put those
transfer in an internal waiting queue, waiting for a connection "slot" to
become available.

> curl_multi_setopt(multi_handle_m, CURLMOPT_PIPELINING, CURLPIPE_MULTIPLEX);

This is the default since 7.62.0, it enables multiplexing if curl finds that
it can. Like when doing HTTP/2 to the same host with two concurrent requests.

> For each transfer:
> curl_easy_setopt(easy_handle, CURLOPT_PIPEWAIT, 1L);

This makes transfers prefer to (wait and) multiplex on an existing connection
rather than immediately start a separate new connection, if that situation
arises.

> Desired behavior in HTTP/2 for me would be to re-use the same connection
> until some limit on requests on a connection,

Why would you limit this to something else than the underlying maximum number
of streams allowed? The limit we provide for that is the one *you* implemented
and is provided since 7.67.0: CURLMOPT_MAX_CONCURRENT_STREAMS.

> something like may be max concurrent streams and only then create new
> connection based on 'CURLMOPT_MAX_HOST_CONNECTIONS'.

Right: CURLMOPT_MAX_CONCURRENT_STREAMS.

> Current observation is that each new request/transfer starts a new
> connection upto 'CURLMOPT_MAX_HOST_CONNECTIONS' and then re-uses
> those.

Yes exactly. Isn't that also what the documentation says?

> For HTTP/1.1 desired would be to pipe the requests upto certain limits and
> then create another connection.

libcurl doesn't support Pipelining (anymore) so it will never pipeline, but it
will reuse connections whenever it can.

-- 
  / daniel.haxx.se | Get the best commercial curl support there is - from me
                   | Private help, bug fixes, support, ports, new features
                   | https://www.wolfssl.com/contact/
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html