Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Pipelining in HTTP 1.1
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Daniel Stenberg via curl-library <curl-library_at_cool.haxx.se>
Date: Thu, 1 Oct 2020 23:25:36 +0200 (CEST)
On Thu, 1 Oct 2020, Dorit Mari via curl-library wrote:
> I know that support for pipelining in HTTP 1.1 was removed
> (https://github.com/curl/curl/commit/2f44e94efb3df8e50bb2ddbc4ec6b569a6424517).
> In our application, we need to use pipelining in HTTP 1.1, so we are trying
> to restore and fix the pipelining support in libcurl.
Wow! That's indeed a worthy task.
> In the commit that removed pipelining, it's written below that "pipelining
> code had a few rather nasty known issues". Does anyone know what were these
> issues, and what part of the code caused them?
I can recall two specific problems, but they were both really mostly the
result of the primary problem: we didn't have test cases that verfied the
basic functionality nor the complicated scenarios. This made several little
issues people saw with pipelining really hard to address or work on since
pipelining becomes such timing dependent and "racy" functionality.
To the best of my recollection:
Case 1: - removing the middle transfer
Fire off three requests to the pipeline, they all go off and you start getting
contents back for the first request. Since each transfer has its own easy
handle, they're all individually controlled by the application. Now, the
application gets cold feet and removes the second (middle) handle from the
multi stack - before it has even got any response back or perhaps while
getting it. The third one remains, still awaiting its contents... This case
was not handled at all by libcurl.
Case 2: - retrying on broken connections
Add four easy handles to a multi handle that all can pipeline. All four HTTP
requests go away over the connection. The first response comes in, and the
second response starts to arrive when the connection is dropped. This should
make the client create a new connection to resend the third and forth requests
over.
> Currently, we "patched" the old pipelining code on a recent libcurl version.
> Under low load it works OK, but under high load with TLS (e.g. 300 HTTP
> requests per second) we encounter problems; requests time out, and
> afterwards libcurl keeps opening and closing connections repeatedly, for no
> clear reason. Is this one of the known problems that led to removing the
> pipelining code?
I don't recognize that specific description, but the details I mentioned above
together with your problem here are certainly signs of a too fragile
implementation in my opinion.
Date: Thu, 1 Oct 2020 23:25:36 +0200 (CEST)
On Thu, 1 Oct 2020, Dorit Mari via curl-library wrote:
> I know that support for pipelining in HTTP 1.1 was removed
> (https://github.com/curl/curl/commit/2f44e94efb3df8e50bb2ddbc4ec6b569a6424517).
> In our application, we need to use pipelining in HTTP 1.1, so we are trying
> to restore and fix the pipelining support in libcurl.
Wow! That's indeed a worthy task.
> In the commit that removed pipelining, it's written below that "pipelining
> code had a few rather nasty known issues". Does anyone know what were these
> issues, and what part of the code caused them?
I can recall two specific problems, but they were both really mostly the
result of the primary problem: we didn't have test cases that verfied the
basic functionality nor the complicated scenarios. This made several little
issues people saw with pipelining really hard to address or work on since
pipelining becomes such timing dependent and "racy" functionality.
To the best of my recollection:
Case 1: - removing the middle transfer
Fire off three requests to the pipeline, they all go off and you start getting
contents back for the first request. Since each transfer has its own easy
handle, they're all individually controlled by the application. Now, the
application gets cold feet and removes the second (middle) handle from the
multi stack - before it has even got any response back or perhaps while
getting it. The third one remains, still awaiting its contents... This case
was not handled at all by libcurl.
Case 2: - retrying on broken connections
Add four easy handles to a multi handle that all can pipeline. All four HTTP
requests go away over the connection. The first response comes in, and the
second response starts to arrive when the connection is dropped. This should
make the client create a new connection to resend the third and forth requests
over.
> Currently, we "patched" the old pipelining code on a recent libcurl version.
> Under low load it works OK, but under high load with TLS (e.g. 300 HTTP
> requests per second) we encounter problems; requests time out, and
> afterwards libcurl keeps opening and closing connections repeatedly, for no
> clear reason. Is this one of the known problems that led to removing the
> pipelining code?
I don't recognize that specific description, but the details I mentioned above
together with your problem here are certainly signs of a too fragile
implementation in my opinion.
--
/ daniel.haxx.se | Commercial curl support up to 24x7 is available!
| Private help, bug fixes, support, ports, new features
| https://www.wolfssl.com/contact/
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2020-10-01