On Mon, 6 Aug 2018, Radu Hociung wrote:

>> We got a fresh new HTTP pipelining bug submitted today [1]
>
> I am the submitter of that bug, and now that I understand it better have
> revised the issue report. It is NOT a pipelining bug, but a plumbing bug.

Right, quite ironic I think that it wasn't a pipelining bug that finally
triggered me to put pipelining up for deprecation...

But regarding that bug I disagree. It's not a bug at all, but quite simply a
misunderstanding of what the libcurl API (doesn't) guarantee.

> it looks like HTTP/1.1 pipelining is very well supported and tested,
> especially relative to other "features" that have crept into libcurl, that
> are far less well used and abused, like ssh, smtp, gopher, etc.

Why do you think they are "far less used" ? Is that because you think all the
users who claim they use pipelning are in fact telling the truth?

On what do you base the claim that pipelining in curl "is very well supported
and tested"? The test coverage is weak and cause intermittant failures that
make us have to disable several of them. The company who paid for the last
(latest?) big pipeline work ended up never deploying it in their application
due to its problems (unclear though if it was generic pipelining problems or
issues with our implementation in particular).

> It's worth noting that these people generated a bit of mailing list activity
> at the time they were working on their respective projects, but are not
> regular contributors.

We have *very few* regular contributors, so that's really the way the vast
majority of all issues are brought up and discussed here.

Several of those persons you mentioned idenfied problems or flaws with
pipelining, several of those were never fixed. As we don't get them reported
more I take as an indication that not enough users are using the feature to
hit them that often (to report them).

> I think this is typical of a developer to join the mailing list only when he
> has some kind of issue, but not stick around after he finds the solution.

Absolutely. Basing anything on what happens on the mailing list is just
annecdotal and hardly any proper evidence or actual data. And vice versa; the
lack of traffic about something doesn't necessarily mean much either.

But at the same time there's little doubt that what's being used the most get
the most questions on the mailing list and issues/PRs on github and there's
been lack of fixes and PRs to improve curl's pipelning code for a long time...

> Also I note that none of these people appear to use libcurl in browser
> projects

I know several "browser projects" as well that use libcurl. Or at least used
to - I'm pretty bad at keeping track of how libcurl-using apps change over
time...

> It looks like the users who are interested in HTTP 1.1 pipelining are more
> keen on performance and invariably quote throughput (**not latency**) as the
> the reason for their need to pipeline.

Pipelining does *nothing* for throughput. It only (marginally) helps latency.

> HTTP 1.1 will, in my opinion, continue to live along side HTTP/2 for the
> forseable future.

There's no doubt in my mind about that. I'm sure HTTP/1.1 will be around for
many years to come.

> While HTTP/2 has some benefits in browsing applications, it is its lack
> of pipelining

IMHO, HTTP/2's multiplexing is superior to pipelining in every way, and it has
none of the downsides HTTP/1.1 pipelining has.

> "Note: Stream dependencies and weights express a transport
> preference, not a requirement, and as such do not guarantee a
> particular processingor transmission order.

I see where you found this quote, and while there's no way to absolutely force
the order, RFC 7540 says:

  Inside the dependency tree, a dependent stream SHOULD only be allocated
  resources if either all of the streams that it depends on (the chain of
  parent streams up to 0x0) are closed or it is not possible to make progress
  on them.

The use of SHOULD there does of course not prevent servers from disobeying it.
But it would be against the advice of the spec.

> Case in point, Viren Baraiya, on Nov 14, 2017, while Engineering Manager at
> Netflix responded in a github issue [2] that Netflix were not working on
> HTTP/2 support.

They're such a big company they're likely to make everything and make all
possible network decisions in different areas so that doesn't prove much.

Netflix has sponsored HTTP/2 work in curl and engineers from them have fixed
numerous bugs in our HTTP/2 code because... Netflix uses (or at least used)
libcurl and HTTP/2 *extensively*.

> I think what is likely to happen is that someone will fork libcurl, bring it
> back to its core functionality, bring it into RFC compliance, and clean up
> the cruft

Aha! So that mysterious someone is just waiting for me to cut off pipelining
support *first*, then that excellent person will step forward and get working
on fixing it? That work we've been hoping would happen for many years by now?

It makes me wonder why this good resource doesn't step up already now so that
they can save themselves quite a lot of trouble.

> while libcurl is free to continue addressing the needs of a
> browser-developer audience,

Is that really what you think we're doing? Or are you just trying to be rude?
And lack of pipelining support is what going to make that sudden big change?

> Certainly if libcurl still supports a long defunct protocol like gopher,

I've explained this many times but let me do it again:

1 - we don't get bug reports on gopher (possibly because it is so little used)
2 - we have a set of test cases verifying gopher
3 - the gopher code is simply out of the way whenever we work on the code

There's really *no* effort in keeping it. We don't pay anything to have it.
There's no cost.

Pipelining is a very different story. It is intermixed with HTTP - the most
used code in curl. It sits in the transfer code path there. You can't change
much HTTP code without affecting, or risk to affect, pipelining. A very
different story compared to gopher (and many other fringe features).

> it should still support pipelining which was only revised 4 years ago
> (HTTP 1.1 RFCs were published June 6, 2014).

1. yes I know, I participated in that work (see Acknowledgements)

2. of course it is, it was a refresh of the documentation it was not an update
of the protocol. pipelining was added to HTTP/1.1 in 1996 so *all* documents
that document HTTP/1.1 will always document it!

3. it will be in the next refresh too (the work has started, likely to be
published within a few years)

So you've written this huge email to basically say I'm wrong and pipelining is
great but not a single word on what you can bring to the table to improve its
weaknesses or how you can help out to make it better so that it we can keep it
around with pride.

I'm prepared to reconsider or reschedule the deprecation plans for pipelining,
but that will require that someone actually shows - with working code - that
they're ready to help out. We need more test cases (especially such ones that
break the pipes) and we need bug fixes to fix the known flaws (like removing a
handle that is used in a pipe) and the ones the new test cases will find.
Ideally some code could even be improved so that it can co-exist easier with
non-pipelining code.

If nobody steps up to this, I will take that as yet another indication that
not that many people actually use or care about Pipelining support in libcurl.

Words are easy. Working long-term code, is harder work.

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html