On 06/08/2018 4:53 PM, Daniel Stenberg wrote:
> Right, quite ironic I think that it wasn't a pipelining bug that
finally triggered me to put pipelining up for deprecation...

You keep saying there are lots of bugs and missing test cases. Since I
am interested in pipelining, I did some research on this ML and through
the issue tracker, and found no evidence of remaining bugs in the
pipelining functionality. I did find evidence that lots of other
developers are interested in using pipelining in their projects. This
was the point of me writing the (long) summary. The state of the feature
looks quite solid to me. Having looked at the inner workings of multi.c,
it's a miracle that it works, but it mostly does work.

> Several of those persons you mentioned idenfied problems or flaws with
pipelining, several of those were never fixed. As we don't get them
reported more I take as an indication that not enough users are using
the feature to hit them that often (to report them).

I have looked at every one of those reports, and I found no unresolved
bugs. Care to list the remaining unaddressed issues?

I noted that some are looking for functionality which does not exist in
the protocol (like cancelling already issued requests). It's not clear
to me what those devs expect would happen in that scenario; Also I don't
know why they expect this functionality to exist. In most of computer
science and internet protocols, there isn't much "unshoot the arrow"
type of functionality; why do they expect it in curl?

This being said, I do know there are some segfaults and crashes when
attempting "unexpected" API calls, like removing active easy handles, or
cleaning up the multi handle at "the wrong time". These are real bugs,
but they are not related to pipelining. They are due to the mess of data
members sprawled across unrelated internal structures. IMO, patching
individual segfaults is futile, the messy data organization would just
cause new bugs to pop up in other places, either immediately or at a
future time. The test suite is not thorough enough to catch regressions,
and I'm not talking about pipelining, but generally.

> I know several "browser projects" as well that use libcurl. Or at
least used to - I'm pretty bad at keeping track of how libcurl-using
apps change over time...

They are a minority. The main attraction of libcurl is in unattended,
automated work involving the HTTP protocol, not in supporting an HTML
rendering engine. Think yum, apt-get, git, etc.

>> It looks like the users who are interested in HTTP 1.1 pipelining are
more keen on performance and invariably quote throughput (**not
latency**) as the the reason for their need to pipeline.
>
> Pipelining does *nothing* for throughput. It only (marginally) helps
latency.

I cannot agree with you. Pipelining enables full-duplex HTTP comm.
Without it, the two parties would have a half-duplex channel, each
waiting for the other to be done talking before talking itself. Surely
you understand that each direction will experience higher throughput if
the wait is eliminated. This was the drive behind HTTP/1.1 pipelining,
not latency. Latency was never a problem, a browser could open as many
TCP connections as needed to request objects in parallel and minimize
latency, and still can, even if each connection is pipelined. In fact
pipelining increases latency, as new requests are queued after those
already in the pipeline. A queue increases latency, by definition, as
each request is not sent immediately, but after waiting in queue.

Connection persistence lowers latency in the case of serialized/ordered
requests by doing away with connection setup. Pipelining increases
latency, connection persistence lowers it. When combined, one can
calculate what happens to latency. It depends on the size of each
request, the pipeline depth, bandwidth, etc.

> IMHO, HTTP/2's multiplexing is superior to pipelining in every way,
and it has none of the downsides HTTP/1.1 pipelining has.
>
>> "Note: Stream dependencies and weights express a transport
>> preference, not a requirement, and as such do not guarantee a
>> particular processingor transmission order.
>
> I see where you found this quote, and while there's no way to
absolutely force the order, RFC 7540 says:
>
> Inside the dependency tree, a dependent stream SHOULD only be allocated
> resources if either all of the streams that it depends on (the chain of
> parent streams up to 0x0) are closed or it is not possible to make
progress
> on them.

> The use of SHOULD there does of course not prevent servers from
disobeying it. But it would be against the advice of the spec.

So strict ordering is not guaranteed, like it is in pipelining? For
applications that *require* strict ordering and full link utilization,
there is no substitute for HTTP/1.1 pipelining. HTTP/2 needs a wrapper
to defeat all the fancy multiplexed "stream" logic in order that strict
ordering be guaranteed.

>> I think what is likely to happen is that someone will fork libcurl,
bring it back to its core functionality, bring it into RFC compliance,
and clean up the cruft
>
> Aha! So that mysterious someone is just waiting for me to cut off
pipelining support *first*, then that excellent person will step forward
and get working on fixing it? That work we've been hoping would happen
for many years by now?
>
> It makes me wonder why this good resource doesn't step up already now
so that they can save themselves quite a lot of trouble.

A lot of devs venture on your mailing list, and are turned away, like I
am. I offered to rewrite multi.c and contribute it back, but there is no
interest from you apparently.

If I decide to keep using libcurl (I don't have feasible alternatives),
I will be maintaining my own private fork with multi.c cleaned up. I
have noted in my summary that others appear to have taken the same route
(wrt. pipelining POST requests). There are forks of libcurl being
maintained and used, just nobody has yet volunteered to maintain their
own fork as a lightweight, packageable alternative to libcurl yet. I
don't think I have this calling either.

>> while libcurl is free to continue addressing the needs of a
browser-developer audience,
>
> Is that really what you think we're doing? Or are you just trying to
be rude? And lack of pipelining support is what going to make that
sudden big change?
>
I have observed that every time pipelining is mentioned, you promote
HTTP/2 and pile on the FUD about HTTP/1.1, saying the former is 'safer'
than the later. You cite the major browser's disabling pipelining and
adopting HTTP/2. URL parsing is not apparently RFC compliant, but WHATWG
compliant (ie, a browser interest group). Your libcurl release schedule
is fast and furious, like the browsers. Judging from the seriousness of
bugs in the github issue tracker, the releases contain alpha-level code.
For a foundation library, this is unacceptable to many projects.
Foundational = many important packages depend on it, like the package
managers of debian and offshoots, redhat, etc, git,

I'm sorry if I seem rude to you, but the arguments you make are (to me)
clearly from a narrow, browser-specific perspective. My main
professional experience is at the opposite end of tech, in
control/automation/high-performance computing, where things remain
stable for a long time. Your arguments are very fallacious from my
perspective, and I don't buy them for a second. You are entitled to your
perspective. Libcurl is your project, so you can dismiss my perspective.

In April 2016, you proposed removing pipeling support, and said "Can we
really remove it? Oh heck, we can do whatever we want.". You're free to
ignore evidence that many projects do in fact rely on pipelining as a
source of performance, and substitute your own fabricated FUD. It
certainly doesn't appear that you're looking for feedback (in fact you
dismiss your own survey results), but for support of your decisions.
Just do whatever you want, why waste time asking the ML, if you don't
want the feedback?

> So you've written this huge email to basically say I'm wrong and
pipelining is great but not a single word on what you can bring to the
table to improve its weaknesses or how you can help out to make it
better so that it we can keep it around with pride.

In bug 2701 which you closed, I offered to contribute back my rewritten
multi.c, that would guarantee order of execution. I provided a test
case, but I don't have time at the moment to work on multi.c. You closed it.

In bug 2782 I pointed out the examples with external event loops contain
a use-after-free/segfaulting bug (which took me at least 1 day of hair
pulling to debug), and provided a patch. You rejected it.

Sorry to say, but you reject offered help and spew FUD. That is my
impression.

As far as I can see, the current implementation of pipelining in libcurl
does not need bugfixes other than the out-of-order execution bug, and
it is the bug I intended to fix and contribute back, until you closed
issue 2701.

> I'm prepared to reconsider or reschedule the deprecation plans for
pipelining, but that will require that someone actually shows - with
working code - that they're ready to help out. We need more test cases
(especially such ones that break the pipes) and we need bug fixes to fix
the known flaws (like removing a handle that is used in a pipe) and the
ones the new test cases will find. Ideally some code could even be
improved so that it can co-exist easier with non-pipelining code.
>
> If nobody steps up to this, I will take that as yet another indication
that not that many people actually use or care about Pipelining support
in libcurl.

Removing an active handle from a pipeline is hardly a pipelining bug. I
think you should maintain a list of pipelining bugs that need fixing,
and test case scenarios that you know to break pipes. Who better to know
what they are?

I have listed many names and dates when various people have expressed a
need for pipelining. Do you expect them to keep reading the ML and
periodically reassure you that they are still using it?

Regards.
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2018-08-09