curl / Mailing Lists / curl-library / Single Mail


Re: Pipelining is a pain, can we ditch it now?

From: Daniel Stenberg via curl-library <>
Date: Thu, 9 Aug 2018 12:03:19 +0200 (CEST)

On Thu, 9 Aug 2018, Radu Hociung via curl-library wrote:

> I have looked at every one of those reports, and I found no unresolved bugs.
> Care to list the remaining unaddressed issues?

Off the top of my head, here are a few things I think are important to get
fixed to increase my confidence-level in pipelining:

1. the flaky tests suggests things are not stable

2. we have too few tests for such a complicted feature so clearly we're not
testing enough. In particular we need more tests for broken connections with
several unfinished requests on the connection.

3. stopping a pipelined request before its done causes segfaults. It'd be
interesting with test cases that remove the handles at various stages during a
pipelined transfer.

> I noted that some are looking for functionality which does not exist in
> the protocol (like cancelling already issued requests).

I don't think that's a correct reading of the problem.

libcurl offers an API that makes generic transfers. In the multi API you can
do many parallell transfers and they may or may not use pipelning. The API has
*always* said that you can just remove an easy handle from the multi when
you're done with it (you want to abort it) and want to take it out of the
transfer loop. A typical application doesn't know nor does it have to care if
that particular transfer uses pipelining or not at that point. When it wants
to stop one of the transfers, it removes the easy handle.

The fact that it works to remove the handle in all cases *except* for some
situations when the transfer happens to do pipelining is a bug. A pretty
serious bug too since it might cause a segfault.

> It's not clear to me what those devs expect would happen in that scenario;

We (should) expect libcurl to deal with it appropriately. Not crashing is a
required first step.

I would imagine that depending on the state of the transfer, we'd have to
somehow mark it as pretend-its-removed for the user (and stop delivering data
to the callback etc) until the transfer has proceeded far enough so that we
can remove it for real.

> Also I don't know why they expect this functionality to exist.

Because it is documented to work like that? Because users want to stop

> the messy data organization would just cause new bugs to pop up

So maybe start there and clean up? Or work on improving the test suite perhaps
to make sure it catches remaining issues.

> I offered to rewrite multi.c and contribute it back, but there is no
> interest from you apparently.

Nobody here need rewrites. We need iterative improvements. Grab one
issue/problem, fix it and land it. One at a time. Over time that equals the
same thing as a rewrite, but it gets done by taking many small steps.

But if you truly think a larger take is necessary for something specific, then
do that larger take as small as possible and explain to us why we want to do
it like that. And a word of advice: try not to be rude and sprinkle the
explanation with insults, it doesn't get received as good then.

So please, do help us out!

> There are forks of libcurl being maintained and used, just nobody has yet
> volunteered to maintain their own fork as a lightweight, packageable
> alternative to libcurl yet.

While I think it is sad that we can't all work together on a common goal in a
single project instead of spreading the effort and developers on several
forks, they're all of course entitled to do that and so are you if you prefer.

> URL parsing is not apparently RFC compliant, but WHATWG compliant

I think you shuld go back and read my posts and blogs again on this topic
because that's... wildly incorrect.

> Your libcurl release schedule is fast and furious, like the browsers.

Allow me to point out that I did "fast and furious" releases many many years
before the browsers figured out it was a good idea.

Frequent releases is good for the community.

> I'm sorry if I seem rude to you

You don't "seem" rude to me. You are, and deliberately so too.

> In bug 2701 which you closed, I offered to contribute back my rewritten
> multi.c, that would guarantee order of execution. I provided a test case,
> but I don't have time at the moment to work on multi.c. You closed it.

There was no code in issue 2701; the issue was a bug report about transfer
ordering - something that libcurl doesn't guarantee (and then it side-tracked
out on a long rant about libcurl internals that was clearly off-topic). The
bug report was closed, as I don't see a bug there. I didn't feel you cared
about the actual bug much anymore at that point.

If you provide a patch/PR with actual code, I would review and discuss it.

Like we *could* discuss what tranfer ordering libcurl could guarantee and how
we could go about and make it work. If we could stay on topic and be civil.

> In bug 2782 I pointed out the examples with external event loops contain
> a use-after-free/segfaulting bug (which took me at least 1 day of hair
> pulling to debug), and provided a patch. You rejected it.

That was a bad patch since it "fixed" the examples in a way that wasn't
necessary, mentioning a problem that doesn't exist there. I'm sorry you feel
hurt by that, but it was not personal and I still can't see a reason why we
would accept that patch.

You may have identified a problem with calling curl_multi_cleanup() at some
point, and that would be good to get clarified, but that example patch did
nothing to help us with that. The examples already use the API as documented
so if the examples are problematic, then our API is wrong/bad and we should
start by fixing it in that end.

I said so in the issue. You have not provided a recipe or code for others to
use to reproduce your problem. You have not provided a patch that fixes the
problem you say you experience. If you did, I must've missed it and I

> Sorry to say, but you reject offered help and spew FUD. That is my
> impression.

I "spew" my opinons and views on the state of things. Like you do. Like I hope
and wish everyone here does! Calling my views FUD is just insulting.

> Removing an active handle from a pipeline is hardly a pipelining bug. I
> think you should maintain a list of pipelining bugs that need fixing, and
> test case scenarios that you know to break pipes.

I rest my case.

Received on 2018-08-09