On Tue, 1 Apr 2014, Cristian Morales Vega wrote:

>> We've discussed such alternatives many times and I'll be willing to do it
>> again.
>
> Can you point me to those discussions? I am sure there are a problems/user
> cases I am unaware of.

No, I think you can find them in the archives if you include search terms like
"zero copy interface" and similar.

Usually when people have asked for this, they have shunned away again pretty
soon when we taken a few steps toward actually getting it done and discussing
some of the challanges involved.

> I was thinking about simply changing (struct UrlState).buffer from an
> array to a pointer.

That doesn't get you very far though as it won't be enough even for your
simplest use case.

> I am not trying to solve the problem of downloading an arbitrary big file to
> memory for later use, which I guess is what you are thinking about. For such
> a thing, at least in Linux, I guess the future "memfd"s plus splice() could
> be used.

I'm only interested in solving this problem if it covers all sensible use
cases in a decent way.

I'm not interested in a quick fix that possibly works for a limited set of
users with strange conditions around its use.

> I am not using SSL here. I must admit I am totally unaware of the
> problems it adds.

But libcurl has an API that is mostly agnostic to if you use SSL or not. We
have to design this so that it works the same for HTTP as for HTTPS transfers.

> But aren't these *extra* copies? The data still ends in (struct
> UrlState).buffer and any copy from it can still be avoided, no?

Yes, ideally. I'm just suggesting that all data won't be delivered nicely in a
plain recv() call that libcurl itself invokes.

> My measurement is being able to let libavformat do all the processing in the
> callback time in a BCM4716A_at_457MHz. With the memcpy()s it stops libcurl for
> too long, and by the time recv() is called the TCP receiver window is
> already full.

The buffer is at maximum 16K to copy in the callback. A memcpy of 16K on a
457MHz CPU for you. 457000000 cycles per second, surely it can copy more than
one byte per cycle but let's say it takes one cycle per byte 16384 cycles at
457MHz. That's about 36 microseconds...

It is hard to imagine that you actually affect the TCP window much with a
sub millisecond delay!

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html