Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
libcurl read-llike interface.
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: XSLT2.0 via curl-library <curl-library_at_cool.haxx.se>
Date: Mon, 14 Dec 2020 12:37:14 +0100
> https://github.com/curl/fcurl
Many thanks Daniel, that was extremely helpful for me to better
understand the "multi - select" way of doing thing.
> It has been untouched for years since nobody seems to use it,
> we don't get any feedback and nobody writes any pull requests for it.
The code is obviously incomplete, which helps a lot understanding things
because it is "short enough", but does probably not encourage to write PR?
There are also obvious "bugs" such as a missing curl_multi_cleanup() in
the fclose function!
It is also pretty un-optimised: the write callback could be called
multiple times in a multi_perform cycle.
This is correctly handled, but can lead to unknown memory consumption
(bad if you work with limited memory) and multiple copies of the same
data due the realloc technique!
> -2) A step towards zero-copy libcurl (a topic much discussed!)
> I don't think it will help with that...
Vocabulary (as often)! It depends on what you call "zero-copy" here.
I rule out here "zero-copy-socket" (on Linux for instance) which could
be beneficial and is indeed independent of the read-like interface since
it would not change the current interface (or maybe the multi-socket only?)
I also rule out the "splicing" technique used by the Linux kernel moving
4K pages pointers instead of bytes: too "kernel specific"!..
libcurl being a "userland" library, I would consider close enough to
"zero-copy", working with "caller provided buffers" instead of "curl
internal buffers".
Doing so, the relation is obvious: half of the code of "fcurl" is about
moving data from the curl internal buffers to the "caller provided buffers"!
So if the caller could:
- call libcurl to provide buffer/size where the data is needed
- call curl_multi_perform (or whatever existing/new function)
- curl_multi_perform returns when the provided buffer is full.
That would trim down the fread code to almost nothing: 2 calls and error
handling... but possibly pushing down the rest of the fcurl code inside
the library, which hopefully would take care of limiting the number of
memcpy/realloc!
The ideal being no malloc/realloc at all and only one copy, directly
from the socket's buffer into the "caller provided buffer".
So maybe it is the other way round: "caller provided buffers" as close
enough to "zero copy", would make quite obvious developping a fcurl
interface (if that would be needed!).
But maybe the "caller provided buffers" technique is already possible to
achieve with "multi-socket"?
Could someone point me to a similarly enlightening example as fcurl, but
using "multi-socket"?
Regards
Alain
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Received on 2020-12-14
Date: Mon, 14 Dec 2020 12:37:14 +0100
> https://github.com/curl/fcurl
Many thanks Daniel, that was extremely helpful for me to better
understand the "multi - select" way of doing thing.
> It has been untouched for years since nobody seems to use it,
> we don't get any feedback and nobody writes any pull requests for it.
The code is obviously incomplete, which helps a lot understanding things
because it is "short enough", but does probably not encourage to write PR?
There are also obvious "bugs" such as a missing curl_multi_cleanup() in
the fclose function!
It is also pretty un-optimised: the write callback could be called
multiple times in a multi_perform cycle.
This is correctly handled, but can lead to unknown memory consumption
(bad if you work with limited memory) and multiple copies of the same
data due the realloc technique!
> -2) A step towards zero-copy libcurl (a topic much discussed!)
> I don't think it will help with that...
Vocabulary (as often)! It depends on what you call "zero-copy" here.
I rule out here "zero-copy-socket" (on Linux for instance) which could
be beneficial and is indeed independent of the read-like interface since
it would not change the current interface (or maybe the multi-socket only?)
I also rule out the "splicing" technique used by the Linux kernel moving
4K pages pointers instead of bytes: too "kernel specific"!..
libcurl being a "userland" library, I would consider close enough to
"zero-copy", working with "caller provided buffers" instead of "curl
internal buffers".
Doing so, the relation is obvious: half of the code of "fcurl" is about
moving data from the curl internal buffers to the "caller provided buffers"!
So if the caller could:
- call libcurl to provide buffer/size where the data is needed
- call curl_multi_perform (or whatever existing/new function)
- curl_multi_perform returns when the provided buffer is full.
That would trim down the fread code to almost nothing: 2 calls and error
handling... but possibly pushing down the rest of the fcurl code inside
the library, which hopefully would take care of limiting the number of
memcpy/realloc!
The ideal being no malloc/realloc at all and only one copy, directly
from the socket's buffer into the "caller provided buffer".
So maybe it is the other way round: "caller provided buffers" as close
enough to "zero copy", would make quite obvious developping a fcurl
interface (if that would be needed!).
But maybe the "caller provided buffers" technique is already possible to
achieve with "multi-socket"?
Could someone point me to a similarly enlightening example as fcurl, but
using "multi-socket"?
Regards
Alain
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Received on 2020-12-14