>> I would like to have a "CURLOPT_BUFFER" option, so I can instruct
>> libcurl to copy the data there directly. What are your thoughts about
>> adding it/accepting a patch?

>Since libcurl may download an infintely large amount of data, just pointing out a single buffer is not enough. It would have to be the first in a series of buffers. The question is then how libcurl spends the buffer(s) and how you give it new/more buffers to fill.

>Also, in the attempt to avoid memcpy'ing data in your application, it is important to realize that you it won't be easy to avoid them in libcurl - I'm suggesting that it easily will just make the copy get done within libcurl instead. Things like (abstracted) SSL and chunked encoding for example make it very hard to avoid that.

>If you have any ideas, please let us know and we can go from there. It is easier to discuss around and improve an actual suggestion.

I had to deal with such problem but from different aspect - I had to handle lot of SSL requests to internal servers (in parallel) but the response from the server came only after several minutes.

I my case, CPU usage was very low (as most of time the connection was waiting for the response to come) but my memory consumption was very high (as each request allocated curl read+write buffer as part of the curl_easy handle + SSL read+write buffers). While I managed to reduce the memory consumption of SSL by using SSL_MODE_RELEASE_BUFFERS, I wasn't able to use such approach with curl internal buffers.

I think the first step should be taking out the read/write buffers from the curl_easy struct and have several options to get the buffer when needed:
1. From a user callback - curl will request buffer of size X and it is the application responsibility to give it such buffer (or return error code).
2. Manage a buffer pool in a curl_share object if requested to do so by the application. Application is able to limit the pool size. (this is very similar to what openssl do when using SSL_MODE_RELEASE_BUFFERS).
3. Manage a buffer per easy_handle as done today (default approach if none of the above was configured), but the buffer will not be part of the curl_easy struct, just allocated during curl_easy_init and free'd during curl_easy_cleanup).

In options 1+2 curl should try to return the buffer to the owner (shared object/application) as soon as possible (at least before each I/O call).

What do you think about such option?

Yehezkel

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2014-04-01