Daniel Stenberg wrote:
> Since libcurl may download an infintely large amount of data, just pointing
> out a single buffer is not enough. It would have to be the first in a series
> of buffers. The question is then how libcurl spends the buffer(s) and how
> you give it new/more buffers to fill.

Commenting from the sidelines... (I'm not very active currently.)

Seems to me that one way to go about more flexible buffering would be to
make new versions of the read and write callback routines. If they were
passed a struct containing at least the buffer address and count, then the
callback could manipulate the buffers as desired for the next operation.
For example:

An application allocates a buffer big enough to hold the whole object being
retrieved. As libcurl delivers each chunk of data, the callback simply
advances the pointer and decreases the available count.

An application which is processing line oriented data processes what lines
it can in the buffer. It deals with a line spanning into the next buffer by
moving the partial line to the top of the buffer and adjusting pointer and
count to continue receiving.

A sending application memory maps the file, adjusting pointer and count in
read callbacks as curl sends the data.

An application sets up an array of buffers. As libcurl delivers data via
the write callback, it queues up the freshly received buffer for its
processing and gives libcurl the next buffer.

There is the issue of setting up the initial buffer. It is conceivable
that buffers could be managed entirely from within the callbacks without
needing special calls to setup. A callback with address, count NULL, 0
would mean 'gimme buffer' while a call with address, count of NULL, -1 could
mean 'done with buffers'. The sticky case would be where the callback
doesn't have the next buffer. It would have to do something akin to a pause
and then at some later time have a way of giving a new buffer to libcurl so
it can continue the transaction. Guess there is need for a set buffer
operation.

Having callbacks with a struct as an argument would mean that additional
information can be passed back and forth at callback time. For example, a
single io_callback() might have an op field that means something like:

0 - read callback
1 - write callback (header if separate handling requested)
2 - write callback (data)
3 - need a buffer
4 - done with buffers
5 - other good stuff??

Or there could be a header/data flag in the struct.

If there are operations which are sensible to have occur at callback time,
perhaps the easy & multi handles could be included in the struct.

Just some thoughts,
Rich

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2014-04-01