curl / Mailing Lists / curl-library / Single Mail

curl-library

Re: cURL Write Data Buffer size

From: Pawel Veselov <pawel.veselov_at_gmail.com>
Date: Thu, 16 Mar 2017 12:37:13 -0700

> I'm using the API to fetch data and am using the WRITEDATA, WRITEFUNCTION
> options to handle it. Unfortunately I am finding that in some instances
> where the data is too long it gets separated in more than one call of the
> WRITEFUNCTION callback function. At this point the application loses track
> of the data because I am using the API to fetch other URLs as well. Is there
> any way to control the size of the buffer so that all of the data can be
> handled in a single call of WRITEFUNCTION?
>
> An earlier approach that I tried was to have predefined slots for each of
> the URLs (tracks of data). In which case each of the slots got populated
> with data at leisure over multiple calls of the WRITEFUNCTION. This works
> but there's some drawback in efficiency and this implementation faces some
> other symptomatic inadequacies with the cURL API.
>
> I have now decided to use a single WRITEDATA buffer. But I wish to set or
> increase its size so that WRITEFUNCTION need only be called a single time
> for any given track of data or URL.

I'm wondering what is a "WRITEDATA buffer"?

WRITEDATA (in the presence of WRITEFUNCTION) is simply an address (usually)
value that gives your function a context. If you want to have it point to a
buffer, have it point to a structure that points to a buffer (a single
address to a buffer would probably not carry enough information).

It's normal to have multiple write calls. It's unclear how you would instruct
cURL to make a single read, when the destination can be endless. It won't
even work well if it is chunked.

I don't know what you'd do with the data upon receiving it, and
implementation of your write function should heavily depend on that. But
it's generally recommended to code as if there would be multiple write calls,
and the function should "stash" (in whatever way applicable) the data before
more arrives. The WRITEDATA is a value that you create yourself that should
contain (or rather point to) information that is sufficient to know which of
the requests the current write call applies to.
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2017-03-16