> Date: Thu, 23 Aug 2012 18:01:09 -0400
> Subject: Re: question about curl_multi_perform, callbacks, and presenting a pull interface
> From: rich.bramante_at_gmail.com
> To: curl-library_at_cool.haxx.se
>
> On Thu, Aug 23, 2012 at 2:18 PM, Daniel Stenberg <daniel_at_haxx.se> wrote:
> > On Thu, 23 Aug 2012, Rich Bramante wrote:
>
> >> Question 3: What is the best way to craft an interface like this using
> >> curl?
> >
> > The best way is to take care of all data you receive. realloc() the buffer
> > or use multiple ones.
> > --
> >
> > / daniel.haxx.se
>
> I had considered this approach but was fearful that in the case where
> the server was returning a large amount of data quickly, curl would be
> in a constant read-callback loop and and my library would not be able
> to return control back out to the caller to start draining the data
> possibly causing my library to buffer the entire response in memory.
> Since the client can download an arbitrarily large amount of data I
> did not pursue this.

Depending on the specifics of your application keeping received data in a temporary file on disk may be a fine solution. To optimize you can even define maximum buffer size you feel comfortable keeping in memory, once reached memory buffers can be flushed to disk and all client reads will be redirected there. Naturally, if you need to concurrently manage many connections that are receiving lots of data fast then this will not work because disk i/o will soon become a bottleneck.

-V.
                                               
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2012-08-24