fopen.c uses the multi interface to read data from the stream using a
buffer that automatically increases in size via realloc. As url_fread
is called, two things happen: (1) curl_multi_perform is called, and (2)
once enough bytes have been read from the stream, data is copied out of
the buffer and the remaining data in the buffer is shifted down to the
beginning of the buffer. Artificially, using just one curl handle, if
you tried to read just 1 byte of the stream of a very large transfer,
curl_multi_perform would be called so frequently relative to the amount
of data actually being read off the buffer, that the buffer would grow
until memory was exhausted.

A more interesting problem I've discovered uses two curl handles and
alternating reads. Early on, and regardless of which handle is used
when calling url_fread(), when curl_multi_perform is called (after
waiting on select()), sometimes it transfers data to the first curl
handle's buffer, sometimes it transfers data to the second curl handle's
buffer. After a couple megs of each buffer getting a some data,
curl_multi_perform suddenly only transfers data to the first handle's
buffer, ignoring the second until the first is finished. This is a
problem when url_fread() is called on the second curl handle... it has
to wait until the first is done transferring, and the first must be
transferred in its entirety to dynamically resized and ever-expanding
memory until it is finished or is killed/crashes.

The real world problem I'm trying to solve is using FUSE to read from an
http stream. Basically a couple files are getting opened and read
through, but sometimes one of those files gets no data because the multi
interface is busy adding data to the file not being accessed, that is,
until the kernel kills the process after it runs out of memory.

Any ideas on a solutions?

Thanks!
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2011-05-19