I'm having a little bit of trouble with the "pull" aspect of the multi
interface in libcurl v7.16.2.

Before I explain what I'm doing, let me explain the motivation.

TCP/IP has built-in flow control. That means that if a receiver is not
ready or able to process data, it can stop acknowledging packets. The
sending server will then stop sending packets until the receiver says its
ready. In that way, the receiver is prevented from being overwhelmed by a
fast server.

From the application point of view, this is accomplished by simply not
reading data from the socket. Data will back up, causing the server to
suspend transmission until the application resumes reading data from the
socket.

My goal is to use the "pull" aspect of the multi interface in exactly the
same way. When my application wants data (and it detects via select that
the socket has data), it calls curl_multi_perform. If it is not ready for
more data, it does not call curl_multi_perform. Note that even if
curl_multi_perform returns CURLM_CALL_MULTI_PERFORM, if my application is
not ready for more data, then it does not call curl_multi_perform. Once it
is ready for data, then it resumes calling curl_multi_perform.

curl_multi_perform delivers data via the CURLOPT_WRITEFUNCTION callback.
The maximum amount of data that it will deliver per callback is
CURL_MAX_WRITE_SIZE.

I had assumed that curl_multi_perform would call the CURLOPT_WRITEFUNCTION
function at most once every time it is called. This ensures that my
application never has to deal with more than CURL_MAX_WRITE_SIZE bytes until
it is ready for more.

The problem is that I have seen curl_multi_perform call
CURLOPT_WRITEFUNCTION more than once per invocation, delivering in total
more than CURL_MAX_WRITE_SIZE bytes. So far in testing, it has sometimes
called it twice, but there seems to be no guarantee. If can call
CURLOPT_WRITEFUNCTION more than once, it could conceivably call
CURLOPT_WRITEFUNCTION an indeterminate number of times, essentially breaking
the "pull" aspect of the interface.

When I want and expect is that each call to curl_multi_perform will invoke
CURLOPT_WRITEFUNCTION at most one time, and deliver at most
CURL_MAX_WRITE_SIZE bytes. This would allow me to ensure that my
application is ready for more data before I receive it, and would allow me
to effectively use the flow-control aspects of TCP/IP to prevent the host
from being overwhelmed with data.

So in summary, the problem is that curl_multi_perform sometimes calls
CURLOPT_WRITEFUNCTION more than once each time it is called, which delivers
in total more than CURL_MAX_WRITE_SIZE and overwhelms my application with
data. The solution would be to ensure that curl_multi_perform can call
CURLOPT_WRITEFUNCTION at most one time before returning.

Thank you,

Allen
Received on 2007-06-21