cURL / Mailing Lists / curl-library / Single Mail


Re: libcurl don't write data directly (internal buffering?)

From: Eric Beuque <>
Date: Wed, 3 Oct 2012 10:33:28 +0200

Hi, thank you again for your help, i tried the curl debug function but it's
the same as the write callback, so i think there it's probably a cache in
the TCP layer.

Below is the code i use with libcurl:

            // set URL to get
            curl_easy_setopt(pCurlHandle, CURLOPT_URL, szMRL);

            // no progress meter please
            curl_easy_setopt(pCurlHandle, CURLOPT_NOPROGRESS, 0L);

            // Force to don't use cached connection
            curl_easy_setopt(pCurlHandle, CURLOPT_FRESH_CONNECT, 0);
            curl_easy_setopt(pCurlHandle, CURLOPT_FORBID_REUSE, 0);

            // disable timeout that can raise signal
            curl_easy_setopt(pCurlHandle, CURLOPT_NOSIGNAL, 1L);
            curl_easy_setopt(pCurlHandle, CURLOPT_TIMEOUT, 0);
            curl_easy_setopt(pCurlHandle, CURLOPT_CONNECTTIMEOUT, 0);

            // send all data to this function
            curl_easy_setopt(pCurlHandle, CURLOPT_HEADERFUNCTION,
            curl_easy_setopt(pCurlHandle, CURLOPT_WRITEHEADER, pContext);

            // send all data to this function
            curl_easy_setopt(pCurlHandle, CURLOPT_WRITEFUNCTION,
            curl_easy_setopt(pCurlHandle, CURLOPT_WRITEDATA, pContext);

            // Set progress function
            curl_easy_setopt(pCurlHandle, CURLOPT_PROGRESSFUNCTION,
            curl_easy_setopt(pCurlHandle, CURLOPT_PROGRESSDATA, pContext);

            curl_easy_setopt(pCurlHandle, CURLOPT_VERBOSE, 1L);
            curl_easy_setopt(pCurlHandle, CURLOPT_DEBUGFUNCTION, onDebug);

            curlResult = curl_easy_perform(pCurlHandle);

The purpose of the sleep function is to simulate a very long frame
processing time. So consider that every 70kb, i have found a new boundary,
so the previous received data that have been keept in a temporary buffer
are processed. The very simplified version of the algorithm is:

onWriteData(data, size) {
  // check if the data contains a boundary
  if(containsBoundary(data, size)){
    processBuffer(m_buffer); // process the frame
    sleep(1); // just to simulate a long processing time
    m_buffer.clear(); // erase the buffer
      m_buffer.append(data, size); // keep data in a temporary buffer

What is surprising me, as i told, is that when i shutdown my server,
libcurl goes to the write data callback around 6-7 frame (so 6-7 second).
How can we explain that libcurl still receive data while the connection
with the server is closed? If libcurl doesn't cache anything, i think it's
the tcp layer that has a internal buffer.

Note that i upgrade my system to libcurl 7.26.

On Sat, Sep 29, 2012 at 3:49 PM, Daniel Stenberg <> wrote:

> On Fri, 28 Sep 2012, Eric Beuque wrote:
> In fact, i add a sleep(1) in the function that process my frame. I also
>> develop a server application that generate an MJPEG stream (but i got the
>> same problem directly from an ip camera).
> Sorry, but I don't quite understand where you added the sleep() and what
> that proved! Also, you haven't shown us how your libcurl using code looks
> like so I can't exclude that there's a problem with it.
> Internally in libcurl when it finds that the socket is readable, it will
> read (up to 16K) and then immediately hand over the read data (after some
> possible massaging of it depending on what protocol, libcurl options and in
> which shape the data arrived) to the write callback.
> There is no internal buffering but libcurl will then immediately use the
> same buffer to again read more data from the socket as soon as it becomes
> readable again and so it loops. This goes on until the transfer is
> completed.
> CURLOPT_DEBUGFUNCTION is a great option to use to get to see exactly which
> data that is received by libcurl and what data it sends and when.
> --
> /
> ------------------------------**------------------------------**-------
> List admin:**listinfo/curl-library<>
> Etiquette:**etiquette.html<>

List admin:
Received on 2012-10-03