cURL / Mailing Lists / curl-library / Single Mail


Re: Recommended architecture for streaming over multiple sockets

From: Boutin Maël <>
Date: Tue, 8 Mar 2016 10:25:20 +0100


Thank you for your anwser Daniel.

I went for the multi_interface solution, coupled with a pause when there
are no data to send and unpause when there are more data. I may have
several connection (~1000), with a global throughput of 800Mb/s on a
localhost endpoint

Currently i call multi_perform when there are data to be sent. How can i
obtain the send status ? By performing a select on the handles ? Here is
what i am doing at the moment :

    if (bSomethingToDo)
      // Call multi_perform
      curl_multi_perform(m_pstCurlMultiHandle, &iRunningFDs);

As i'm doing chunked encoding transfer, i don't have to check for a
response on the handle but i'd like to know when packets could not be sent,
and reinitiate the connection in that case (i have a init buffer to send at
connection initialization)


2016-03-07 13:30 GMT+01:00 Daniel Stenberg <>:

> On Wed, 2 Mar 2016, Boutin Maël wrote:
> I'm currently using libcurl to send data over a nginx local server using
>> HTTP POST and transfer-encoding chunked.
>> My application is fed with buffers which i have to send as soon as i get
>> them. These buffers may belong to various endpoints on the nginx server.
>> My question is, what use of libcurl would you recommend for this task. I
>> saw two options:
> Sure, both ways work and you can pretty much pick whichever you prefer.
> 1. Multiple threads
>> I'd have one thread with a FIFO queue that would get the buffers and get a
>> thread from a pool. This thread would then send the buffer to nginx
> I think it depends a little on how long period you expect to pass without
> having any data to send. As libcurl will call the read callback when it is
> ready to send data, you can basically just wait there until you get data,
> but that might not be the nicest design if that can take a very long time.
> 2. One thread and multi interface
>> I'd here work in non blocking mode with the multi interface. I'm not sure
>> how i'd have to implement it and it seems quite overkill to me.
> You'd still have the same callback for that, but then you could return a
> pause and you could have your select() wait for your own file handle
> waiting for more data to arrive and then kick it back into running again
> once data has arrived.
> * The connection with the server must not be closed even if there are no
>> more data to send.
> Generic servers tend to not like too long pauses without data though.
> * If we cannot send data on the connection that it must be closed and
>> reopened.
> That would imply a disconnect and libcurl will discover that and return an
> error accordingly.
> --
> /
> -------------------------------------------------------------------
> List admin:
> Etiquette:


List admin:
Received on 2016-03-08