cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: Recommended architecture for streaming over multiple sockets

From: Daniel Stenberg <daniel_at_haxx.se>
Date: Mon, 7 Mar 2016 13:30:00 +0100 (CET)

On Wed, 2 Mar 2016, Boutin Maël wrote:

> I'm currently using libcurl to send data over a nginx local server using
> HTTP POST and transfer-encoding chunked.
>
> My application is fed with buffers which i have to send as soon as i get
> them. These buffers may belong to various endpoints on the nginx server.
>
> My question is, what use of libcurl would you recommend for this task. I
> saw two options:

Sure, both ways work and you can pretty much pick whichever you prefer.

> 1. Multiple threads
> I'd have one thread with a FIFO queue that would get the buffers and get a
> thread from a pool. This thread would then send the buffer to nginx

I think it depends a little on how long period you expect to pass without
having any data to send. As libcurl will call the read callback when it is
ready to send data, you can basically just wait there until you get data, but
that might not be the nicest design if that can take a very long time.

> 2. One thread and multi interface
> I'd here work in non blocking mode with the multi interface. I'm not sure
> how i'd have to implement it and it seems quite overkill to me.

You'd still have the same callback for that, but then you could return a pause
and you could have your select() wait for your own file handle waiting for
more data to arrive and then kick it back into running again once data has
arrived.

> * The connection with the server must not be closed even if there are no
> more data to send.

Generic servers tend to not like too long pauses without data though.

> * If we cannot send data on the connection that it must be closed and
> reopened.

That would imply a disconnect and libcurl will discover that and return an
error accordingly.

-- 
  / daniel.haxx.se

-------------------------------------------------------------------
List admin: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2016-03-07