Re: Is there a guarantee that for multiple parallel responses header function and write function will not be interleaved?

From: Tomalak Geret'kal via curl-library <curl-library_at_cool.haxx.se>
Date: Fri, 13 Nov 2020 14:30:28 +0000

On 13/11/2020 13:59, Arvind Srivaths via curl-library wrote:
>
> Firstly, I would like to thank all the developers working
> on Curl for a reliable library that is easy to use for
> beginners and yet is customisable for very advanced work.
>
> I am using curl to make REST calls and each request
> contains a request Id in the HTTP header.
>
> I detect the request Id (copied back into the response by
> the server) in the header handler and use that in the data
> handler to map to the CommHandle (my class, which has a
> reference to the Curl handle) and copy the data into the
> handle specific buffer.  With a single outstanding
> request, it's guaranteed that nothing will come between
> the call to the two handlers and I can rely on the fact
> that the request Id detected in the header handler can be
> used by the data handler without anything stomping on it. 
>
> I am not sure whether this will work if Curl uses multiple
> threads for multiple handles and if another response can
> overwrite the request Id before the data handler gets to
> it.  Is there any guarantee that the library (as of today)
> will not interleave calls to header function and write
> function for two responses?  While this may not be the
> case with Curl today, is there a chance that it may change
> in future?
>
> I can think of several solutions to this:
> . Ideal solution is if Curl itself can provide a single
> handler to handle both header and data - then it's atomic
> and we can use locks to guarantee mutual exclusion.
> . I can lock the request Id in the header handler and
> unlock after getting it in the data handler.  Depending on
> how Curl implements response handling it may result in
> deadlock.  With two threads if it calls header handlers in
> order, the second  can block waiting for the first (which
> will not release the lock till its data handler is called)
> and if curl waits for both to end before calling data
> handlers, you have a deadlock.  If this Curl code is
> unlikely to be run in multiple threads, I can use this
> solution temporarily.
> . I can assume in the header handler that the data starts
> at the end of the header and simulate a single handler. 
> If Curl implementation changes (eg. for HTTP 2 header is
> compressed and decompressed header may not be contiguous
> with data) this may break.
>
> I would like your opinion on what is the best solution to
> this issue.
>
> Thanks,
>
> Arvind


What you should really do, is assume that requests can be
interleaved, and design your CommHandle class accordingly.

When you get headers, don't just store one request ID: store
all pending request IDs. Map them to something unique for
the duration of a request, like the cURL easy handle. Then,
when you get body content, you take the easy handle and
determine the request ID from it. Finally, clean up the map
entry when your request completes.

All of this can be done with locks in your CommHandle for
thread-safety if you're expecting different requests to be
serviced on multiple threads (which depends on how you're
using cURL.)

Cheers



-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2020-11-13