Thanks!

I am planning to create X easy handles for the X worker threads and add
them to the multi handle before executing curl_multi_perform in a loop.

These X worker threads would then continuously change the URL (same
external server but with different query parameters) for the X easy handles
and keep calling curl_easy_perform (or an equivalent) to make a callout to
the external server for the incoming requests.

Would that work?

Regards
Raj

On Fri, Jul 10, 2015 at 7:52 AM, Ray Satiro via curl-library <
curl-library_at_cool.haxx.se> wrote:

> On 7/10/2015 2:06 AM, Rajalakshmi Iyer wrote:
>
>> Thanks for getting back. Here is some background to the application in
>> its current state -
>>
>> The application has X worker threads that receives 1000s of concurrent
>> requests per second.
>>
>> As part of handling these incoming requests, the X worker threads need to
>> make a callout to an external HTTP server. This callout typically takes up
>> to 100 ms.
>>
>> Currently, the X worker threads have implemented this by using a
>> curl_easy_handle per thread, i.e. a connection per thread that is
>> continuously making callouts and handling the responses in the callback
>> function CURLOPT_WRITEFUNCTION.
>>
>> However, this also means that the application is not doing much except
>> waiting on external HTTP server. The CPU is hardly utilised.
>>
>> In order to overcome this, I am planning to use the curl multi interface.
>> So each of the X curl easy handles from X threads will be added to the curl
>> multi handle.
>>
>> Does that sound a reasonable thing to do?
>>
>
> Yeah it seems beneficial assuming you don't need to block each thread
> waiting for the server reply (which is the way curl_easy_perform works as
> you've seen).
>
> Generally there is something in libcurl I'm unclear about (I've asked on
> the list before but didn't get a reply addressing it) which is it seems
> that it's acceptable behavior for curl_multi_perform to always be
> performing and not return for extended periods of time and I thought in
> that case you can't call multi_add_handle from another thread during that
> time. Whether that's even true (my reading of the code at the time was yes)
> and you will encounter that in your situation is for Daniel or someone more
> knowledgeable to say.
>
> Assuming so you have to account for that in your design because as you add
> easy handles to the multi handle from different threads you could have a
> lot of locking if thread 1 holds the lock because it's is in the middle of
> using the multi handle (ie you called multi_perform) and you've got thread
> {2,9} locked waiting to add easy handles but can't because the multi is in
> use for extended periods by thread 1. It seems to me the way to work this
> is create a global queue list with a lock and each thread holds the lock as
> it adds to the queue. Then back in thread 1 in your multi loop you hold the
> lock only while you move from the queue to multi. Note in this case if
> multi_perform is out for extended periods of time nothing is added for
> extended periods of time.
>
> In other words
> multi_perform
> lock
> add from queue
> unlock
> multi_wait
> multi_info_read
>
>
>
>
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/list/listinfo/curl-library
> Etiquette: http://curl.haxx.se/mail/etiquette.html
>

-- 
    Rajalakshmi Iyer  *Senior Software Engineer*    raj-blismedia.com
<http://is.gd/L42zBR>[image: Skype icon]       [image: Blis logo]
<http://www.blismedia.com/>    <http://www.blismedia.com/>
<https://www.facebook.com/BlisMedia?fref=ts>
<https://twitter.com/BlisMedia>
<https://www.linkedin.com/company/blismedia>
<https://plus.google.com/u/0/103190831541608378945/about>
-- 
This email and any attachments to it may be confidential and are 
intended solely for the use of the individual to whom it is addressed. Any 
views or opinions expressed are solely those of the author and do not 
necessarily represent those of BlisMedia Ltd, a company registered in 
England and Wales with registered number 06455773. Its registered office is 
3rd Floor, 101 New Cavendish St, London, W1W 6XH, United Kingdom.
If you are not the intended recipient of this email, you must neither take 
any action based upon its contents, nor copy or show it to anyone. Please 
contact the sender if you believe you have received this email in error.

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2015-07-10