curl / Mailing Lists / curl-library / Single Mail


Re: Using libcurl sending much more HTTP request in parallel

From: Jack via curl-library <>
Date: Thu, 10 Aug 2017 23:42:43 +0000

> ok, please ignore this,could you comment for this or my previous review
two mails thanks

Stenberg <> wrote:

> On Wed, 9 Aug 2017, fan fan via curl-library wrote:
> In my application, I using libcurl to send 40 HTTP requests in every
>> one second. As I test, this consume much CPU, my application such as:
>> main thread for sending request and using epoll listen fds whether read
>> or write.
>> while(running) {
>> int ret = epoll_wait(epollFd, events, maxEventNum, 10); //10ms
> Unconditionally returning every 10 milliseconds like this will make you
> spend a lot of CPU in vein. Don't do that. libcurl tells you how long it
> wants you to wait at most. No need to spin around a lap when there's
> nothing for libcurl to do.
> Do you also call libcurl to tell there's a timeout if the 10 milliseconds
> lapse or when do you do that?

   [Fan] Maybe this is not root cause, because when no http request send,
CPU almost 0.1%. of course I need to expand waiting time.
           this epoll_wait for listening every easy handle fd, I
set CURLMOPT_SOCKETFUNCTION, when callback, I add this fd to epoll events.
           of course I set CURLMOPT_TIMERFUNCTION, when timeout, I
call curl_multi_socket_action

> for (i = 0; i < ret; i++) {
>> if (events[i].events & EPOLLIN) {
>> curl_multi_socket_action // set CURL_CSELECT_IN
>> curl_multi_info_read // for reading
> I think you should put the curl_multi_info_read() call outside of the loop
> checking for activities, since it might not have anything to return and
> then its a bit of a waste to call it up to 40 times.
> But sure, curl_multi_socket_action() may of course consume CPU since
> that's the function that will handle the traffic on the socket you give it
> (and possibly on a few others if necessary).

   [Fan] So, when do I call curl_multi_info_read?
            I refer to:
            when socket have something to read, call curl_multi_info_read,
is it correct?

> As I test, send 100 or 200 or 300 or 400 or 500 in parallel maybe consume
>> cpu 100%.
> On modern hardware that shouldn't happen.

Fan] Agree

> I found there are too much EPOLLIN event, this consume much cpu.
> What do you mean with too much EPOLLIN event? If libcurl tells you to wait
> for that, shouldn't you then be glad you get it so that you can download
> the data?

   [Fan] I add socket fd to epoll event when CURLMOPT_SOCKETFUNCTION
             If there has a lot of easy handle in processing, trigger
             CURLMOPT_TIMERFUNCTION not about epoll fd, is it right?
             CURLMOPT_TIMERFUNCTION only indicating whether time out or

Received on 2017-08-11