For hiperfifo.c test, I do perf record, htest process is test process.
complete data, please see attachment! some data below, could you give any
optimize advice?

​

On Thu, Aug 10, 2017 at 11:11 AM, Jack <sipgurume_at_gmail.com> wrote:

> By the way I use hiperfifo.c to send HTTPS 40 requests at the same time in
> https://curl.haxx.se/libcurl/c/hiperfifo.html
>
> CPU sometimes burst to 69% or 80%, quickly down to 25% then to 0.0%
>
> Code below:
> static void fifo_cb(int fd, short event, void *arg)
> {
> char s[1024];
> long int rv=0;
> int n=0;
> GlobalInfo *g = (GlobalInfo *)arg;
> (void)fd; /* unused */
> (void)event; /* unused */
>
> int i=0;
> do {
> s[0]='\0';
> rv=fscanf(g->input, "%1023s%n", s, &n);
> s[n]='\0';
> if(n && s[0]) {
> for(i=0; i<40; i++) {
> //printf("new_conn start\n");
> new_conn("https://test.example.com", arg); //This code I add
> }
> }
> else
> break;
> } while(rv != EOF);
> }
>
> Complete code see attachment.
>
> On Thu, Aug 10, 2017 at 9:23 AM, Jack <sipgurume_at_gmail.com> wrote:
>
>>
>>
>> On Thu, Aug 10, 2017 at 6:33 AM, Daniel Stenberg <daniel_at_haxx.se> wrote:
>>
>>> On Wed, 9 Aug 2017, fan fan via curl-library wrote:
>>>
>>> In my application, I using libcurl to send 40 HTTP requests in every
>>>> one second. As I test, this consume much CPU, my application such as:
>>>>
>>>> main thread for sending request and using epoll listen fds whether read
>>>> or write.
>>>>
>>>> while(running) {
>>>> int ret = epoll_wait(epollFd, events, maxEventNum, 10); //10ms
>>>>
>>>
>>> Unconditionally returning every 10 milliseconds like this will make you
>>> spend a lot of CPU in vein. Don't do that. libcurl tells you how long it
>>> wants you to wait at most. No need to spin around a lap when there's
>>> nothing for libcurl to do.
>>>
>>> Do you also call libcurl to tell there's a timeout if the 10
>>> milliseconds lapse or when do you do that?
>>
>>
>> [Fan] Maybe this is not root cause, because when no http request
>> send, CPU almost 0.1%. of course I need to expand waiting time.
>> this epoll_wait for listening every easy handle fd, I
>> set CURLMOPT_SOCKETFUNCTION, when callback, I add this fd to epoll events.
>> of course I set CURLMOPT_TIMERFUNCTION, when timeout, I call curl_multi_socket_action
>> again.
>>
>>>
>>>
>>> for (i = 0; i < ret; i++) {
>>>> if (events[i].events & EPOLLIN) {
>>>> curl_multi_socket_action // set CURL_CSELECT_IN
>>>> curl_multi_info_read // for reading
>>>>
>>>
>>> I think you should put the curl_multi_info_read() call outside of the
>>> loop checking for activities, since it might not have anything to return
>>> and then its a bit of a waste to call it up to 40 times.
>>>
>>> But sure, curl_multi_socket_action() may of course consume CPU since
>>> that's the function that will handle the traffic on the socket you give it
>>> (and possibly on a few others if necessary).
>>
>>
>> [Fan] So, when do I call curl_multi_info_read?
>> I refer to:
>> https://curl.haxx.se/libcurl/c/hiperfifo.html
>> when socket have something to read, call
>> curl_multi_info_read, is it correct?
>>
>>>
>>>
>>> As I test, send 100 or 200 or 300 or 400 or 500 in parallel maybe
>>>> consume
>>>> cpu 100%.
>>>>
>>>
>>> On modern hardware that shouldn't happen.
>>
>> [Fan] Agree
>>
>>>
>>>
>>> I found there are too much EPOLLIN event, this consume much cpu.
>>>>
>>>
>>> What do you mean with too much EPOLLIN event? If libcurl tells you to
>>> wait for that, shouldn't you then be glad you get it so that you can
>>> download the data?
>>
>>
>> [Fan] I add socket fd to epoll event when CURLMOPT_SOCKETFUNCTION
>> callback.
>> If there has a lot of easy handle in processing, trigger
>> frequently.
>> CURLMOPT_TIMERFUNCTION not about epoll fd, is it right?
>> CURLMOPT_TIMERFUNCTION only indicating whether time out or
>> not.
>>
>>>
>>>
>>> 1. Does my use libcurl wrong?
>>>>
>>>
>>> Ineffective at least. But you also didn't show the whole program so its
>>> not possible to tell the whole story.
>>
>> [Fan] Agree
>>
>>>
>>>
>>> 2. Have any performance measurement for libcurl process in parallel?
>>>>
>>>
>>> It should easily handle thousands of parallel connections, CPU wise.
>>> Very likely tens of thousands too. Of course they may have to struggle with
>>> each other for other resources (like bandwidth).
>>
>> [Fan]Awesome
>>
>>>
>>>
>>> --
>>>
>>> / daniel.haxx.se
>>>
>>
>>
>

-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html

• application/octet-stream attachment: 40Req.perf