curl / Mailing Lists / curl-library / Single Mail

curl-library

Using libcurl sending much more HTTP request in parallel

From: fan fan via curl-library <curl-library_at_cool.haxx.se>
Date: Wed, 9 Aug 2017 16:44:43 +0800

Hi all,
   This is my fist post to this list, so I want to use this opportunity to
thanks all of you (mainly Daniel) for the development, maintenance and
support efforts you invest on this amazing tool "cUrl".
    In my application, I using libcurl to send 40 HTTP requests in every
one second.
As I test, this consume much CPU, my application such as:

    main thread for sending request and using epoll listen fds whether read
or write.

    while(running) {
          int ret = epoll_wait(epollFd, events, maxEventNum, 10); //10ms

          for (i = 0; i < ret; i++) {
               if (events[i].events & EPOLLIN) {
                      curl_multi_socket_action // set CURL_CSELECT_IN
                      curl_multi_info_read // for reading
                      //consume much cpu
               }

              if (events[i].events & EPOLLOUT) {
                   curl_multi_socket_action // set CURL_CSELECT_OUT
              }
          }

          //Sending 40 http request in every one second in parallel
     }

    main thread add every easy handle fd to epoll.

As I test, send 100 or 200 or 300 or 400 or 500 in parallel maybe consume
cpu 100%.
I found there are too much EPOLLIN event, this consume much cpu.
By the way, I use HTTPS.

I have questions:
1. Does my use libcurl wrong?
2. Have any performance measurement for libcurl process in parallel?
3. Any suggestion?

-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2017-08-09