Improve cpu utilization

From: Nikos Dragazis via curl-library <curl-library_at_cool.haxx.se>
Date: Thu, 25 Mar 2021 22:23:25 +0200

Hi all,

While investigating the cpu utilization of our software, we believe that
we have spotted some inefficiencies in libcurl. Specifically, we have
seen that libcurl is issuing a lot of `poll(2)` system calls and this
increases the overall cpu utilization.

Here is some more context:

We are using libs3 [1] to read/write objects from/to an S3 bucket. libs3
uses libcurl to communicate with S3 via HTTP. Specifically, it spawns
multiple HTTP(S) connections to simultaneously and synchronously serve
S3 calls from multiple threads. libs3 interacts with libcurl via its
`easy` interface.  We have traced the "hot" code paths with `perf(1)`
and we have seen that 1/4 of the overall cpu utilization comes from
libcurl's `poll(2)` system calls. Specifically, the code paths to blame
are:

* `easy_transfer()` -> `curl_multi_poll()` -> `multi_wait()` -> `Curl_poll()`
* `Curl_readwrite()` -> `Curl_socket_check()` -> `Curl_poll()`

We have also confirmed that we can improve the cpu utilization with
respect to bandwidth by applying the following changes:

1. Remove `poll(2)` system calls from the critical path, that is from
    `easy_transfer()` and `Curl_readwrite()`.
2. Make connection sockets blocking instead of non-blocking.
3. Force `Curl_readwrite()` to ignore `KEEP_RECV` if `KEEP_SEND` is set.
4. Use socket send/recv timeouts to control the time interval we can get
    blocked on send/recv system calls on blocking sockets.

We know though that the above changes have the following tradeoffs:

1. The client (i.e., function `Curl_readwrite()`) cannot receive
    responses from the server while sending data.
2. Making the connection sockets blocking breaks the non-blocking nature
    of the public function `curl_multi_perform()`.

That said, we believe that it would make sense to introduce ablocking
mode/feature in libcurl with no polling in the critical pathfor
performance reasons.

Looking forward to your feedback.

--Nikos

[1] https://github.com/bji/libs3
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Received on 2021-03-25