Slow performance using multi_curl_* with HTTPs requests

From: Thomas Stähle <staelche_at_gmail.com>
Date: Wed, 23 Jun 2021 10:39:37 +0200

Hi,

I am experiencing performance issues using multi curl combined with HTTPs
requests on a lately updated OS. It seems only related to multi curl and
HTTPs. Single, serialized, curl requests with HTTPs do not show any slow
down. So it seems not to be related to e.g. the OpenSSL version or the
SSL/TLS Handshake. I am not quite sure if this is related to the PHP side
of the multi curl implementation or libcurl multi curl interface...

I ran some tests to track the problem down but until now without any
success.

The tests I did do either 100 requests in a row (no multi curl; script name
curl_plain.php) or 100 times 10 requests in parallel with multi curl
(curl_multi.php). Both call a PHP script just sleeping for 1 second. One
time I called it via HTTP the other time via HTTPs. I ran all the tests on
both nodes. One node is on Debian Jessie (the old node). The other one is
on Debian Stretch (the new node). The following run times are all above 100
seconds since it can't get faster with a sleep of 1 second called 100
times. The interesting part is the overhead above the 100 seconds.

The slow down happened with the upgrade from Jessie to Stretch. I tested it
with Buster and with PHP7.4, as well. All combinations show the same issue.
So it does not go away with any further upgrade.

Since I am heavily relying on a performant multi_curl for some processes it
is necessary for me to resolve the performance issue.

The old node:
Debian Jessie (8.6)
PHP 7.1.17-1+0~20180505045956.17+jessie~1.gbpde69c6
OpenSSL 1.0.1t
libcurl(3) 7.38.0-4+deb8u4

The new node:
Debian Stretch (9.13)
PHP 7.1.33-37+0~20210501.55+debian9~1.gbpcb9127
OpenSSL 1.1.0l
libcurl(3) 7.52.1-5+deb9u13

Both nodes are exactly the same hardware wise. It is a dual CPU system with
NUMA activated to share the memory between both sockets. Both nodes are
located in the same network and call, of course, the same script on another
node in a different network.

The tests with HTTP:
old $ php curl_plain.php
   Total: 101.47104287148 seconds
old $ php curl_multi.php
   Total: 101.5175549984 seconds
new $ php curl_plain.php
   Total: 101.44491195679 seconds
new $ php curl_multi.php
   Total: 101.5433011055 seconds
-> no issue

The tests with HTTPs:
old$ php src/curl_plain.php
  Total: 103.15611720085 seconds
old$ php src/curl_multi.php
  Total: 104.17670083046 seconds
new$ php src/curl_plain.php
  Total: 103.91886115074 seconds
new$ php src/curl_multi.php
  Total: 110.56924200058 seconds
-> It makes sense that with multi_curl there is a little bit more overhead
but the big gap on the new node is a little bit too much I would say.

Does anyone have an idea what the issue could be?

Kind regards
Thomas


_______________________________________________
https://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Received on 2021-06-23