Re: Slow performance using multi_curl_* with HTTPs requests

From: Thomas Stähle via curl-and-php <curl-and-php_at_cool.haxx.se>
Date: Thu, 24 Jun 2021 14:32:25 +0200

Hi Jess,

thanks for your fast response.

Have you done the tests via command line without php? Can I see your php
> file content?
>

No. Since there is no way of using the multi curl interface via any
commandline tools without PHP, I know about, I was only able to execute a
bash script doing the 100 requests without any parallel requests. The
runtime is fine as it is with PHP without the parallel requests.


> I give you these 2 links, maybe can help you:
>
> http://www.onlineaspect.com/2009/01/26/how-to-use-curl_multi-without-blocking/
> https://webkul.com/blog/simultaneous-curl-requests-in-php/
>

I checked out the links. Everyone implements it a little bit differently
but it is very close to the implementations from the links. Since I really
really like the rolling implementation from onlineaspect.com I will do
further tests (to satisfy Daniels arguments) with this implementation.
Though, I don't need the rolling feature for the tests since I do only 10
requests in parallel and this is not that much to limit it down.

Here are my test implementations.

curl_plain.php:
$time = microtime(true);

if($argv[1] == 'http') {
    $url = 'http://.../sleep.php';
} else {
    $url = 'https://.../sleep.php';
}

for ($i = 0; $i < 100; $i++) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url . sprintf('?i=%s', $i));
    curl_setopt($ch, CURLOPT_RETURNTRANSFER , true);
    curl_exec($ch);
}

echo "Total: " . (microtime(true) - $time) . " seconds\n";

curl_multi.php:
$time = microtime(true);

if($argv[1] == 'http') {
    $url = 'http://.../sleep.php';
} else {
    $url = 'https://.../sleep.php';
}

function rolling_curl($urls, $callback, $custom_options = null) {
    ...
    $rolling_window = 100;
    ...
                // small fix here since we have actually less requests to
do as we have "slots"
                if(!empty($urls[$i + 1])) {
                    $ch = curl_init();
                    $options[CURLOPT_URL] = $urls[$i++]; // increment i
                    curl_setopt_array($ch, $options);
                    curl_multi_add_handle($master, $ch);
                }
    ...
}

// simulates 100 requests of our API, just to make the issues are more
significant and easier to see
for ($i = 0; $i < 100; $i++) {
    $handles = [];
    for($l = 0; $l < 10; $l++) {
        $handles[$l] = $url . sprintf('?i=%s&l=%s', $i, $l);
    }

    rolling_curl($handles, function() {});
}

echo "Total: " . (microtime(true) - $time) . " seconds\n";

I will post new numbers with state of the art versions as response to
Daniel.

Cheers
Thomas


_______________________________________________
https://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Received on 2021-06-24