Re: High latency with 100 active handles

From: Stefan Eissing via curl-library <curl-library_at_lists.haxx.se>
Date: Mon, 25 Nov 2024 12:12:47 +0100

Hi Avni,

there is no known slowdown in libcurl in regard to 50 vs 100 transfers in parallel. The first question is where the time is lost. Is curl using 100% cpu or is it waiting for data to arrive? Does the server maybe expose differences in handling 50 vs 100 transfers? Are the transfers done using HTTP/1.1 or HTTP/2? How many connections are involved here?

Have you tried to run the curl command line tool itself with 100 parallel transfers against the URL? Something like

> curl --parallel-max 100 --range 0-1024 https://host/?[0-100]

Also, there are quite some differences between curl 7.88 and 8.10. For analysis, it is be better to look at the 8.10 behaviour.

Best Regards,
Stefan

> Am 25.11.2024 um 11:14 schrieb Avni, Maor via curl-library <curl-library_at_lists.haxx.se>:
>
> Hi,
>
> We're having high latency issues when using CURL in our product. We've reduced the problem to the piece of code below.
> Github project: https://github.com/maor-avni-hpe/curlmultipoc
>
> Our product performs thousands of HTTP requests to Azure/AWS from a machine running on Azure/AWS. The transfer sizes vary from 128K to several MBs.
> We're using the multi API.
>
> The attached program accepts 5 arguments:
> * URL
> * Chunk size to download in K
> * Total GB to download
> * Batch Size
> * Maximum Open Handles
>
> * The program requests <batch size> downloads, each transfer <chunk size>, until reaching total GB.
> * The program makes sure that no more than <max open handles> are requested from CURL.
> * When a download completes its handle is removed and a new one is added until batch is completed.
>
> The program is executed on a VM running on Azure, downloading a file from an Azure storage account, to make sure that the network is as predictable as possible. The file downloaded is a 10GB file, although we always request the same first few bytes. We ran it both on Linux and Windows.
>
> Linux: Debian 12.8, curl 7.88
> Windows: Server 2022, curl 7.88, curl 8.10
>
> The results are for downloading a 5GB file using 128K chunks.
>
> The results:
> * single unlimited batch, 50 handles: Linux: 7s, Windows: 11s.
> * single unlimited batch, 100 handles: Linux: 6s, Windows 11s.
> * 50 per batch, 50 handles: Linux 12s, Windows 15s.
> * 100 per batch, 50 handles: Linux 9s, Windows 14s.
> * 100 per batch, 100 handles: Linux 36s, Windows 290s.
>
> Logging the remaining FDs after each poll shows that when running 100 handles there's 1 handle that takes longer to finish.
>
> Our product works in the 100/100 configuration. Under load we get very high latency and low data rates. This test explains why we get those slowdowns.
>
> Is there a reasonable explanation for this slowdown?
> Is it our usage of the curl API?
> What can be done to improve the performance?
>
> #include <iostream>
> #include <list>
> #include <set>
> #include <string>
> #include <thread>
> #include <vector>
> #include <curl/curl.h>
>
> struct Chunk
> {
> std::string data;
> size_t size;
> };
>
> size_t WriteCallback(void* contents, size_t size, size_t nmemb, void* userp)
> {
> size_t totalSize = size * nmemb;
> Chunk* chunk = (Chunk*)userp;
> // chunk->data.append((char *)contents, totalSize);
> chunk->size += totalSize;
> // sleep for 0.1ms
> return totalSize;
> }
>
> CURL* createCurlHandle(long start, long end, Chunk* chunk, std::string url)
> {
> CURL* curl = curl_easy_init();
> if (curl)
> {
> std::string range = std::to_string(start) + "-" + std::to_string(end);
> curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
> curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
> curl_easy_setopt(curl, CURLOPT_MAXREDIRS, 3L);
> curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1L);
> curl_easy_setopt(curl, CURLOPT_RANGE, range.c_str());
> curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
> curl_easy_setopt(curl, CURLOPT_WRITEDATA, chunk);
> curl_easy_setopt(curl, CURLOPT_BUFFERSIZE, 128L * 1024L);
> curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 1L); // Enable SSL verification
> curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 2L); // Verify the host
> }
> return curl;
>
> }
>
> int main(int argc, char* argv[])
> {
> if (argc != 6)
> {
> std::cerr << "Usage: " << argv[0] << " <URL> <Chunk Size> <Total GB> <Batch Size> <Number of Concurrent Connections>" << std::endl;
> return 1;
> }
>
> std::string url = argv[1];
> long chunkSize = std::stol(argv[2]) * 1024;
> long totalGB = std::stol(argv[3]);
> int batchSize = std::stoi(argv[4]);
> int numConnections = std::stoi(argv[5]);
>
> // Initialize CURL
> curl_global_init(CURL_GLOBAL_DEFAULT);
>
> // Get the total size of the file
> CURL* curl = curl_easy_init();
> double fileSize = 0.0;
> if (curl)
> {
> curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
> curl_easy_setopt(curl, CURLOPT_NOBODY, 1L);
> curl_easy_setopt(curl, CURLOPT_HEADER, 1L);
> curl_easy_setopt(curl, CURLOPT_FILETIME, 1L);
> curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
> curl_easy_setopt(curl, CURLOPT_MAXREDIRS, 3L);
> curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1L);
> curl_easy_setopt(curl, CURLOPT_BUFFERSIZE, 128L * 1024L);
> curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); // Enable SSL verification
> curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); // Verify the host
>
> CURLcode res = curl_easy_perform(curl);
> if (res == CURLE_OK)
> {
> curl_easy_getinfo(curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD, &fileSize);
> }
> else
> {
> // get the http status code
> long http_code = 0;
> curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code);
> std::cerr << "Failed to get file size: " << curl_easy_strerror(res) << " HTTP code: " << http_code << std::endl;
> return 1;
> }
>
> curl_easy_cleanup(curl);
> }
>
> long long numOfChunks = (long long)totalGB * 1024 * 1024 * 1024 / chunkSize;
> long long totalBatches;
> if (batchSize == 0)
> {
> batchSize = numOfChunks;
> totalBatches = 1;
> }
> else
> totalBatches = numOfChunks / batchSize;
>
> std::vector<Chunk> chunks;
> std::set<CURL*> handles;
> std::cout << "chunks: " << numOfChunks << std::endl;
>
> CURLM* multi_handle = curl_multi_init();
>
> // curl_multi_setopt(multi_handle, CURLMOPT_MAXCONNECTS, 0);
> // curl_multi_setopt(multi_handle, CURLMOPT_MAX_TOTAL_CONNECTIONS, 1);
> // curl_multi_setopt(multi_handle, CURLMOPT_PIPELINING, 0L);
>
> // measure time taken to download
> clock_t outerStart, outerEnd;
> outerStart = clock();
> std::cout << outerStart << std::endl;
>
> int still_running = 0;
> int maxHandles = 0;
> int maxPolledFd = 0;
> long long currentBatch = 0;
> do
> {
> do
> {
> // Add new connections
> while (handles.size() < numConnections && chunks.size() < batchSize)
> {
> long start = 0;
> long end = chunkSize - 1;
> if (end >= fileSize) end = fileSize - 1;
>
> Chunk chunk;
> auto handle = createCurlHandle(start, end, &chunk, url);
> handles.insert(handle);
> curl_multi_add_handle(multi_handle, handle);
> chunks.push_back(chunk);
> }
> if (handles.size() > maxHandles)
> {
> maxHandles = handles.size();
> }
>
> // Perform connections
> CURLMcode mc = curl_multi_perform(multi_handle, &still_running);
>
> // Wait for activity, timeout or "nothing"
> int currentPolledFd = 0;
> if (still_running)
> {
> // measure call time in milliseconds
> clock_t start, end;
> start = clock();
> mc = curl_multi_poll(multi_handle, NULL, 0, 1000, nullptr);
> end = clock();
> double time_taken = double(end - start) / double(CLOCKS_PER_SEC);
> //if (time_taken > 0.1)
> //std::cout << "Time taken to poll: " << time_taken << " seconds. Still running: " << still_running << std::endl;
>
> if (mc != CURLM_OK)
> {
> std::cerr << "curl_multi_poll failed: " << curl_multi_strerror(mc) << std::endl;
> }
>
> if (currentPolledFd > maxPolledFd)
> {
> maxPolledFd = currentPolledFd;
> }
> }
>
> int handlesBeforeClean = handles.size();
> while (CURLMsg* msg = curl_multi_info_read(multi_handle, &still_running))
> {
> if (msg->msg == CURLMSG_DONE)
> {
> // std::cout << "http request done" << std::endl;
> CURL* handle = msg->easy_handle;
> CURLcode result = msg->data.result;
> if (result != CURLE_OK)
> {
> std::cerr << "Download failed: " << curl_easy_strerror(result) << std::endl;
> }
>
> handles.erase(handle);
> curl_multi_remove_handle(multi_handle, handle);
> curl_easy_cleanup(handle);
> }
> }
>
> int handlesAfterClean = handles.size();
> // print handles before clean, after clean, and max on one line, but only if there isn't any change from previous iteration.
> // std::cout << "Handles before clean: " << handlesBeforeClean << " Handles after clean: " << handlesAfterClean << " Max handles: " << maxHandles << std::endl;
> } while (!(!still_running && handles.empty() && chunks.size() >= batchSize));
> ++currentBatch;
> chunks.clear();
> } while (currentBatch < totalBatches);
>
> outerEnd = clock();
> std::cout << outerEnd << std::endl;
> double time_taken = double(outerEnd - outerStart) / double(CLOCKS_PER_SEC);
> std::cout << "Time taken to download: " << time_taken << " seconds" << std::endl;
> // calculate download speed
> double downloadSpeed = (fileSize / 1024) / time_taken;
> std::cout << "Download speed: " << downloadSpeed << " KB/s" << std::endl;
> std::cout << "max polled fd: " << maxPolledFd << std::endl;
>
> curl_multi_cleanup(multi_handle);
> curl_global_cleanup();
>
> // Combine chunks
> std::string result;
> for (const auto& chunk : chunks)
> {
> result.append(chunk.data);
> }
>
> // Output the result
> std::cout << "Downloaded data: " << result << std::endl;
>
> return 0;
> }
>
>
> Maor Avni
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2024-11-25