Weird behavior when using DoH with the multi interface

From: kartatz via curl-library <curl-library_at_lists.haxx.se>
Date: Mon, 03 Jun 2024 15:47:27 -0300

I have a simple C program where I use the CURL multi interface to download a bunch of files. My code looks like this:

#include <stdlib.h>
#include <stdio.h>

#include <curl/curl.h>

int main(void) {
    
    int code = 0;
    size_t index = 0;
    
    int running = 1;
    CURLMsg* msg = NULL;
    int left = 0;
    
    CURL* curl = NULL;
    CURLM* curl_multi = NULL;
    
    curl = curl_easy_init();
    curl_multi = curl_multi_init();
    
    curl_easy_setopt(curl, CURLOPT_URL, "https://example.com/");
    curl_easy_setopt(curl, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
    
    curl_multi_setopt(curl_multi, CURLMOPT_MAX_HOST_CONNECTIONS, 5L);
    
    for (index = 0; index < 15; index++) {
        CURL* handle = curl_easy_duphandle(curl);
        curl_multi_add_handle(curl_multi, handle);
    }
    
    while (running) {
        curl_multi_perform(curl_multi, &running);
        
        if (running) {
            curl_multi_poll(curl_multi, NULL, 0, 1000, NULL);
        }
        
        while ((msg = curl_multi_info_read(curl_multi, &left))) {
            if (msg->msg != CURLMSG_DONE) {
                continue;
            }
            
            curl_multi_remove_handle(curl_multi, msg->easy_handle);
            
            printf("result: %i\n", (int) msg->data.result);
        }
    }
    
    return 0;
    
}

I firstly create an easy handle and then duplicate that handle with curl_easy_duphandle() for usage with the multi interface. For each handle, I set the CURLOPT_URL to the URL of the file I want to download.

I set a maximum number of host connections here (CURLMOPT_MAX_HOST_CONNECTIONS) because I don't want to cause overhead on the server.

This code does what I intended: download up to 5 files simultaneously.

However, things get a little weird if I set a CURLOPT_DOH_URL on the main easy handle:

...
curl_easy_setopt(curl, CURLOPT_DOH_URL, "https://dns.google/dns-query");
...

This causes the multi interface to hang indefinitely. I tried setting CURLOPT_VERBOSE to discover what's going on, and here are the logs:

* Found bundle for host: 0xb4000073436429d0 [serially]
* Found bundle for host: 0xb4000073436429d0 [serially]
* Found bundle for host: 0xb4000073436429d0 [serially]
* Found bundle for host: 0xb4000073436429d0 [serially]
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.

It then stays like this. No requests are performed.

After some tests, I found that setting CURLMOPT_MAX_HOST_CONNECTIONS to anything higher than 10 resolves this issue:

```
curl_multi_setopt(curl_multi, CURLMOPT_MAX_HOST_CONNECTIONS, 10L);
```

However, this setting causes the multi interface to make 10 requests simultaneously instead of just 5, which is undesired for my specific use case.

Here is the output of curl_version():

libcurl/8.7.1 OpenSSL/1.0.2k-fips zlib/1.2.7 brotli/1.0.9 libidn2/2.3.7 libpsl/0.7.0 (+libicu/50.1.2) libssh2/1.10.0 nghttp2/1.33.0 OpenLDAP/2.4.44

I'm on CentOS 7.

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2024-06-03