Re: How to dry-run a http request

From: Stefan Eissing via curl-library <curl-library_at_lists.haxx.se>
Date: Sat, 15 Feb 2025 17:32:24 +0100

Hi Jeroen,

> Am 15.02.2025 um 16:21 schrieb Jeroen Ooms via curl-library <curl-library_at_lists.haxx.se>:
>
> For the R bindings, we have a popular debugging mode to 'dry-run' a
> http request. By this we mean that the user builds and performs a
> http-request in the regular way, but instead of actually connecting to
> the requested host, we trick libcurl to make the http request to a
> local server, which records the full payload that libcurl has
> uploaded, and we return this blob to the user.
>
> Currently the way this is implemented is a bit cumbersome. We first
> start a http-server in the background. Then we take the handle
> provided by the user, and replace CURLOPT_URL to substitute the host
> with "http://localhost". Then we modify CURLOPT_HTTPHEADER to set the
> 'Host' request header to the host:port that was specified by the user
> in the original url, and let libcurl perform the request. The fake
> server always returns 200.

To make curl use another address, you can add `--resolve dnsname:port:ipaddress`,
so you can start a server on localhost, you can run a curl command using
a domain name of your choice as, like "test.mydomain.net":

> curl --resolve test.mydomain.net:80:127.0.0.1

You can do this directly with libcurl, which is what `curl` does as well:

struct curl_slist *resolves;
resolves = curl_slist_append(NULL, "test.mydomain.net:80:127.0.0.1");
...
curl_easy_setopt(easy, CURLOPT_RESOLVE, resolves);

So, you do not have to fiddle with changes to urls or "Host:" headers.
This what curl uses in its test suite as well. (If you need to use different
ports, there is `--connect-to` which sets CURLOPT_CONNECT_TO in a similar
fashion.)

> Especially the part where we need to run our own http server is a bit
> annoying. I was wondering if there would perhaps be a simpler method
> trick libcurl to send a http request to a local file or buffer, from
> where we can read it. Preferably something that we can do entirely in
> C (things like nc are difficult to port).

There is no mechanism in libcurl to make requests to a local file or buffer. Since the whole communication handling relies on sockets and events, a change here would be far from simple.

Also, such a method would be very limited, usable only for HTTP/1.1 without SSL and even uploads expecting a "100 Continue" intermediate response from the server would not work. So, from my personal point of view, it seems not worth to invest in this.

That is also the reason, we use "real" servers, like Apache httpd, nghttpx and Caddy, in our pytest suite. We need to verify things with complete server behaviour, for all HTTP versions, at least on those platforms where these servers are easily available.

Cheers,
Stefan

> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2025-02-15