Buy commercial curl support. We
help you work out your issues, debug your libcurl applications, use the API,
port to new platforms, add new features and more. With a team lead by the
curl founder Daniel himself.
Re: How to dry-run a http request
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Jeroen Ooms via curl-library <curl-library_at_lists.haxx.se>
Date: Sat, 15 Feb 2025 20:48:48 +0100
On Sat, Feb 15, 2025 at 5:17 PM Dan Fandrich via curl-library
<curl-library_at_lists.haxx.se> wrote:
>
> On Sat, Feb 15, 2025 at 04:21:04PM +0100, Jeroen Ooms via curl-library wrote:
> > For the R bindings, we have a popular debugging mode to 'dry-run' a
> > http request. By this we mean that the user builds and performs a
> > http-request in the regular way, but instead of actually connecting to
> > the requested host, we trick libcurl to make the http request to a
> > local server, which records the full payload that libcurl has
> > uploaded, and we return this blob to the user.
> >
> > Currently the way this is implemented is a bit cumbersome. We first
> > start a http-server in the background. Then we take the handle
> > provided by the user, and replace CURLOPT_URL to substitute the host
> > with "http://localhost". Then we modify CURLOPT_HTTPHEADER to set the
> > 'Host' request header to the host:port that was specified by the user
> > in the original url, and let libcurl perform the request. The fake
> > server always returns 200.
> >
> > Especially the part where we need to run our own http server is a bit
> > annoying. I was wondering if there would perhaps be a simpler method
> > trick libcurl to send a http request to a local file or buffer, from
> > where we can read it. Preferably something that we can do entirely in
> > C (things like nc are difficult to port).
>
> When curl is talking HTTP, it expects an HTTP server on the other end. Avoiding
> that is not going to be generally possible. But, if you're just interested in
> the client side of the transaction and stick with HTTP (not HTTPS) and HTTP
> 1.1 and avoid a negotiated authentication protocol, you might be able to get
> away with providing a CURLOPT_OPENSOCKETFUNCTION that points to a file and
> setting instantaneous CURLOPT_EXPECT_100_TIMEOUT_MS and
> CURLOPT_SERVER_RESPONSE_TIMEOUT timeouts. It's quite possible that libcurl
> won't like a file descriptor in place of a socket, but you might get lucky.
Thanks this is worth trying. I think it does not have to be a file
per-se; I could open a local socket and read that to a buffer while
curl is uploading.
The difficult part is that I need a callback function for the moment
where libcurl is done uploading (When curlopt_verbose says "Request
completely sent off") such that I know I have received full request
data and can close() the socket at that point.
Date: Sat, 15 Feb 2025 20:48:48 +0100
On Sat, Feb 15, 2025 at 5:17 PM Dan Fandrich via curl-library
<curl-library_at_lists.haxx.se> wrote:
>
> On Sat, Feb 15, 2025 at 04:21:04PM +0100, Jeroen Ooms via curl-library wrote:
> > For the R bindings, we have a popular debugging mode to 'dry-run' a
> > http request. By this we mean that the user builds and performs a
> > http-request in the regular way, but instead of actually connecting to
> > the requested host, we trick libcurl to make the http request to a
> > local server, which records the full payload that libcurl has
> > uploaded, and we return this blob to the user.
> >
> > Currently the way this is implemented is a bit cumbersome. We first
> > start a http-server in the background. Then we take the handle
> > provided by the user, and replace CURLOPT_URL to substitute the host
> > with "http://localhost". Then we modify CURLOPT_HTTPHEADER to set the
> > 'Host' request header to the host:port that was specified by the user
> > in the original url, and let libcurl perform the request. The fake
> > server always returns 200.
> >
> > Especially the part where we need to run our own http server is a bit
> > annoying. I was wondering if there would perhaps be a simpler method
> > trick libcurl to send a http request to a local file or buffer, from
> > where we can read it. Preferably something that we can do entirely in
> > C (things like nc are difficult to port).
>
> When curl is talking HTTP, it expects an HTTP server on the other end. Avoiding
> that is not going to be generally possible. But, if you're just interested in
> the client side of the transaction and stick with HTTP (not HTTPS) and HTTP
> 1.1 and avoid a negotiated authentication protocol, you might be able to get
> away with providing a CURLOPT_OPENSOCKETFUNCTION that points to a file and
> setting instantaneous CURLOPT_EXPECT_100_TIMEOUT_MS and
> CURLOPT_SERVER_RESPONSE_TIMEOUT timeouts. It's quite possible that libcurl
> won't like a file descriptor in place of a socket, but you might get lucky.
Thanks this is worth trying. I think it does not have to be a file
per-se; I could open a local socket and read that to a buffer while
curl is uploading.
The difficult part is that I need a callback function for the moment
where libcurl is done uploading (When curlopt_verbose says "Request
completely sent off") such that I know I have received full request
data and can close() the socket at that point.
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2025-02-15