Feature request: data stream upload after server response

From: fungs via curl-users <curl-users_at_lists.haxx.se>
Date: Mon, 27 May 2024 13:58:40 +0000

Hey,

I'm using curl (command line version) to upload a continuous data stream to a
remote server like

     generate_data | curl --show-error --fail --silent -T . REMOTE_URL

That command generates a (PUT) request with chunked transfer encoding to stream
the data. I observed that streaming large data is not a problem in a regular
setup, but on some cloud providers and network topologies, the request is passed
between proxies for various purposes, for instance authentication, geographic
routing or load balancing. These setups often put size or time restrictions on
the requests and may fail if the data is too large, either because it cannot be
buffered or because the processing of such requests takes too long, even if only
authorization headers are being checked.

In such a scenario, I found myself writing a naive Bourne shell function to wait
for the final destination server reply before sending the actual data stream.

send() {
data_fifo="$chunk_identifier"_data.fifo
header_fifo="$chunk_identifier"_response.fifo
mkfifo"$data_fifo""$header_fifo"
cat"$data_fifo"|curl--show-error--fail--silent--dump-header"$header_fifo"-T."$REMOTE_URL"&
curl_pid=$!
response_http_code="$({ read -r line && echo $line | cut -d ' ' -f 2; } <
"$header_fifo")"
if[ "$response_http_code" -ne200]; then
echo"Server response for chunk '$1' is not 200: $response_http_code"
return1
fi
cat>"$data_fifo"
wait"$curl_pid"
rm"$data_fifo" "$header_fifo"
}
Basically, this emulates a functionality in curl to wait before sending a
continuous data stream until it has gotten a first response 200 from the server.
This way, all the authentication and routing has been done by involved proxies
and the data, using a tiny request, before the data gets sent.
I'd propose this as a feature flag in curl and would like to discuss it here.
Best
Johannes

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2024-05-27