Buy commercial curl support. We
help you work out your issues, debug your libcurl applications, use the API,
port to new platforms, add new features and more. With a team lead by the
curl founder Daniel himself.
Re: Feature request: data stream upload after server response
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Hans Henrik Bergan via curl-users <curl-users_at_lists.haxx.se>
Date: Mon, 27 May 2024 20:50:47 +0200
Rsync
On Mon, May 27, 2024, 17:07 fungs via curl-users <curl-users_at_lists.haxx.se>
wrote:
> Hey,
>
> I'm using curl (command line version) to upload a continuous data stream
> to a
> remote server like
>
> generate_data | curl --show-error --fail --silent -T . REMOTE_URL
>
> That command generates a (PUT) request with chunked transfer encoding to
> stream
> the data. I observed that streaming large data is not a problem in a
> regular
> setup, but on some cloud providers and network topologies, the request is
> passed
> between proxies for various purposes, for instance authentication,
> geographic
> routing or load balancing. These setups often put size or time
> restrictions on
> the requests and may fail if the data is too large, either because it
> cannot be
> buffered or because the processing of such requests takes too long, even
> if only
> authorization headers are being checked.
>
> In such a scenario, I found myself writing a naive Bourne shell function
> to wait
> for the final destination server reply before sending the actual data
> stream.
>
> send() {
> data_fifo="$chunk_identifier"_data.fifo
> header_fifo="$chunk_identifier"_response.fifo
> mkfifo"$data_fifo""$header_fifo"
>
> cat"$data_fifo"|curl--show-error--fail--silent--dump-header"$header_fifo"-T."$REMOTE_URL"&
> curl_pid=$!
> response_http_code="$({ read -r line && echo $line | cut -d ' ' -f 2; } <
> "$header_fifo")"
> if[ "$response_http_code" -ne200]; then
> echo"Server response for chunk '$1' is not 200: $response_http_code"
> return1
> fi
> cat>"$data_fifo"
> wait"$curl_pid"
> rm"$data_fifo" "$header_fifo"
> }
> Basically, this emulates a functionality in curl to wait before sending a
> continuous data stream until it has gotten a first response 200 from the
> server.
> This way, all the authentication and routing has been done by involved
> proxies
> and the data, using a tiny request, before the data gets sent.
> I'd propose this as a feature flag in curl and would like to discuss it
> here.
> Best
> Johannes
>
>
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
> Etiquette: https://curl.se/mail/etiquette.html
>
Date: Mon, 27 May 2024 20:50:47 +0200
Rsync
On Mon, May 27, 2024, 17:07 fungs via curl-users <curl-users_at_lists.haxx.se>
wrote:
> Hey,
>
> I'm using curl (command line version) to upload a continuous data stream
> to a
> remote server like
>
> generate_data | curl --show-error --fail --silent -T . REMOTE_URL
>
> That command generates a (PUT) request with chunked transfer encoding to
> stream
> the data. I observed that streaming large data is not a problem in a
> regular
> setup, but on some cloud providers and network topologies, the request is
> passed
> between proxies for various purposes, for instance authentication,
> geographic
> routing or load balancing. These setups often put size or time
> restrictions on
> the requests and may fail if the data is too large, either because it
> cannot be
> buffered or because the processing of such requests takes too long, even
> if only
> authorization headers are being checked.
>
> In such a scenario, I found myself writing a naive Bourne shell function
> to wait
> for the final destination server reply before sending the actual data
> stream.
>
> send() {
> data_fifo="$chunk_identifier"_data.fifo
> header_fifo="$chunk_identifier"_response.fifo
> mkfifo"$data_fifo""$header_fifo"
>
> cat"$data_fifo"|curl--show-error--fail--silent--dump-header"$header_fifo"-T."$REMOTE_URL"&
> curl_pid=$!
> response_http_code="$({ read -r line && echo $line | cut -d ' ' -f 2; } <
> "$header_fifo")"
> if[ "$response_http_code" -ne200]; then
> echo"Server response for chunk '$1' is not 200: $response_http_code"
> return1
> fi
> cat>"$data_fifo"
> wait"$curl_pid"
> rm"$data_fifo" "$header_fifo"
> }
> Basically, this emulates a functionality in curl to wait before sending a
> continuous data stream until it has gotten a first response 200 from the
> server.
> This way, all the authentication and routing has been done by involved
> proxies
> and the data, using a tiny request, before the data gets sent.
> I'd propose this as a feature flag in curl and would like to discuss it
> here.
> Best
> Johannes
>
>
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
> Etiquette: https://curl.se/mail/etiquette.html
>
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2024-05-27