Re: Force content decoding

From: Wolf Vollprecht via curl-library <curl-library_at_lists.haxx.se>
Date: Thu, 7 Jul 2022 12:42:16 +0200

I understand your concerns, but I think I know pretty well "what I am
doing" :)

For example, if you upload an gzipped file to an S3 bucket and then
retrieve it, it also won't automatically come with the right
Content-Encoding header.
On AWS, however, one can set it up in a way that the correct
Content-Encoding header is returned by manually adding the headers to the
object.

However, in my situation I have no way of setting up things so that the
proper header is returned (but I do know that the file is gzipped properly,
since I uploaded it).

But yeah, maybe the better solution is to unzip it "manually" (without
libcurl).

Best,

Wolf

Am Do., 7. Juli 2022 um 11:40 Uhr schrieb Timothe Litt via curl-library <
curl-library_at_lists.haxx.se>:

> It sounds like you're on the wrong track.
>
> I'm not familiar with that service. I would expect it to return what you
> provide.
>
> If you upload gzip'd data, the server should return gzip'd data - it's a
> blob. This makes sense if you want to minimize stored size on the server.
>
> If you want to upload an uncompressed binary blob and get back the
> uncompressed data, but use gzip on the wire - consider transfer-encoding.
> This will store the uncompressed data on the server - which may be what
> you're trying to avoid.
>
> Did you use the correct content-encoding (or transfer-encoding) headers
> when you uploaded the blobs?
>
> Are you specifying the correct accept-encoding header on the download?
>
> As for changing incoming headers - I don't think that's doable, but in any
> case, it's a bad idea. "There be dragons there."
>
> curl uses zlib for encoding/decoding, so aside from convenience, you would
> get nothing from forcing a path thru the internal functions.
>
>
> Timothe Litt
> ACM Distinguished Engineer
> --------------------------
> This communication may not represent the ACM or my employer's views,
> if any, on the matters discussed.
>
> On 07-Jul-22 05:19, Wolf Vollprecht wrote:
>
> Hi Timothe,
>
> Thanks for your message. Indeed, I want to decode the data. The server is
> not broken (it’s the Github OCI registry server), I just cannot control
> what headers it sends for the binary blobs that I uploaded.
>
> Ideally I would have wanted to re-use the internal libcurl functions for
> decoding that would be used if the server would reply with the correct
> Content-Encoding vs implementing it in my WRITEFUNCTION.
>
> Best,
>
> Wolf
>
>
> On 7. Jul 2022, at 11:15, Timothe Litt via curl-library <
> curl-library_at_lists.haxx.se> wrote:
>
>
> On 07-Jul-22 03:56, Wolf Vollprecht via curl-library wrote:
>
> Hi all,
>
> I have some data that I am downloading where I know that it is gzip-encoded. However, the server does not respond with the correct header (and I don’t have control over it). I was wondering if there is a way to either “manually” append the proper Content-Encoding header to the response or force libcurl to encode the response with gzip?
>
> Best,
>
> Wolf
>
> I assume you mean "decode", not "encode". After the mandatory lecture
> about not tolerating broken servers...
>
> For the command line:
>
> curl https://foo.example.com/bar | gunzip > data
>
> If you are using libcurl, depending on your requirements see zlib (libz)
> with CURLOPT_WRITEFUNCTION or popen() gunzip (or gzip -d).
>
>
> Timothe Litt
> ACM Distinguished Engineer
> --------------------------
> This communication may not represent the ACM or my employer's views,
> if any, on the matters discussed.
>
>
> --
> Unsubscribe: https://lists.haxx.se/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
>
>
> --
> Unsubscribe: https://lists.haxx.se/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
>

-- 
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2022-07-07