curl / Mailing Lists / curl-library / Single Mail
Buy commercial curl support from WolfSSL. We help you work out your issues, debug your libcurl applications, use the API, port to new platforms, add new features and more. With a team lead by the curl founder himself.

Re: Force content decoding

From: Wolf Vollprecht via curl-library <curl-library_at_lists.haxx.se>
Date: Thu, 7 Jul 2022 12:42:16 +0200

I understand your concerns, but I think I know pretty well "what I am
doing" :)

For example, if you upload an gzipped file to an S3 bucket and then
retrieve it, it also won't automatically come with the right
Content-Encoding header.
On AWS, however, one can set it up in a way that the correct
Content-Encoding header is returned by manually adding the headers to the
object.

However, in my situation I have no way of setting up things so that the
proper header is returned (but I do know that the file is gzipped properly,
since I uploaded it).

But yeah, maybe the better solution is to unzip it "manually" (without
libcurl).

Best,

Wolf

Am Do., 7. Juli 2022 um 11:40 Uhr schrieb Timothe Litt via curl-library <
curl-library_at_lists.haxx.se>:

> It sounds like you're on the wrong track.
>
> I'm not familiar with that service. I would expect it to return what you
> provide.
>
> If you upload gzip'd data, the server should return gzip'd data - it's a
> blob. This makes sense if you want to minimize stored size on the server.
>
> If you want to upload an uncompressed binary blob and get back the
> uncompressed data, but use gzip on the wire - consider transfer-encoding.
> This will store the uncompressed data on the server - which may be what
> you're trying to avoid.
>
> Did you use the correct content-encoding (or transfer-encoding) headers
> when you uploaded the blobs?
>
> Are you specifying the correct accept-encoding header on the download?
>
> As for changing incoming headers - I don't think that's doable, but in any
> case, it's a bad idea. "There be dragons there."
>
> curl uses zlib for encoding/decoding, so aside from convenience, you would
> get nothing from forcing a path thru the internal functions.
>
>
> Timothe Litt
> ACM Distinguished Engineer
> --------------------------
> This communication may not represent the ACM or my employer's views,
> if any, on the matters discussed.
>
> On 07-Jul-22 05:19, Wolf Vollprecht wrote:
>
> Hi Timothe,
>
> Thanks for your message. Indeed, I want to decode the data. The server is
> not broken (it’s the Github OCI registry server), I just cannot control
> what headers it sends for the binary blobs that I uploaded.
>
> Ideally I would have wanted to re-use the internal libcurl functions for
> decoding that would be used if the server would reply with the correct
> Content-Encoding vs implementing it in my WRITEFUNCTION.
>
> Best,
>
> Wolf
>
>
> On 7. Jul 2022, at 11:15, Timothe Litt via curl-library <
> curl-library_at_lists.haxx.se> wrote:
>
>
> On 07-Jul-22 03:56, Wolf Vollprecht via curl-library wrote:
>
> Hi all,
>
> I have some data that I am downloading where I know that it is gzip-encoded. However, the server does not respond with the correct header (and I don’t have control over it). I was wondering if there is a way to either “manually” append the proper Content-Encoding header to the response or force libcurl to encode the response with gzip?
>
> Best,
>
> Wolf
>
> I assume you mean "decode", not "encode". After the mandatory lecture
> about not tolerating broken servers...
>
> For the command line:
>
> curl https://foo.example.com/bar | gunzip > data
>
> If you are using libcurl, depending on your requirements see zlib (libz)
> with CURLOPT_WRITEFUNCTION or popen() gunzip (or gzip -d).
>
>
> Timothe Litt
> ACM Distinguished Engineer
> --------------------------
> This communication may not represent the ACM or my employer's views,
> if any, on the matters discussed.
>
>
> --
> Unsubscribe: https://lists.haxx.se/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
>
>
> --
> Unsubscribe: https://lists.haxx.se/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
>


-- 
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html
Received on 2022-07-07