Re: Force content decoding

From: Timothe Litt <litt_at_acm.org>
Date: Thu, 7 Jul 2022 05:39:53 -0400

It sounds like you're on the wrong track.

I'm not familiar with that service.  I would expect it to return what
you provide.

If you upload gzip'd data, the server should return gzip'd data - it's a
blob.  This makes sense if you want to minimize stored size on the server.

If you want to upload an uncompressed binary blob and get back the
uncompressed data, but use gzip on the wire - consider
transfer-encoding.  This will store the uncompressed data on the server
- which may be what you're trying to avoid.

Did you use the correct content-encoding (or transfer-encoding) headers
when you uploaded the blobs?

Are you specifying the correct accept-encoding header on the download?

As for changing incoming headers - I don't think that's doable, but in
any case, it's a bad idea.  "There be dragons there."

curl uses zlib for encoding/decoding, so aside from convenience, you
would get nothing from forcing a path thru the internal functions.


Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.

On 07-Jul-22 05:19, Wolf Vollprecht wrote:
> Hi Timothe,
>
> Thanks for your message. Indeed, I want to decode the data. The server
> is not broken (it’s the Github OCI registry server), I just cannot
> control what headers it sends for the binary blobs that I uploaded.
>
> Ideally I would have wanted to re-use the internal libcurl functions
> for decoding that would be used if the server would reply with the
> correct Content-Encoding vs implementing it in my WRITEFUNCTION.
>
> Best,
>
> Wolf
>
>
>> On 7. Jul 2022, at 11:15, Timothe Litt via curl-library
>> <curl-library_at_lists.haxx.se> wrote:
>>
>>
>> On 07-Jul-22 03:56, Wolf Vollprecht via curl-library wrote:
>>> Hi all,
>>>
>>> I have some data that I am downloading where I know that it is gzip-encoded. However, the server does not respond with the correct header (and I don’t have control over it). I was wondering if there is a way to either “manually” append the proper Content-Encoding header to the response or force libcurl to encode the response with gzip?
>>>
>>> Best,
>>>
>>> Wolf
>>
>> I assume you mean "decode", not "encode". After the mandatory lecture
>> about not tolerating broken servers...
>>
>> For the command line:
>>
>>     curl https://foo.example.com/bar | gunzip > data
>>
>> If you are using libcurl, depending on your requirements see zlib
>> (libz) with CURLOPT_WRITEFUNCTION or popen() gunzip (or gzip -d).
>>
>>
>> Timothe Litt
>> ACM Distinguished Engineer
>> --------------------------
>> This communication may not represent the ACM or my employer's views,
>> if any, on the matters discussed.
>>
>> --
>> Unsubscribe: https://lists.haxx.se/listinfo/curl-library
>> Etiquette: https://curl.se/mail/etiquette.html
>

-- 
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2022-07-07