Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: cloudflare
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Timothe Litt <litt_at_acm.org>
Date: Mon, 28 Feb 2022 16:47:27 -0500
On 28-Feb-22 15:37, Dan Fandrich via curl-users wrote:
> On Mon, Feb 28, 2022 at 01:29:33PM -0500, Dennis Nezic via curl-users wrote:
>> Any idea why I'm not able to fetch
>> https://grandtheftworld.com/feed/podcast/
>> using curl, but I am with wget?
>>
>> *Even when the http headers are exactly the same!?*
> In a quick test, curl puts the Host: header first whereas wget puts it
> second-to-last. That's a difference that could be used for browser
> fingerprinting.
Yes, but it doesn't make sense to allow self-identified automation a
pass but make life difficult for people. (Most bots pretend to be
browsers anyhow.)
Although header order is not functionally significant, cURL's order is
preferred per https://datatracker.ietf.org/doc/html/rfc7230#section-3.2.2
In any case, using curl --user-agent "Wget/1.13.4 (linux-gnu)" (or
whatever your current version of wget is) should be telling.
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
Received on 2022-02-28
Date: Mon, 28 Feb 2022 16:47:27 -0500
On 28-Feb-22 15:37, Dan Fandrich via curl-users wrote:
> On Mon, Feb 28, 2022 at 01:29:33PM -0500, Dennis Nezic via curl-users wrote:
>> Any idea why I'm not able to fetch
>> https://grandtheftworld.com/feed/podcast/
>> using curl, but I am with wget?
>>
>> *Even when the http headers are exactly the same!?*
> In a quick test, curl puts the Host: header first whereas wget puts it
> second-to-last. That's a difference that could be used for browser
> fingerprinting.
Yes, but it doesn't make sense to allow self-identified automation a
pass but make life difficult for people. (Most bots pretend to be
browsers anyhow.)
Although header order is not functionally significant, cURL's order is
preferred per https://datatracker.ietf.org/doc/html/rfc7230#section-3.2.2
In any case, using curl --user-agent "Wget/1.13.4 (linux-gnu)" (or
whatever your current version of wget is) should be telling.
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-users Etiquette: https://curl.haxx.se/mail/etiquette.html
- application/pgp-signature attachment: OpenPGP digital signature