Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: cloudflare
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Ray Satiro via curl-users <curl-users_at_lists.haxx.se>
Date: Mon, 28 Feb 2022 17:29:36 -0500
On 2/28/2022 4:47 PM, Dennis Nezic via curl-users wrote:
> On Mon, 28 Feb 2022 15:31:19 -0500, Timothe Litt via curl-users wrote:
>> On 28-Feb-22 13:29, Dennis Nezic via curl-users wrote:
>>> Any idea why I'm not able to fetch
>>> https://grandtheftworld.com/feed/podcast/
>>> using curl, but I am with wget?
>>>
>>> *Even when the http headers are exactly the same!?*
>>>
>>> For example, to match wget's headers, I ran the command:
>>>
>>> curl \
>>> -A whatever \
>>> --header "Accept-Encoding: identity" \
>>> --header "Connection: Keep-Alive" \
>>> https://grandtheftworld.com/feed/podcast/
>>>
>>> The headers are even in the same order:
>>>
>>> GET /feed/podast/ HTTP/1.1
>>> Host: grand...
>>> User-Agent: whatever
>>> Accept: */*
>>> Accept-Encoding: identity
>>> Connection: Keep-Alive
>>>
>>> But nasty cloudflare returns that nasty captcha/javascript webpage
>>> with curl, and the proper feed with wget.
>>>
>>> I tested with 7.79.1 and 7.81.0
>> Could be cloudflare checking the user-agent to allow some non-humans
>> to avoid captchas. (Which is odd, since normally the problem is
>> non-humans
>> - but people aren't logical.)
>>
>> Try --user-agent matching wget's.
> I said the headers were EXACTLY the same :p. Both had a user agent of
> "whatever".
This has come up before with other websites and we thought it was
cloudflare blocking some uncommon SSL handshakes to prevent abuse.
There's nothing we can do about it unless we support spoofed handshakes,
which I really don't think is worth it. [1]
[1]: https://github.com/curl/curl/issues/8119
Date: Mon, 28 Feb 2022 17:29:36 -0500
On 2/28/2022 4:47 PM, Dennis Nezic via curl-users wrote:
> On Mon, 28 Feb 2022 15:31:19 -0500, Timothe Litt via curl-users wrote:
>> On 28-Feb-22 13:29, Dennis Nezic via curl-users wrote:
>>> Any idea why I'm not able to fetch
>>> https://grandtheftworld.com/feed/podcast/
>>> using curl, but I am with wget?
>>>
>>> *Even when the http headers are exactly the same!?*
>>>
>>> For example, to match wget's headers, I ran the command:
>>>
>>> curl \
>>> -A whatever \
>>> --header "Accept-Encoding: identity" \
>>> --header "Connection: Keep-Alive" \
>>> https://grandtheftworld.com/feed/podcast/
>>>
>>> The headers are even in the same order:
>>>
>>> GET /feed/podast/ HTTP/1.1
>>> Host: grand...
>>> User-Agent: whatever
>>> Accept: */*
>>> Accept-Encoding: identity
>>> Connection: Keep-Alive
>>>
>>> But nasty cloudflare returns that nasty captcha/javascript webpage
>>> with curl, and the proper feed with wget.
>>>
>>> I tested with 7.79.1 and 7.81.0
>> Could be cloudflare checking the user-agent to allow some non-humans
>> to avoid captchas. (Which is odd, since normally the problem is
>> non-humans
>> - but people aren't logical.)
>>
>> Try --user-agent matching wget's.
> I said the headers were EXACTLY the same :p. Both had a user agent of
> "whatever".
This has come up before with other websites and we thought it was
cloudflare blocking some uncommon SSL handshakes to prevent abuse.
There's nothing we can do about it unless we support spoofed handshakes,
which I really don't think is worth it. [1]
[1]: https://github.com/curl/curl/issues/8119
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-users Etiquette: https://curl.haxx.se/mail/etiquette.htmlReceived on 2022-02-28