Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: cloudflare
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Timothe Litt <litt_at_acm.org>
Date: Mon, 28 Feb 2022 17:07:07 -0500
On 28-Feb-22 16:47, Dennis Nezic wrote:
> On Mon, 28 Feb 2022 15:31:19 -0500, Timothe Litt via curl-users wrote:
>> On 28-Feb-22 13:29, Dennis Nezic via curl-users wrote:
>>> Any idea why I'm not able to fetch
>>> https://grandtheftworld.com/feed/podcast/
>>> using curl, but I am with wget?
>>>
>>> *Even when the http headers are exactly the same!?*
>>>
>>> For example, to match wget's headers, I ran the command:
>>>
>>> curl \
>>> -A whatever \
>>> --header "Accept-Encoding: identity" \
>>> --header "Connection: Keep-Alive" \
>>> https://grandtheftworld.com/feed/podcast/
>>>
>>> The headers are even in the same order:
>>>
>>> GET /feed/podast/ HTTP/1.1
>>> Host: grand...
>>> User-Agent: whatever
>>> Accept: */*
>>> Accept-Encoding: identity
>>> Connection: Keep-Alive
>>>
>>> But nasty cloudflare returns that nasty captcha/javascript webpage
>>> with curl, and the proper feed with wget.
>>>
>>> I tested with 7.79.1 and 7.81.0
>> Could be cloudflare checking the user-agent to allow some non-humans
>> to avoid captchas. (Which is odd, since normally the problem is
>> non-humans
>> - but people aren't logical.)
>>
>> Try --user-agent matching wget's.
> I said the headers were EXACTLY the same :p. Both had a user agent of
> "whatever".
Sorry I missed the -A for --user-agent.
Clearly, something is different.
Is that the entire command - e.g. is there anything in .curlrc?
Could be hostname resolution or cloudflare request distributor
difference. With --verbose - are you connecting to the same IP address
for cloudflare? In the response headers, is it the same cloudflare server?
If that doesn't show the difference, next step would be a packet trace.
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
Received on 2022-02-28
Date: Mon, 28 Feb 2022 17:07:07 -0500
On 28-Feb-22 16:47, Dennis Nezic wrote:
> On Mon, 28 Feb 2022 15:31:19 -0500, Timothe Litt via curl-users wrote:
>> On 28-Feb-22 13:29, Dennis Nezic via curl-users wrote:
>>> Any idea why I'm not able to fetch
>>> https://grandtheftworld.com/feed/podcast/
>>> using curl, but I am with wget?
>>>
>>> *Even when the http headers are exactly the same!?*
>>>
>>> For example, to match wget's headers, I ran the command:
>>>
>>> curl \
>>> -A whatever \
>>> --header "Accept-Encoding: identity" \
>>> --header "Connection: Keep-Alive" \
>>> https://grandtheftworld.com/feed/podcast/
>>>
>>> The headers are even in the same order:
>>>
>>> GET /feed/podast/ HTTP/1.1
>>> Host: grand...
>>> User-Agent: whatever
>>> Accept: */*
>>> Accept-Encoding: identity
>>> Connection: Keep-Alive
>>>
>>> But nasty cloudflare returns that nasty captcha/javascript webpage
>>> with curl, and the proper feed with wget.
>>>
>>> I tested with 7.79.1 and 7.81.0
>> Could be cloudflare checking the user-agent to allow some non-humans
>> to avoid captchas. (Which is odd, since normally the problem is
>> non-humans
>> - but people aren't logical.)
>>
>> Try --user-agent matching wget's.
> I said the headers were EXACTLY the same :p. Both had a user agent of
> "whatever".
Sorry I missed the -A for --user-agent.
Clearly, something is different.
Is that the entire command - e.g. is there anything in .curlrc?
Could be hostname resolution or cloudflare request distributor
difference. With --verbose - are you connecting to the same IP address
for cloudflare? In the response headers, is it the same cloudflare server?
If that doesn't show the difference, next step would be a packet trace.
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-users Etiquette: https://curl.haxx.se/mail/etiquette.html
- application/pgp-signature attachment: OpenPGP digital signature