Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: cloudflare
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Timothe Litt <litt_at_acm.org>
Date: Mon, 28 Feb 2022 15:31:19 -0500
On 28-Feb-22 13:29, Dennis Nezic via curl-users wrote:
> Any idea why I'm not able to fetch
> https://grandtheftworld.com/feed/podcast/
> using curl, but I am with wget?
>
> *Even when the http headers are exactly the same!?*
>
> For example, to match wget's headers, I ran the command:
>
> curl \
> -A whatever \
> --header "Accept-Encoding: identity" \
> --header "Connection: Keep-Alive" \
> https://grandtheftworld.com/feed/podcast/
>
> The headers are even in the same order:
>
> GET /feed/podast/ HTTP/1.1
> Host: grand...
> User-Agent: whatever
> Accept: */*
> Accept-Encoding: identity
> Connection: Keep-Alive
>
> But nasty cloudflare returns that nasty captcha/javascript webpage with
> curl, and the proper feed with wget.
>
> I tested with 7.79.1 and 7.81.0
Could be cloudflare checking the user-agent to allow some non-humans to
avoid captchas. (Which is odd, since normally the problem is non-humans
- but people aren't logical.)
Try --user-agent matching wget's.
If that works, open a ticket with Cloudflare. If they allow wget, they
should allow cURL.
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
Received on 2022-02-28
Date: Mon, 28 Feb 2022 15:31:19 -0500
On 28-Feb-22 13:29, Dennis Nezic via curl-users wrote:
> Any idea why I'm not able to fetch
> https://grandtheftworld.com/feed/podcast/
> using curl, but I am with wget?
>
> *Even when the http headers are exactly the same!?*
>
> For example, to match wget's headers, I ran the command:
>
> curl \
> -A whatever \
> --header "Accept-Encoding: identity" \
> --header "Connection: Keep-Alive" \
> https://grandtheftworld.com/feed/podcast/
>
> The headers are even in the same order:
>
> GET /feed/podast/ HTTP/1.1
> Host: grand...
> User-Agent: whatever
> Accept: */*
> Accept-Encoding: identity
> Connection: Keep-Alive
>
> But nasty cloudflare returns that nasty captcha/javascript webpage with
> curl, and the proper feed with wget.
>
> I tested with 7.79.1 and 7.81.0
Could be cloudflare checking the user-agent to allow some non-humans to
avoid captchas. (Which is odd, since normally the problem is non-humans
- but people aren't logical.)
Try --user-agent matching wget's.
If that works, open a ticket with Cloudflare. If they allow wget, they
should allow cURL.
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-users Etiquette: https://curl.haxx.se/mail/etiquette.html
- application/pgp-signature attachment: OpenPGP digital signature