Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: cloudflare
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Dennis Nezic via curl-users <curl-users_at_lists.haxx.se>
Date: Mon, 28 Feb 2022 16:47:52 -0500
On Mon, 28 Feb 2022 15:31:19 -0500, Timothe Litt via curl-users wrote:
>
> On 28-Feb-22 13:29, Dennis Nezic via curl-users wrote:
> > Any idea why I'm not able to fetch
> > https://grandtheftworld.com/feed/podcast/
> > using curl, but I am with wget?
> >
> > *Even when the http headers are exactly the same!?*
> >
> > For example, to match wget's headers, I ran the command:
> >
> > curl \
> > -A whatever \
> > --header "Accept-Encoding: identity" \
> > --header "Connection: Keep-Alive" \
> > https://grandtheftworld.com/feed/podcast/
> >
> > The headers are even in the same order:
> >
> > GET /feed/podast/ HTTP/1.1
> > Host: grand...
> > User-Agent: whatever
> > Accept: */*
> > Accept-Encoding: identity
> > Connection: Keep-Alive
> >
> > But nasty cloudflare returns that nasty captcha/javascript webpage
> > with curl, and the proper feed with wget.
> >
> > I tested with 7.79.1 and 7.81.0
>
> Could be cloudflare checking the user-agent to allow some non-humans
> to avoid captchas. (Which is odd, since normally the problem is
> non-humans
> - but people aren't logical.)
>
> Try --user-agent matching wget's.
I said the headers were EXACTLY the same :p. Both had a user agent of
"whatever".
Date: Mon, 28 Feb 2022 16:47:52 -0500
On Mon, 28 Feb 2022 15:31:19 -0500, Timothe Litt via curl-users wrote:
>
> On 28-Feb-22 13:29, Dennis Nezic via curl-users wrote:
> > Any idea why I'm not able to fetch
> > https://grandtheftworld.com/feed/podcast/
> > using curl, but I am with wget?
> >
> > *Even when the http headers are exactly the same!?*
> >
> > For example, to match wget's headers, I ran the command:
> >
> > curl \
> > -A whatever \
> > --header "Accept-Encoding: identity" \
> > --header "Connection: Keep-Alive" \
> > https://grandtheftworld.com/feed/podcast/
> >
> > The headers are even in the same order:
> >
> > GET /feed/podast/ HTTP/1.1
> > Host: grand...
> > User-Agent: whatever
> > Accept: */*
> > Accept-Encoding: identity
> > Connection: Keep-Alive
> >
> > But nasty cloudflare returns that nasty captcha/javascript webpage
> > with curl, and the proper feed with wget.
> >
> > I tested with 7.79.1 and 7.81.0
>
> Could be cloudflare checking the user-agent to allow some non-humans
> to avoid captchas. (Which is odd, since normally the problem is
> non-humans
> - but people aren't logical.)
>
> Try --user-agent matching wget's.
I said the headers were EXACTLY the same :p. Both had a user agent of
"whatever".
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-users Etiquette: https://curl.haxx.se/mail/etiquette.htmlReceived on 2022-02-28