curl / Mailing Lists / curl-users / Single Mail
Buy commercial curl support from WolfSSL. We help you work out your issues, debug your libcurl applications, use the API, port to new platforms, add new features and more. With a team lead by the curl founder himself.

Re: cloudflare

From: Dennis Nezic via curl-users <curl-users_at_lists.haxx.se>
Date: Mon, 28 Feb 2022 16:47:52 -0500

On Mon, 28 Feb 2022 15:31:19 -0500, Timothe Litt via curl-users wrote:
>
> On 28-Feb-22 13:29, Dennis Nezic via curl-users wrote:
> > Any idea why I'm not able to fetch
> > https://grandtheftworld.com/feed/podcast/
> > using curl, but I am with wget?
> >
> > *Even when the http headers are exactly the same!?*
> >
> > For example, to match wget's headers, I ran the command:
> >
> > curl \
> > -A whatever \
> > --header "Accept-Encoding: identity" \
> > --header "Connection: Keep-Alive" \
> > https://grandtheftworld.com/feed/podcast/
> >
> > The headers are even in the same order:
> >
> > GET /feed/podast/ HTTP/1.1
> > Host: grand...
> > User-Agent: whatever
> > Accept: */*
> > Accept-Encoding: identity
> > Connection: Keep-Alive
> >
> > But nasty cloudflare returns that nasty captcha/javascript webpage
> > with curl, and the proper feed with wget.
> >
> > I tested with 7.79.1 and 7.81.0
>
> Could be cloudflare checking the user-agent to allow some non-humans
> to avoid captchas. (Which is odd, since normally the problem is
> non-humans
> - but people aren't logical.)
>
> Try --user-agent matching wget's.

I said the headers were EXACTLY the same :p. Both had a user agent of
"whatever".
-- 
Unsubscribe: https://lists.haxx.se/listinfo/curl-users
Etiquette:   https://curl.haxx.se/mail/etiquette.html
Received on 2022-02-28