Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: cloudflare
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Dennis Nezic via curl-users <curl-users_at_lists.haxx.se>
Date: Mon, 28 Feb 2022 21:57:03 -0500
On Mon, 28 Feb 2022 17:29:36 -0500, Ray Satiro via curl-users wrote:
> On 2/28/2022 4:47 PM, Dennis Nezic via curl-users wrote:
> > On Mon, 28 Feb 2022 15:31:19 -0500, Timothe Litt via curl-users
> > wrote:
> >> On 28-Feb-22 13:29, Dennis Nezic via curl-users wrote:
> >>> Any idea why I'm not able to fetch
> >>> https://grandtheftworld.com/feed/podcast/
> >>> using curl, but I am with wget?
> >>>
> >>> *Even when the http headers are exactly the same!?*
> >>>
> >>> For example, to match wget's headers, I ran the command:
> >>>
> >>> curl \
> >>> -A whatever \
> >>> --header "Accept-Encoding: identity" \
> >>> --header "Connection: Keep-Alive" \
> >>> https://grandtheftworld.com/feed/podcast/
> >>>
> >>> The headers are even in the same order:
> >>>
> >>> GET /feed/podast/ HTTP/1.1
> >>> Host: grand...
> >>> User-Agent: whatever
> >>> Accept: */*
> >>> Accept-Encoding: identity
> >>> Connection: Keep-Alive
> >>>
> >>> But nasty cloudflare returns that nasty captcha/javascript webpage
> >>> with curl, and the proper feed with wget.
> >>>
> >>> I tested with 7.79.1 and 7.81.0
> >> Could be cloudflare checking the user-agent to allow some
> >> non-humans to avoid captchas. (Which is odd, since normally the
> >> problem is non-humans
> >> - but people aren't logical.)
> >>
> >> Try --user-agent matching wget's.
> > I said the headers were EXACTLY the same :p. Both had a user agent
> > of "whatever".
>
>
> This has come up before with other websites and we thought it was
> cloudflare blocking some uncommon SSL handshakes to prevent abuse.
> There's nothing we can do about it unless we support spoofed
> handshakes, which I really don't think is worth it. [1]
>
> [1]: https://github.com/curl/curl/issues/8119
Thanks for the info/link.
FWIW, I was able to fetch the feed using curl in php with only 3 of the
available CURLOPT_SSLVERSION's:
n CURL_SSLVERSION_TLSv1_0
n CURL_SSLVERSION_TLSv1_1
n CURL_SSLVERSION_TLSv1_2
y CURL_SSLVERSION_TLSv1_3
y CURL_SSLVERSION_MAX_TLSv1_0
y CURL_SSLVERSION_MAX_TLSv1_1
n CURL_SSLVERSION_MAX_TLSv1_2
n CURL_SSLVERSION_MAX_TLSv1_3
n CURL_SSLVERSION_SSLv3
And with CURLOPT_USERAGENT = 'whatever'
Date: Mon, 28 Feb 2022 21:57:03 -0500
On Mon, 28 Feb 2022 17:29:36 -0500, Ray Satiro via curl-users wrote:
> On 2/28/2022 4:47 PM, Dennis Nezic via curl-users wrote:
> > On Mon, 28 Feb 2022 15:31:19 -0500, Timothe Litt via curl-users
> > wrote:
> >> On 28-Feb-22 13:29, Dennis Nezic via curl-users wrote:
> >>> Any idea why I'm not able to fetch
> >>> https://grandtheftworld.com/feed/podcast/
> >>> using curl, but I am with wget?
> >>>
> >>> *Even when the http headers are exactly the same!?*
> >>>
> >>> For example, to match wget's headers, I ran the command:
> >>>
> >>> curl \
> >>> -A whatever \
> >>> --header "Accept-Encoding: identity" \
> >>> --header "Connection: Keep-Alive" \
> >>> https://grandtheftworld.com/feed/podcast/
> >>>
> >>> The headers are even in the same order:
> >>>
> >>> GET /feed/podast/ HTTP/1.1
> >>> Host: grand...
> >>> User-Agent: whatever
> >>> Accept: */*
> >>> Accept-Encoding: identity
> >>> Connection: Keep-Alive
> >>>
> >>> But nasty cloudflare returns that nasty captcha/javascript webpage
> >>> with curl, and the proper feed with wget.
> >>>
> >>> I tested with 7.79.1 and 7.81.0
> >> Could be cloudflare checking the user-agent to allow some
> >> non-humans to avoid captchas. (Which is odd, since normally the
> >> problem is non-humans
> >> - but people aren't logical.)
> >>
> >> Try --user-agent matching wget's.
> > I said the headers were EXACTLY the same :p. Both had a user agent
> > of "whatever".
>
>
> This has come up before with other websites and we thought it was
> cloudflare blocking some uncommon SSL handshakes to prevent abuse.
> There's nothing we can do about it unless we support spoofed
> handshakes, which I really don't think is worth it. [1]
>
> [1]: https://github.com/curl/curl/issues/8119
Thanks for the info/link.
FWIW, I was able to fetch the feed using curl in php with only 3 of the
available CURLOPT_SSLVERSION's:
n CURL_SSLVERSION_TLSv1_0
n CURL_SSLVERSION_TLSv1_1
n CURL_SSLVERSION_TLSv1_2
y CURL_SSLVERSION_TLSv1_3
y CURL_SSLVERSION_MAX_TLSv1_0
y CURL_SSLVERSION_MAX_TLSv1_1
n CURL_SSLVERSION_MAX_TLSv1_2
n CURL_SSLVERSION_MAX_TLSv1_3
n CURL_SSLVERSION_SSLv3
And with CURLOPT_USERAGENT = 'whatever'
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-users Etiquette: https://curl.haxx.se/mail/etiquette.htmlReceived on 2022-03-01