Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: cloudflare
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Dennis Nezic via curl-users <curl-users_at_lists.haxx.se>
Date: Mon, 28 Feb 2022 22:03:49 -0500
On Mon, 28 Feb 2022 21:57:03 -0500, Dennis Nezic via curl-users wrote:
> On Mon, 28 Feb 2022 17:29:36 -0500, Ray Satiro via curl-users wrote:
> > On 2/28/2022 4:47 PM, Dennis Nezic via curl-users wrote:
> > > On Mon, 28 Feb 2022 15:31:19 -0500, Timothe Litt via curl-users
> > > wrote:
> > >> On 28-Feb-22 13:29, Dennis Nezic via curl-users wrote:
> > >>> Any idea why I'm not able to fetch
> > >>> https://grandtheftworld.com/feed/podcast/
> > >>> using curl, but I am with wget?
> > >>>
> > >>> *Even when the http headers are exactly the same!?*
> > >>>
> > >>> For example, to match wget's headers, I ran the command:
> > >>>
> > >>> curl \
> > >>> -A whatever \
> > >>> --header "Accept-Encoding: identity" \
> > >>> --header "Connection: Keep-Alive" \
> > >>> https://grandtheftworld.com/feed/podcast/
> > >>>
> > >>> The headers are even in the same order:
> > >>>
> > >>> GET /feed/podast/ HTTP/1.1
> > >>> Host: grand...
> > >>> User-Agent: whatever
> > >>> Accept: */*
> > >>> Accept-Encoding: identity
> > >>> Connection: Keep-Alive
> > >>>
> > >>> But nasty cloudflare returns that nasty captcha/javascript
> > >>> webpage with curl, and the proper feed with wget.
> > >>>
> > >>> I tested with 7.79.1 and 7.81.0
> > >> Could be cloudflare checking the user-agent to allow some
> > >> non-humans to avoid captchas. (Which is odd, since normally the
> > >> problem is non-humans
> > >> - but people aren't logical.)
> > >>
> > >> Try --user-agent matching wget's.
> > > I said the headers were EXACTLY the same :p. Both had a user agent
> > > of "whatever".
> >
> >
> > This has come up before with other websites and we thought it was
> > cloudflare blocking some uncommon SSL handshakes to prevent abuse.
> > There's nothing we can do about it unless we support spoofed
> > handshakes, which I really don't think is worth it. [1]
> >
> > [1]: https://github.com/curl/curl/issues/8119
>
> Thanks for the info/link.
>
> FWIW, I was able to fetch the feed using curl in php with only 3 of
> the available CURLOPT_SSLVERSION's:
>
> n CURL_SSLVERSION_TLSv1_0
> n CURL_SSLVERSION_TLSv1_1
> n CURL_SSLVERSION_TLSv1_2
> y CURL_SSLVERSION_TLSv1_3
> y CURL_SSLVERSION_MAX_TLSv1_0
> y CURL_SSLVERSION_MAX_TLSv1_1
> n CURL_SSLVERSION_MAX_TLSv1_2
> n CURL_SSLVERSION_MAX_TLSv1_3
> n CURL_SSLVERSION_SSLv3
>
> And with CURLOPT_USERAGENT = 'whatever'
Similarly, the cli equivalents should work. For now :S... (Cloudflare
is awful)
curl -A whatever --tlsv1.3
Date: Mon, 28 Feb 2022 22:03:49 -0500
On Mon, 28 Feb 2022 21:57:03 -0500, Dennis Nezic via curl-users wrote:
> On Mon, 28 Feb 2022 17:29:36 -0500, Ray Satiro via curl-users wrote:
> > On 2/28/2022 4:47 PM, Dennis Nezic via curl-users wrote:
> > > On Mon, 28 Feb 2022 15:31:19 -0500, Timothe Litt via curl-users
> > > wrote:
> > >> On 28-Feb-22 13:29, Dennis Nezic via curl-users wrote:
> > >>> Any idea why I'm not able to fetch
> > >>> https://grandtheftworld.com/feed/podcast/
> > >>> using curl, but I am with wget?
> > >>>
> > >>> *Even when the http headers are exactly the same!?*
> > >>>
> > >>> For example, to match wget's headers, I ran the command:
> > >>>
> > >>> curl \
> > >>> -A whatever \
> > >>> --header "Accept-Encoding: identity" \
> > >>> --header "Connection: Keep-Alive" \
> > >>> https://grandtheftworld.com/feed/podcast/
> > >>>
> > >>> The headers are even in the same order:
> > >>>
> > >>> GET /feed/podast/ HTTP/1.1
> > >>> Host: grand...
> > >>> User-Agent: whatever
> > >>> Accept: */*
> > >>> Accept-Encoding: identity
> > >>> Connection: Keep-Alive
> > >>>
> > >>> But nasty cloudflare returns that nasty captcha/javascript
> > >>> webpage with curl, and the proper feed with wget.
> > >>>
> > >>> I tested with 7.79.1 and 7.81.0
> > >> Could be cloudflare checking the user-agent to allow some
> > >> non-humans to avoid captchas. (Which is odd, since normally the
> > >> problem is non-humans
> > >> - but people aren't logical.)
> > >>
> > >> Try --user-agent matching wget's.
> > > I said the headers were EXACTLY the same :p. Both had a user agent
> > > of "whatever".
> >
> >
> > This has come up before with other websites and we thought it was
> > cloudflare blocking some uncommon SSL handshakes to prevent abuse.
> > There's nothing we can do about it unless we support spoofed
> > handshakes, which I really don't think is worth it. [1]
> >
> > [1]: https://github.com/curl/curl/issues/8119
>
> Thanks for the info/link.
>
> FWIW, I was able to fetch the feed using curl in php with only 3 of
> the available CURLOPT_SSLVERSION's:
>
> n CURL_SSLVERSION_TLSv1_0
> n CURL_SSLVERSION_TLSv1_1
> n CURL_SSLVERSION_TLSv1_2
> y CURL_SSLVERSION_TLSv1_3
> y CURL_SSLVERSION_MAX_TLSv1_0
> y CURL_SSLVERSION_MAX_TLSv1_1
> n CURL_SSLVERSION_MAX_TLSv1_2
> n CURL_SSLVERSION_MAX_TLSv1_3
> n CURL_SSLVERSION_SSLv3
>
> And with CURLOPT_USERAGENT = 'whatever'
Similarly, the cli equivalents should work. For now :S... (Cloudflare
is awful)
curl -A whatever --tlsv1.3
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-users Etiquette: https://curl.haxx.se/mail/etiquette.htmlReceived on 2022-03-01