Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Strange redirection to Cloudflare server with Captcha?
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Mac-Fly via curl-users <curl-users_at_cool.haxx.se>
Date: Sun, 16 May 2021 15:16:13 +0200
Hi Petr,
this was a very good hint!!! Here are my new findings on the very same computer, very same connection.
It *seems* really related to CURL!
1.) Using:
curl 7.76.1 (i386-pc-win32) libcurl/7.76.1 OpenSSL/1.1.1k (Schannel) zlib/1.2.11 brotli/1.0.9 zstd/1.5.0 WinIDN libssh2/1.9.0 nghttp2/1.43.0 libgsasl/1.10.0
Release-Date: 2021-04-14
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli gsasl HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP Unicode UnixSockets zstd
...the issue remains!
2.) Using Linux (in a VM) with:
curl 7.64.0 (i686-pc-linux-gnu) libcurl/7.64.0 OpenSSL/1.1.1d zlib/1.2.11 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) libssh2/1.8.0 nghttp2/1.36.0 librtmp/2.3
Release-Date: 2019-02-06
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL
...the issue is gone!!!
3.) Using Windows with built-in CURL (in system32 folder):
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Release-Date: 2017-11-14, security patched: 2019-11-05
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
...the issue is gone!!!
So it looks like either my version is too new or something has changes that requires a certain flag to be set. Whatever was changed is different between version 7.76.1 and 7.64.0 on Windows an Linux. This could (of course) be the SSL layer. But how to find out now whats happening?
Here is the difference in the logs of CURL 7.76.1 on Windows versus Linux (note that the IP is different!):
------------------- <WINDOWS> -------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 172.67.74.133:443...
* Connected to www.audacityteam.org (172.67.74.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: D:\Users\Martin\Documents\Code\CodeBlocks_Projects\WebChangeMonitor\curl\bin\curl-ca-bundle.crt
* CApath: none
------------------- </WINDOWS> -------------------
------------------- <LINUX> -------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Expire in 0 ms for 1 (transfer 0x56713f30)
* Expire in 1 ms for 1 (transfer 0x56713f30)
(a lot of that!)
* Trying 104.26.0.108...
* TCP_NODELAY set
* Connected to www.audacityteam.org (104.26.0.108) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
------------------- </LINUX> -------------------
What does TCP_NODELAY mean??? The same is set for Windows built-in curl 7.55.1.
------------------------------
Message: 2
Date: Sun, 16 May 2021 11:29:54 +0200
From: Petr Pisar <petr.pisar_at_atlas.cz>
To: curl-users_at_cool.haxx.se
Subject: Re: Strange redirection to Cloudflare server with Captcha?
Message-ID: <YKDmEoP2YS/S42ez_at_album.bayer.uni.cx>
Content-Type: text/plain; charset="utf-8"
V Sun, May 16, 2021 at 10:57:18AM +0200, Mac-Fly via curl-users napsal(a):
>
> What is strange though is: I've used a browser (Firefox) with a completely
> empty profile (so no cache, no cookies etc...) and I don't see the captcha.
> So although I am "faking" the user agent to be the same as for Firefox
> ("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101
> Firefox/88.0") there must be something more that Firefox is doing or
> Cloudfare is checking to avoid the captcha.
>
That works for me. But that can be caused by other variables like my IP
address:
$ curl https://www.audacityteam.org/ --header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0' --verbose |head -n 10
* Uses proxy env variable no_proxy == 'localhost,router,router.bayer.uni.cx'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 2606:4700:20::681a:6c:443...
* Connected to www.audacityteam.org (2606:4700:20::681a:6c) port 443 (#0)
* found 147 certificates in /etc/ssl/certs/ca-certificates.crt
* ALPN, offering h2
* ALPN, offering http/1.1
* SSL connection using TLS1.3 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: sni.cloudflaressl.com (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: EC/ECDSA
* certificate version: #3
* subject: C=US,ST=CA,L=San Francisco,O=Cloudflare\, Inc.,CN=sni.cloudflaressl.com
* start date: Tue, 22 Dec 2020 00:00:00 GMT
* expire date: Tue, 21 Dec 2021 23:59:59 GMT
* issuer: C=US,O=Cloudflare\, Inc.,CN=Cloudflare Inc ECC CA-3
* ALPN, server accepted to use h2
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55da9cf91fc0)
> GET / HTTP/2
> Host: www.audacityteam.org
> accept: */*
> user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200
< date: Sun, 16 May 2021 09:26:20 GMT
< content-type: text/html; charset=UTF-8
< vary: Accept-Encoding
< cf-cache-status: DYNAMIC
< cf-request-id: 0a16188cb500004df4e8b28000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=B08U%2B6VwaRx3Y9jtuFtV617avTWMcTbGHi9ObnpG9gR%2FV4%2FLSwUep13ZnJZog%2F5cmf8Mm9Yu2RUIZCLudNRyrARB%2FzygWVvcBAr7B3EKGnAGGVtq49AHHpgv4zQF1QkWkw%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 6503905abdab4df4-FRA
<
{ [875 bytes data]
<!DOCTYPE html>
<html lang="en-US" prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Audacity ® | Free, open source, cross-platform audio software for multi-track recording and editing.</title>
<link rel="profile" href="http://gmpg.org/xfn/11">
<link rel="pingback" href="https://www.audacityteam.org/xmlrpc.php">
-- Petr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://cool.haxx.se/pipermail/curl-users/attachments/20210516/698139d8/attachment-0001.sig>
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2021-05-16
Date: Sun, 16 May 2021 15:16:13 +0200
Hi Petr,
this was a very good hint!!! Here are my new findings on the very same computer, very same connection.
It *seems* really related to CURL!
1.) Using:
curl 7.76.1 (i386-pc-win32) libcurl/7.76.1 OpenSSL/1.1.1k (Schannel) zlib/1.2.11 brotli/1.0.9 zstd/1.5.0 WinIDN libssh2/1.9.0 nghttp2/1.43.0 libgsasl/1.10.0
Release-Date: 2021-04-14
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli gsasl HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP Unicode UnixSockets zstd
...the issue remains!
2.) Using Linux (in a VM) with:
curl 7.64.0 (i686-pc-linux-gnu) libcurl/7.64.0 OpenSSL/1.1.1d zlib/1.2.11 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) libssh2/1.8.0 nghttp2/1.36.0 librtmp/2.3
Release-Date: 2019-02-06
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL
...the issue is gone!!!
3.) Using Windows with built-in CURL (in system32 folder):
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Release-Date: 2017-11-14, security patched: 2019-11-05
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
...the issue is gone!!!
So it looks like either my version is too new or something has changes that requires a certain flag to be set. Whatever was changed is different between version 7.76.1 and 7.64.0 on Windows an Linux. This could (of course) be the SSL layer. But how to find out now whats happening?
Here is the difference in the logs of CURL 7.76.1 on Windows versus Linux (note that the IP is different!):
------------------- <WINDOWS> -------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 172.67.74.133:443...
* Connected to www.audacityteam.org (172.67.74.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: D:\Users\Martin\Documents\Code\CodeBlocks_Projects\WebChangeMonitor\curl\bin\curl-ca-bundle.crt
* CApath: none
------------------- </WINDOWS> -------------------
------------------- <LINUX> -------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Expire in 0 ms for 1 (transfer 0x56713f30)
* Expire in 1 ms for 1 (transfer 0x56713f30)
(a lot of that!)
* Trying 104.26.0.108...
* TCP_NODELAY set
* Connected to www.audacityteam.org (104.26.0.108) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
------------------- </LINUX> -------------------
What does TCP_NODELAY mean??? The same is set for Windows built-in curl 7.55.1.
------------------------------
Message: 2
Date: Sun, 16 May 2021 11:29:54 +0200
From: Petr Pisar <petr.pisar_at_atlas.cz>
To: curl-users_at_cool.haxx.se
Subject: Re: Strange redirection to Cloudflare server with Captcha?
Message-ID: <YKDmEoP2YS/S42ez_at_album.bayer.uni.cx>
Content-Type: text/plain; charset="utf-8"
V Sun, May 16, 2021 at 10:57:18AM +0200, Mac-Fly via curl-users napsal(a):
>
> What is strange though is: I've used a browser (Firefox) with a completely
> empty profile (so no cache, no cookies etc...) and I don't see the captcha.
> So although I am "faking" the user agent to be the same as for Firefox
> ("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101
> Firefox/88.0") there must be something more that Firefox is doing or
> Cloudfare is checking to avoid the captcha.
>
That works for me. But that can be caused by other variables like my IP
address:
$ curl https://www.audacityteam.org/ --header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0' --verbose |head -n 10
* Uses proxy env variable no_proxy == 'localhost,router,router.bayer.uni.cx'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 2606:4700:20::681a:6c:443...
* Connected to www.audacityteam.org (2606:4700:20::681a:6c) port 443 (#0)
* found 147 certificates in /etc/ssl/certs/ca-certificates.crt
* ALPN, offering h2
* ALPN, offering http/1.1
* SSL connection using TLS1.3 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: sni.cloudflaressl.com (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: EC/ECDSA
* certificate version: #3
* subject: C=US,ST=CA,L=San Francisco,O=Cloudflare\, Inc.,CN=sni.cloudflaressl.com
* start date: Tue, 22 Dec 2020 00:00:00 GMT
* expire date: Tue, 21 Dec 2021 23:59:59 GMT
* issuer: C=US,O=Cloudflare\, Inc.,CN=Cloudflare Inc ECC CA-3
* ALPN, server accepted to use h2
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55da9cf91fc0)
> GET / HTTP/2
> Host: www.audacityteam.org
> accept: */*
> user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200
< date: Sun, 16 May 2021 09:26:20 GMT
< content-type: text/html; charset=UTF-8
< vary: Accept-Encoding
< cf-cache-status: DYNAMIC
< cf-request-id: 0a16188cb500004df4e8b28000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=B08U%2B6VwaRx3Y9jtuFtV617avTWMcTbGHi9ObnpG9gR%2FV4%2FLSwUep13ZnJZog%2F5cmf8Mm9Yu2RUIZCLudNRyrARB%2FzygWVvcBAr7B3EKGnAGGVtq49AHHpgv4zQF1QkWkw%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 6503905abdab4df4-FRA
<
{ [875 bytes data]
<!DOCTYPE html>
<html lang="en-US" prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Audacity ® | Free, open source, cross-platform audio software for multi-track recording and editing.</title>
<link rel="profile" href="http://gmpg.org/xfn/11">
<link rel="pingback" href="https://www.audacityteam.org/xmlrpc.php">
-- Petr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://cool.haxx.se/pipermail/curl-users/attachments/20210516/698139d8/attachment-0001.sig>
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2021-05-16