Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Strange redirection to Cloudflare server with Captcha?
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Mac-Fly via curl-users <curl-users_at_cool.haxx.se>
Date: Mon, 17 May 2021 21:27:16 +0200
> You need to compare the whole output (as I showed).
> Especially debugging of the TLS version, HTTP version,
> and HTTP headers.
I am afraid I don't know what to look at exactly.
But if you look at the snippets I posted: The IP is different already after the first lines (and these are the first lines of the log in verbose mode). Then it does no change. One IP points to Cloudfare (the windows one), the other does points to audacity (the Linux one). So I _think_ that already at that point something was different for an unknown reason. :-/
Anyway, here is the remaining log for Windows:
* Connected to www.audacityteam.org (172.67.74.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: D:\Users\Martin\Documents\Code\CodeBlocks_Projects\WebChangeMonitor\curl\bin\curl-ca-bundle.crt
* CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2225 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [78 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x2d06268)
} [5 bytes data]
> GET / HTTP/2
> Host: www.audacityteam.org
> accept: */*
> user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 403
< date: Sun, 16 May 2021 12:56:42 GMT
< content-type: text/html; charset=UTF-8
...and Linux:
* Connected to www.audacityteam.org (104.26.0.108) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2225 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [79 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x56713f30)
} [5 bytes data]
> GET / HTTP/2
> Host: www.audacityteam.org
> Accept: */*
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0< HTTP/2 200
< date: Sun, 16 May 2021 12:53:12 GMT
< content-type: text/html; charset=UTF-8
----- Original Message -----
From: curl-users-request_at_cool.haxx.se
To: curl-users_at_cool.haxx.se
Date: Mon, 17 May 2021 12:00:01 +0200
Subject: curl-users Digest, Vol 189, Issue 6
> Send curl-users mailing list submissions to
curl-users_at_cool.haxx.se
To subscribe or unsubscribe via the World Wide Web, visit
https://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
or, via email, send a message with subject or body 'help' to
curl-users-request_at_cool.haxx.se
You can reach the person managing the list at
curl-users-owner_at_cool.haxx.se
When replying, please edit your Subject line so it is more specific
than "Re: Contents of curl-users digest..."
Today's Topics:
1. Re: Strange redirection to Cloudflare server with Captcha?
(Mac-Fly)
2. Re: Strange redirection to Cloudflare server with Captcha?
(Petr Pisar)
----------------------------------------------------------------------
Message: 1
Date: Sun, 16 May 2021 15:16:13 +0200
From: "Mac-Fly" <mac-fly_at_gmx.net>
To: curl-users_at_cool.haxx.se
Subject: Re: Strange redirection to Cloudflare server with Captcha?
Message-ID: <20210516.131613.462.2_at_[0.0.0.0]>
Content-Type: text/plain
Hi Petr,
this was a very good hint!!! Here are my new findings on the very same computer, very same connection.
It *seems* really related to CURL!
1.) Using:
curl 7.76.1 (i386-pc-win32) libcurl/7.76.1 OpenSSL/1.1.1k (Schannel) zlib/1.2.11 brotli/1.0.9 zstd/1.5.0 WinIDN libssh2/1.9.0 nghttp2/1.43.0 libgsasl/1.10.0
Release-Date: 2021-04-14
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli gsasl HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP Unicode UnixSockets zstd
...the issue remains!
2.) Using Linux (in a VM) with:
curl 7.64.0 (i686-pc-linux-gnu) libcurl/7.64.0 OpenSSL/1.1.1d zlib/1.2.11 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) libssh2/1.8.0 nghttp2/1.36.0 librtmp/2.3
Release-Date: 2019-02-06
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL
...the issue is gone!!!
3.) Using Windows with built-in CURL (in system32 folder):
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Release-Date: 2017-11-14, security patched: 2019-11-05
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
...the issue is gone!!!
So it looks like either my version is too new or something has changes that requires a certain flag to be set. Whatever was changed is different between version 7.76.1 and 7.64.0 on Windows an Linux. This could (of course) be the SSL layer. But how to find out now whats happening?
Here is the difference in the logs of CURL 7.76.1 on Windows versus Linux (note that the IP is different!):
------------------- <WINDOWS> -------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 172.67.74.133:443...
* Connected to www.audacityteam.org (172.67.74.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: D:\Users\Martin\Documents\Code\CodeBlocks_Projects\WebChangeMonitor\curl\bin\curl-ca-bundle.crt
* CApath: none
------------------- </WINDOWS> -------------------
------------------- <LINUX> -------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Expire in 0 ms for 1 (transfer 0x56713f30)
* Expire in 1 ms for 1 (transfer 0x56713f30)
(a lot of that!)
* Trying 104.26.0.108...
* TCP_NODELAY set
* Connected to www.audacityteam.org (104.26.0.108) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
------------------- </LINUX> -------------------
What does TCP_NODELAY mean??? The same is set for Windows built-in curl 7.55.1.
------------------------------
Message: 2
Date: Sun, 16 May 2021 11:29:54 +0200
From: Petr Pisar <petr.pisar_at_atlas.cz>
To: curl-users_at_cool.haxx.se
Subject: Re: Strange redirection to Cloudflare server with Captcha?
Message-ID: <YKDmEoP2YS/S42ez_at_album.bayer.uni.cx>
Content-Type: text/plain; charset="utf-8"
V?Sun, May 16, 2021 at 10:57:18AM +0200,?Mac-Fly via curl-users napsal(a):
>
> What is strange though is: I've used a browser (Firefox) with a completely
> empty profile (so no cache, no cookies etc...) and I don't see the captcha.
> So although I am "faking" the user agent to be the same as for Firefox
> ("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101
> Firefox/88.0") there must be something more that Firefox is doing or
> Cloudfare is checking to avoid the captcha.
>
That works for me. But that can be caused by other variables like my IP
address:
$ curl https://www.audacityteam.org/ --header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0' --verbose |head -n 10
* Uses proxy env variable no_proxy == 'localhost,router,router.bayer.uni.cx'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 2606:4700:20::681a:6c:443...
* Connected to www.audacityteam.org (2606:4700:20::681a:6c) port 443 (#0)
* found 147 certificates in /etc/ssl/certs/ca-certificates.crt
* ALPN, offering h2
* ALPN, offering http/1.1
* SSL connection using TLS1.3 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: sni.cloudflaressl.com (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: EC/ECDSA
* certificate version: #3
* subject: C=US,ST=CA,L=San Francisco,O=Cloudflare\, Inc.,CN=sni.cloudflaressl.com
* start date: Tue, 22 Dec 2020 00:00:00 GMT
* expire date: Tue, 21 Dec 2021 23:59:59 GMT
* issuer: C=US,O=Cloudflare\, Inc.,CN=Cloudflare Inc ECC CA-3
* ALPN, server accepted to use h2
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55da9cf91fc0)
> GET / HTTP/2
> Host: www.audacityteam.org
> accept: */*
> user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200
< date: Sun, 16 May 2021 09:26:20 GMT
< content-type: text/html; charset=UTF-8
< vary: Accept-Encoding
< cf-cache-status: DYNAMIC
< cf-request-id: 0a16188cb500004df4e8b28000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=B08U%2B6VwaRx3Y9jtuFtV617avTWMcTbGHi9ObnpG9gR%2FV4%2FLSwUep13ZnJZog%2F5cmf8Mm9Yu2RUIZCLudNRyrARB%2FzygWVvcBAr7B3EKGnAGGVtq49AHHpgv4zQF1QkWkw%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 6503905abdab4df4-FRA
<
{ [875 bytes data]
<!DOCTYPE html>
<html lang="en-US" prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Audacity ? | Free, open source, cross-platform audio software for multi-track recording and editing.</title>
<link rel="profile" href="http://gmpg.org/xfn/11">
<link rel="pingback" href="https://www.audacityteam.org/xmlrpc.php">
-- Petr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://cool.haxx.se/pipermail/curl-users/attachments/20210516/698139d8/attachment-0001.sig>
------------------------------
Message: 2
Date: Sun, 16 May 2021 15:43:23 +0200
From: Petr Pisar <petr.pisar_at_atlas.cz>
To: curl-users_at_cool.haxx.se
Subject: Re: Strange redirection to Cloudflare server with Captcha?
Message-ID: <YKEhe7S1fzRKyUcr_at_album.bayer.uni.cx>
Content-Type: text/plain; charset="utf-8"
V Sun, May 16, 2021 at 03:16:13PM +0200, Mac-Fly via curl-users napsal(a):
> So it looks like either my version is too new or something has changes that
> requires a certain flag to be set. Whatever was changed is different between
> version 7.76.1 and 7.64.0 on Windows an Linux. This could (of course) be the
> SSL layer. But how to find out now whats happening?
>
> Here is the difference in the logs of CURL 7.76.1 on Windows versus Linux (note that the IP is different!):
>
> ------------------- <WINDOWS> -------------------
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
>
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 172.67.74.133:443...
> * Connected to www.audacityteam.org (172.67.74.133) port 443 (#0)
> * ALPN, offering h2
> * ALPN, offering http/1.1
> * successfully set certificate verify locations:
> * CAfile: D:\Users\Martin\Documents\Code\CodeBlocks_Projects\WebChangeMonitor\curl\bin\curl-ca-bundle.crt
> * CApath: none
> ------------------- </WINDOWS> -------------------
>
You need to compare the whole output (as I showed). Especially debugging of
the TLS version, HTTP version, and HTTP headers.
> What does TCP_NODELAY mean??? The same is set for Windows built-in curl 7.55.1.
>
It instructs a TCP layer not to buffer data to be sent into larger chunks.
I.e. it sends every byte as soon as possible.
-- Petr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://cool.haxx.se/pipermail/curl-users/attachments/20210516/1bb2cb8c/attachment-0001.sig>
------------------------------
Subject: Digest Footer
_______________________________________________
curl-users mailing list
curl-users_at_cool.haxx.se
https://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
------------------------------
End of curl-users Digest, Vol 189, Issue 6
******************************************
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2021-05-17
Date: Mon, 17 May 2021 21:27:16 +0200
> You need to compare the whole output (as I showed).
> Especially debugging of the TLS version, HTTP version,
> and HTTP headers.
I am afraid I don't know what to look at exactly.
But if you look at the snippets I posted: The IP is different already after the first lines (and these are the first lines of the log in verbose mode). Then it does no change. One IP points to Cloudfare (the windows one), the other does points to audacity (the Linux one). So I _think_ that already at that point something was different for an unknown reason. :-/
Anyway, here is the remaining log for Windows:
* Connected to www.audacityteam.org (172.67.74.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: D:\Users\Martin\Documents\Code\CodeBlocks_Projects\WebChangeMonitor\curl\bin\curl-ca-bundle.crt
* CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2225 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [78 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x2d06268)
} [5 bytes data]
> GET / HTTP/2
> Host: www.audacityteam.org
> accept: */*
> user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 403
< date: Sun, 16 May 2021 12:56:42 GMT
< content-type: text/html; charset=UTF-8
...and Linux:
* Connected to www.audacityteam.org (104.26.0.108) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2225 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [79 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x56713f30)
} [5 bytes data]
> GET / HTTP/2
> Host: www.audacityteam.org
> Accept: */*
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0< HTTP/2 200
< date: Sun, 16 May 2021 12:53:12 GMT
< content-type: text/html; charset=UTF-8
----- Original Message -----
From: curl-users-request_at_cool.haxx.se
To: curl-users_at_cool.haxx.se
Date: Mon, 17 May 2021 12:00:01 +0200
Subject: curl-users Digest, Vol 189, Issue 6
> Send curl-users mailing list submissions to
curl-users_at_cool.haxx.se
To subscribe or unsubscribe via the World Wide Web, visit
https://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
or, via email, send a message with subject or body 'help' to
curl-users-request_at_cool.haxx.se
You can reach the person managing the list at
curl-users-owner_at_cool.haxx.se
When replying, please edit your Subject line so it is more specific
than "Re: Contents of curl-users digest..."
Today's Topics:
1. Re: Strange redirection to Cloudflare server with Captcha?
(Mac-Fly)
2. Re: Strange redirection to Cloudflare server with Captcha?
(Petr Pisar)
----------------------------------------------------------------------
Message: 1
Date: Sun, 16 May 2021 15:16:13 +0200
From: "Mac-Fly" <mac-fly_at_gmx.net>
To: curl-users_at_cool.haxx.se
Subject: Re: Strange redirection to Cloudflare server with Captcha?
Message-ID: <20210516.131613.462.2_at_[0.0.0.0]>
Content-Type: text/plain
Hi Petr,
this was a very good hint!!! Here are my new findings on the very same computer, very same connection.
It *seems* really related to CURL!
1.) Using:
curl 7.76.1 (i386-pc-win32) libcurl/7.76.1 OpenSSL/1.1.1k (Schannel) zlib/1.2.11 brotli/1.0.9 zstd/1.5.0 WinIDN libssh2/1.9.0 nghttp2/1.43.0 libgsasl/1.10.0
Release-Date: 2021-04-14
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli gsasl HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP Unicode UnixSockets zstd
...the issue remains!
2.) Using Linux (in a VM) with:
curl 7.64.0 (i686-pc-linux-gnu) libcurl/7.64.0 OpenSSL/1.1.1d zlib/1.2.11 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) libssh2/1.8.0 nghttp2/1.36.0 librtmp/2.3
Release-Date: 2019-02-06
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL
...the issue is gone!!!
3.) Using Windows with built-in CURL (in system32 folder):
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Release-Date: 2017-11-14, security patched: 2019-11-05
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
...the issue is gone!!!
So it looks like either my version is too new or something has changes that requires a certain flag to be set. Whatever was changed is different between version 7.76.1 and 7.64.0 on Windows an Linux. This could (of course) be the SSL layer. But how to find out now whats happening?
Here is the difference in the logs of CURL 7.76.1 on Windows versus Linux (note that the IP is different!):
------------------- <WINDOWS> -------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 172.67.74.133:443...
* Connected to www.audacityteam.org (172.67.74.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: D:\Users\Martin\Documents\Code\CodeBlocks_Projects\WebChangeMonitor\curl\bin\curl-ca-bundle.crt
* CApath: none
------------------- </WINDOWS> -------------------
------------------- <LINUX> -------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Expire in 0 ms for 1 (transfer 0x56713f30)
* Expire in 1 ms for 1 (transfer 0x56713f30)
(a lot of that!)
* Trying 104.26.0.108...
* TCP_NODELAY set
* Connected to www.audacityteam.org (104.26.0.108) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
------------------- </LINUX> -------------------
What does TCP_NODELAY mean??? The same is set for Windows built-in curl 7.55.1.
------------------------------
Message: 2
Date: Sun, 16 May 2021 11:29:54 +0200
From: Petr Pisar <petr.pisar_at_atlas.cz>
To: curl-users_at_cool.haxx.se
Subject: Re: Strange redirection to Cloudflare server with Captcha?
Message-ID: <YKDmEoP2YS/S42ez_at_album.bayer.uni.cx>
Content-Type: text/plain; charset="utf-8"
V?Sun, May 16, 2021 at 10:57:18AM +0200,?Mac-Fly via curl-users napsal(a):
>
> What is strange though is: I've used a browser (Firefox) with a completely
> empty profile (so no cache, no cookies etc...) and I don't see the captcha.
> So although I am "faking" the user agent to be the same as for Firefox
> ("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101
> Firefox/88.0") there must be something more that Firefox is doing or
> Cloudfare is checking to avoid the captcha.
>
That works for me. But that can be caused by other variables like my IP
address:
$ curl https://www.audacityteam.org/ --header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0' --verbose |head -n 10
* Uses proxy env variable no_proxy == 'localhost,router,router.bayer.uni.cx'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 2606:4700:20::681a:6c:443...
* Connected to www.audacityteam.org (2606:4700:20::681a:6c) port 443 (#0)
* found 147 certificates in /etc/ssl/certs/ca-certificates.crt
* ALPN, offering h2
* ALPN, offering http/1.1
* SSL connection using TLS1.3 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: sni.cloudflaressl.com (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: EC/ECDSA
* certificate version: #3
* subject: C=US,ST=CA,L=San Francisco,O=Cloudflare\, Inc.,CN=sni.cloudflaressl.com
* start date: Tue, 22 Dec 2020 00:00:00 GMT
* expire date: Tue, 21 Dec 2021 23:59:59 GMT
* issuer: C=US,O=Cloudflare\, Inc.,CN=Cloudflare Inc ECC CA-3
* ALPN, server accepted to use h2
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55da9cf91fc0)
> GET / HTTP/2
> Host: www.audacityteam.org
> accept: */*
> user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200
< date: Sun, 16 May 2021 09:26:20 GMT
< content-type: text/html; charset=UTF-8
< vary: Accept-Encoding
< cf-cache-status: DYNAMIC
< cf-request-id: 0a16188cb500004df4e8b28000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=B08U%2B6VwaRx3Y9jtuFtV617avTWMcTbGHi9ObnpG9gR%2FV4%2FLSwUep13ZnJZog%2F5cmf8Mm9Yu2RUIZCLudNRyrARB%2FzygWVvcBAr7B3EKGnAGGVtq49AHHpgv4zQF1QkWkw%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 6503905abdab4df4-FRA
<
{ [875 bytes data]
<!DOCTYPE html>
<html lang="en-US" prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Audacity ? | Free, open source, cross-platform audio software for multi-track recording and editing.</title>
<link rel="profile" href="http://gmpg.org/xfn/11">
<link rel="pingback" href="https://www.audacityteam.org/xmlrpc.php">
-- Petr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://cool.haxx.se/pipermail/curl-users/attachments/20210516/698139d8/attachment-0001.sig>
------------------------------
Message: 2
Date: Sun, 16 May 2021 15:43:23 +0200
From: Petr Pisar <petr.pisar_at_atlas.cz>
To: curl-users_at_cool.haxx.se
Subject: Re: Strange redirection to Cloudflare server with Captcha?
Message-ID: <YKEhe7S1fzRKyUcr_at_album.bayer.uni.cx>
Content-Type: text/plain; charset="utf-8"
V Sun, May 16, 2021 at 03:16:13PM +0200, Mac-Fly via curl-users napsal(a):
> So it looks like either my version is too new or something has changes that
> requires a certain flag to be set. Whatever was changed is different between
> version 7.76.1 and 7.64.0 on Windows an Linux. This could (of course) be the
> SSL layer. But how to find out now whats happening?
>
> Here is the difference in the logs of CURL 7.76.1 on Windows versus Linux (note that the IP is different!):
>
> ------------------- <WINDOWS> -------------------
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
>
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 172.67.74.133:443...
> * Connected to www.audacityteam.org (172.67.74.133) port 443 (#0)
> * ALPN, offering h2
> * ALPN, offering http/1.1
> * successfully set certificate verify locations:
> * CAfile: D:\Users\Martin\Documents\Code\CodeBlocks_Projects\WebChangeMonitor\curl\bin\curl-ca-bundle.crt
> * CApath: none
> ------------------- </WINDOWS> -------------------
>
You need to compare the whole output (as I showed). Especially debugging of
the TLS version, HTTP version, and HTTP headers.
> What does TCP_NODELAY mean??? The same is set for Windows built-in curl 7.55.1.
>
It instructs a TCP layer not to buffer data to be sent into larger chunks.
I.e. it sends every byte as soon as possible.
-- Petr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://cool.haxx.se/pipermail/curl-users/attachments/20210516/1bb2cb8c/attachment-0001.sig>
------------------------------
Subject: Digest Footer
_______________________________________________
curl-users mailing list
curl-users_at_cool.haxx.se
https://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
------------------------------
End of curl-users Digest, Vol 189, Issue 6
******************************************
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2021-05-17