Buy commercial curl support. We
help you work out your issues, debug your libcurl applications, use the API,
port to new platforms, add new features and more. With a team lead by the
curl founder Daniel himself.
Re: You've been blocked
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: ToddAndMargo via curl-users <curl-users_at_lists.haxx.se>
Date: Tue, 21 Oct 2025 03:04:50 -0700
On 10/20/25 5:50 AM, Hans Henrik Bergan wrote:
> On Sat, 18 Oct 2025 at 12:03, ToddAndMargo via curl-users
> <curl-users_at_lists.haxx.se> wrote:
>>
>> Hi All,
>>
>> curl-8.9.1-4.fc41.x86_64
>>
>> How do I get around "You've been blocked" on
>> this web sire:
>>
>> $ curl -L
>> https://www.softpedia.com/get/System/Back-Up-and-Recovery/Icedrive.shtml#download
>> -o eraseme.html
>>
>> adding
>> --user-agent 'Mozilla/5.0 (X11; Linux x86_64; rv:143.0)
>> Gecko/20100101 Firefox/143.0'
>>
>> Did not help.
>>
>> Yours in confusion,
>> -T
>
> softpedia.com use the cloudflare WAF thing.
> Cloudflare WAF detect curl by minor details in its TLS handshake, and
> block curl,
> even if the user-agent is whitelisted, WAF detect that it's actually
> not firefox because
> the TLS details doesn't match.
> There is a project called curl-impersonate,
> https://github.com/lwthiker/curl-impersonate ,
> where curl has been modified to carefully mimic the Chromium and
> Firefox TLS implementation, for example:
> ./curl-impersonate-ff --user-agent 'Mozilla/5.0 (Windows NT 10.0;
> Win64; x64; rv:140.0) Gecko/20100101 Firefox/140.0'
> 'https://www.softpedia.com/get/System/Back-Up-and-Recovery/Icedrive.shtml'
>
> works. The useragent claims to be firefox, and it's been modified to
> mimic Firefox TLS.
Hi Hans,
Well, my work around randomly fail. Probably dependent on what
cloutflare server I am logging into at the server farm.
But curl-impersonate is getting around it.
Thank you!
-T
My notes:
How to install and use curl-impersonate with Fedora:
Fedora:
# rpm install nss nss-pem ca-certificates
Download a precompiled binary from:
https://github.com/lwthiker/curl-impersonate/releases
https://github.com/lwthiker/curl-impersonate/releases/download/v0.6.1/curl-impersonate-v0.6.1.x86_64-linux-gnu.tar.gz
create a directory and de-tar the above into it.
The following site, about half way downm will give you what
each script impersonates.
The following examples uses
chrome116 (Chrome build 116.0.5845.180 and Windows 10)
Download a web page:
$ curl_chrome116 https://icedrive.net/apps/desktop-laptop > eraseme.html
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
100 4986 0 4986 0 0 17467 0 --:--:-- --:--:-- --:--:--
17494
Download a binary file:
$ curl_chrome116
https://cdn.icedrive.net/static/apps/win/IcedriveSetup-v3.56.exe >
eraseme.exe
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
100 256M 100 256M 0 0 525k 0 0:08:19 0:08:19 --:--:--
458k
Date: Tue, 21 Oct 2025 03:04:50 -0700
On 10/20/25 5:50 AM, Hans Henrik Bergan wrote:
> On Sat, 18 Oct 2025 at 12:03, ToddAndMargo via curl-users
> <curl-users_at_lists.haxx.se> wrote:
>>
>> Hi All,
>>
>> curl-8.9.1-4.fc41.x86_64
>>
>> How do I get around "You've been blocked" on
>> this web sire:
>>
>> $ curl -L
>> https://www.softpedia.com/get/System/Back-Up-and-Recovery/Icedrive.shtml#download
>> -o eraseme.html
>>
>> adding
>> --user-agent 'Mozilla/5.0 (X11; Linux x86_64; rv:143.0)
>> Gecko/20100101 Firefox/143.0'
>>
>> Did not help.
>>
>> Yours in confusion,
>> -T
>
> softpedia.com use the cloudflare WAF thing.
> Cloudflare WAF detect curl by minor details in its TLS handshake, and
> block curl,
> even if the user-agent is whitelisted, WAF detect that it's actually
> not firefox because
> the TLS details doesn't match.
> There is a project called curl-impersonate,
> https://github.com/lwthiker/curl-impersonate ,
> where curl has been modified to carefully mimic the Chromium and
> Firefox TLS implementation, for example:
> ./curl-impersonate-ff --user-agent 'Mozilla/5.0 (Windows NT 10.0;
> Win64; x64; rv:140.0) Gecko/20100101 Firefox/140.0'
> 'https://www.softpedia.com/get/System/Back-Up-and-Recovery/Icedrive.shtml'
>
> works. The useragent claims to be firefox, and it's been modified to
> mimic Firefox TLS.
Hi Hans,
Well, my work around randomly fail. Probably dependent on what
cloutflare server I am logging into at the server farm.
But curl-impersonate is getting around it.
Thank you!
-T
My notes:
How to install and use curl-impersonate with Fedora:
Fedora:
# rpm install nss nss-pem ca-certificates
Download a precompiled binary from:
https://github.com/lwthiker/curl-impersonate/releases
https://github.com/lwthiker/curl-impersonate/releases/download/v0.6.1/curl-impersonate-v0.6.1.x86_64-linux-gnu.tar.gz
create a directory and de-tar the above into it.
The following site, about half way downm will give you what
each script impersonates.
The following examples uses
chrome116 (Chrome build 116.0.5845.180 and Windows 10)
Download a web page:
$ curl_chrome116 https://icedrive.net/apps/desktop-laptop > eraseme.html
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
100 4986 0 4986 0 0 17467 0 --:--:-- --:--:-- --:--:--
17494
Download a binary file:
$ curl_chrome116
https://cdn.icedrive.net/static/apps/win/IcedriveSetup-v3.56.exe >
eraseme.exe
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
100 256M 100 256M 0 0 525k 0 0:08:19 0:08:19 --:--:--
458k
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2025-10-21