Buy commercial curl support. We
help you work out your issues, debug your libcurl applications, use the API,
port to new platforms, add new features and more. With a team lead by the
curl founder Daniel himself.
Re: I Nerd help downloading a page
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: ToddAndMargo via curl-users <curl-users_at_lists.haxx.se>
Date: Sat, 11 Oct 2025 17:40:47 -0700
On 10/11/25 5:39 PM, ToddAndMargo via curl-users wrote:
> On 10/11/25 5:03 PM, ToddAndMargo via curl-users wrote:
>>>> On Sun, 12 Oct 2025, 01:37 ToddAndMargo via curl-users, <curl-
>>>> users_at_lists.haxx.se <mailto:curl-users_at_lists.haxx.se>> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> curl-8.9.1-4.fc41.x86_64
>>>>
>>>> How come Firefox can see this page, but curl can not?
>>>>
>>>>
>>>> First attempt:
>>>>
>>>> $ curl -L https://icedrive.net/apps/desktop-laptop <https://
>>>> icedrive.net/apps/desktop-laptop> -o eraseme.html
>>>> % Total % Received % Xferd Average Speed Time Time
>>>> Time
>>>> Current
>>>> Dload Upload Total Spent
>>>> Left
>>>> Speed
>>>> 100 6995 100 6995 0 0 50638 0 --:--:-- --:--:--
>>>> --:--:--
>>>> 51058
>>>>
>>>>
>>>> Second attempt (that is Brave Browser user agent.
>>>> Firefox's did not work either):
>>>>
>>>> $ curl -L --user-agent "Mozilla/5.0 (X11; Linux x86_64)
>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0
>>>> <http://141.0.0.0> Safari/537.36"
>>>> https://icedrive.net/apps/desktop-laptop/ <https://icedrive.net/
>>>> apps/desktop-laptop/> -o eraseme.html
>>>> % Total % Received % Xferd Average Speed Time Time
>>>> Time
>>>> Current
>>>> Dload Upload Total Spent
>>>> Left
>>>> Speed
>>>> 100 7215 100 7215 0 0 48312 0 --:--:-- --:--:--
>>>> --:--:--
>>>> 48422
>>>>
>>>>
>>>> Yours in confusion,
>>>> -T
>>
>> On 10/11/25 4:47 PM, Bastian Jesuiter via curl-users wrote:
>>> Hi,
>>>
>>> Check the headers.
>>> Some sites might block curls User-Agent.
>>>
>>> Alternatively, the server might block you because curl is missing
>>> cookies or other identifiable headers, for example authorization
>>> headers.
>>>
>>> But we can only guess.
>>>
>>> If you need the download links again, just check the JavaScript, or
>>> monitor network calls from browser when you click the drop down menu.
>>> It will be filled somewhere.
>>> Either at initial page load, or later via xhr or fetch.
>>>
>>> I might take a quick look later today.
>>>
>>> Just experiment a bit though.
>>> Use the browser network tab as help.
>>>
>>> The browser offers copy as curl in the conxt menu for each network call.
>>>
>>> Compare both curl calls, yours and the browsers.
>>>
>>> Good luck,
>>>
>>> Bastian
>>>
>>>
>> I am seeing these two under Firefox's "Network":
>>
>> https://icedrive.net/assets/frontend/lib/jq/jquery-3.3.1.min.js
>> https://icedrive.net/assets/builds/icedrive.frontend.4029.min.js
>
> Does this help?
>
> $ curl -L
> https://icedrive.net/assets/frontend/lib/jq/jquery-3.3.1.min.js -o -
> curl: (2) no URL specified
> curl: try 'curl --help' for more information
> bash: https://icedrive.net/assets/frontend/lib/jq/jquery-3.3.1.min.js:
> No such file or directory
>
>
> $ wget https://icedrive.net/apps/desktop-laptop -o -
> [0] Downloading 'https://icedrive.net/apps/desktop-laptop' ...
> HTTP ERROR response 403 [https://icedrive.net/apps/desktop-laptop]
>
>
The page I do get start with:
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title>
"Just a moment...". Hmmmmmm
Date: Sat, 11 Oct 2025 17:40:47 -0700
On 10/11/25 5:39 PM, ToddAndMargo via curl-users wrote:
> On 10/11/25 5:03 PM, ToddAndMargo via curl-users wrote:
>>>> On Sun, 12 Oct 2025, 01:37 ToddAndMargo via curl-users, <curl-
>>>> users_at_lists.haxx.se <mailto:curl-users_at_lists.haxx.se>> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> curl-8.9.1-4.fc41.x86_64
>>>>
>>>> How come Firefox can see this page, but curl can not?
>>>>
>>>>
>>>> First attempt:
>>>>
>>>> $ curl -L https://icedrive.net/apps/desktop-laptop <https://
>>>> icedrive.net/apps/desktop-laptop> -o eraseme.html
>>>> % Total % Received % Xferd Average Speed Time Time
>>>> Time
>>>> Current
>>>> Dload Upload Total Spent
>>>> Left
>>>> Speed
>>>> 100 6995 100 6995 0 0 50638 0 --:--:-- --:--:--
>>>> --:--:--
>>>> 51058
>>>>
>>>>
>>>> Second attempt (that is Brave Browser user agent.
>>>> Firefox's did not work either):
>>>>
>>>> $ curl -L --user-agent "Mozilla/5.0 (X11; Linux x86_64)
>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0
>>>> <http://141.0.0.0> Safari/537.36"
>>>> https://icedrive.net/apps/desktop-laptop/ <https://icedrive.net/
>>>> apps/desktop-laptop/> -o eraseme.html
>>>> % Total % Received % Xferd Average Speed Time Time
>>>> Time
>>>> Current
>>>> Dload Upload Total Spent
>>>> Left
>>>> Speed
>>>> 100 7215 100 7215 0 0 48312 0 --:--:-- --:--:--
>>>> --:--:--
>>>> 48422
>>>>
>>>>
>>>> Yours in confusion,
>>>> -T
>>
>> On 10/11/25 4:47 PM, Bastian Jesuiter via curl-users wrote:
>>> Hi,
>>>
>>> Check the headers.
>>> Some sites might block curls User-Agent.
>>>
>>> Alternatively, the server might block you because curl is missing
>>> cookies or other identifiable headers, for example authorization
>>> headers.
>>>
>>> But we can only guess.
>>>
>>> If you need the download links again, just check the JavaScript, or
>>> monitor network calls from browser when you click the drop down menu.
>>> It will be filled somewhere.
>>> Either at initial page load, or later via xhr or fetch.
>>>
>>> I might take a quick look later today.
>>>
>>> Just experiment a bit though.
>>> Use the browser network tab as help.
>>>
>>> The browser offers copy as curl in the conxt menu for each network call.
>>>
>>> Compare both curl calls, yours and the browsers.
>>>
>>> Good luck,
>>>
>>> Bastian
>>>
>>>
>> I am seeing these two under Firefox's "Network":
>>
>> https://icedrive.net/assets/frontend/lib/jq/jquery-3.3.1.min.js
>> https://icedrive.net/assets/builds/icedrive.frontend.4029.min.js
>
> Does this help?
>
> $ curl -L
> https://icedrive.net/assets/frontend/lib/jq/jquery-3.3.1.min.js -o -
> curl: (2) no URL specified
> curl: try 'curl --help' for more information
> bash: https://icedrive.net/assets/frontend/lib/jq/jquery-3.3.1.min.js:
> No such file or directory
>
>
> $ wget https://icedrive.net/apps/desktop-laptop -o -
> [0] Downloading 'https://icedrive.net/apps/desktop-laptop' ...
> HTTP ERROR response 403 [https://icedrive.net/apps/desktop-laptop]
>
>
The page I do get start with:
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title>
"Just a moment...". Hmmmmmm
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2025-10-12