Re: I Nerd help downloading a page

From: ToddAndMargo via curl-users <curl-users_at_lists.haxx.se>
Date: Sat, 11 Oct 2025 17:39:29 -0700

On 10/11/25 5:03 PM, ToddAndMargo via curl-users wrote:
>>> On Sun, 12 Oct 2025, 01:37 ToddAndMargo via curl-users, <curl-
>>> users_at_lists.haxx.se <mailto:curl-users_at_lists.haxx.se>> wrote:
>>>
>>>     Hi All,
>>>
>>>     curl-8.9.1-4.fc41.x86_64
>>>
>>>     How come Firefox can see this page, but curl can not?
>>>
>>>
>>>     First attempt:
>>>
>>>     $ curl -L https://icedrive.net/apps/desktop-laptop <https://
>>>     icedrive.net/apps/desktop-laptop> -o eraseme.html
>>>         % Total    % Received % Xferd  Average Speed   Time    Time
>>>       Time
>>>     Current
>>>                                        Dload  Upload   Total   Spent
>>>     Left
>>>     Speed
>>>     100  6995  100  6995    0     0  50638      0 --:--:-- --:--:--
>>>     --:--:--
>>>     51058
>>>
>>>
>>>     Second attempt (that is Brave Browser user agent.
>>>     Firefox's did not work either):
>>>
>>>     $ curl -L --user-agent "Mozilla/5.0 (X11; Linux x86_64)
>>>     AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0
>>>     <http://141.0.0.0> Safari/537.36"
>>>     https://icedrive.net/apps/desktop-laptop/ <https://icedrive.net/
>>>     apps/desktop-laptop/> -o eraseme.html
>>>         % Total    % Received % Xferd  Average Speed   Time    Time
>>>       Time
>>>     Current
>>>                                        Dload  Upload   Total   Spent
>>>     Left
>>>     Speed
>>>     100  7215  100  7215    0     0  48312      0 --:--:-- --:--:--
>>>     --:--:--
>>>     48422
>>>
>>>
>>>     Yours in confusion,
>>>     -T
>
> On 10/11/25 4:47 PM, Bastian Jesuiter via curl-users wrote:
>> Hi,
>>
>> Check the headers.
>> Some sites might block curls User-Agent.
>>
>> Alternatively, the server might block you because curl is missing
>> cookies or other identifiable headers, for example authorization headers.
>>
>> But we can only guess.
>>
>> If you need the download links again, just check the JavaScript, or
>> monitor network calls from browser when you click the drop down menu.
>> It will be filled somewhere.
>> Either at initial page load, or later via xhr or fetch.
>>
>> I might take a quick look later today.
>>
>> Just experiment a bit though.
>> Use the browser network tab as help.
>>
>> The browser offers copy as curl in the conxt menu for each network call.
>>
>> Compare both curl calls, yours and the browsers.
>>
>> Good luck,
>>
>> Bastian
>>
>>
> I am seeing these two under Firefox's "Network":
>
> https://icedrive.net/assets/frontend/lib/jq/jquery-3.3.1.min.js
> https://icedrive.net/assets/builds/icedrive.frontend.4029.min.js

Does this help?

$ curl -L
https://icedrive.net/assets/frontend/lib/jq/jquery-3.3.1.min.js -o -
curl: (2) no URL specified
curl: try 'curl --help' for more information
bash: https://icedrive.net/assets/frontend/lib/jq/jquery-3.3.1.min.js:
No such file or directory


$ wget https://icedrive.net/apps/desktop-laptop -o -
[0] Downloading 'https://icedrive.net/apps/desktop-laptop' ...
HTTP ERROR response 403 [https://icedrive.net/apps/desktop-laptop]

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2025-10-12