curl / Mailing Lists / curl-users / Single Mail


Re: How do I download this guy?

From: Jeremy Nicoll <>
Date: Mon, 24 Dec 2018 07:20:07 -0500

On Sun, 23 Dec 2018, at 21:53, ToddAndMargo wrote:
> Hi All,
> How do I download this guy with cURL?
> Firefox's web developer Network says this is what
> is happening:
> curl
> ''
> -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:64.0)
> Gecko/20100101 Firefox/64.0' -H 'Accept:
> text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H
> 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer:
> -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1'
> But how do I get there with just cURL and not Firefox?

In general the method is you look at what happens on the nominal page - in this case - to find out how that page creates the
curl command, and replicate that process yourself.

It might be impossible... if there's a humungous chunk of Javascript that executes and
builds the token parts of the command in a way that you can't fathom out. You might
be very lucky and be able to plug the token stuff from the fetch you've seen work into
another curl command and have that work.

Also, it's possible that just the process of grabbing the first page might give you a
tailored copy of it, from which you could parse out the details that would need to be
fed into a subsequent curl command.

A good starting place is using curl to grab the first page several times, to find out if
anything on it varies from time to time. Then work out what happens when someone
clicks the download button.

Jeremy Nicoll - my opinions are my own.
Received on 2018-12-24