cURL / Mailing Lists / curl-users / Single Mail

curl-users

RE: Problem Downloading files from webpage using cURL -- still struggling

From: Roth, Kevin P. <kproth_at_mapllc.com>
Date: Wed, 26 May 2004 08:43:03 -0400

You'll find that the following sequence of steps works OK:

1. This one requests the "main" page, which apparently (based on a
little trial and error) sets a cookie that you need to get the actual
file. I'm saving the session cookie in a file named tmp.cky, using the
-D flag. Notice that I've used the actual domain name instead of the IP
address, which eliminates the need to turn off certificate checking
(which is what -k does). Finally, I've sent the output of that page into
a 'special' file named NUL (I'm assuming you're using windows?), which
is the digital equivalent of a circular file (a.k.a. the bit-bucket).

C:\> curl -D tmp.cky -o NUL
https://guarantytechnology.efanniemae.com/PDD/

2. This one uses the cookie from the previous command, and requests the
file you needed. Note the URL - if you hover over the mbs052004.zip link
on that web page (using Netscape or IE), you'll see where we've gotten
that URL; you've made a faulty assumption that the filename simply gets
tacked onto the end of the main URL... I'm also using the -o flag to
redirect the output into a zip file (as opposed to dumping it back to
the screen which is the default behavior).

C:\> curl -b tmp.cky -o mbs052004.zip
https://guarantytechnology.efanniemae.com/PDD/download.do?filename=mbs05
2004.zip

Daniel - while testing this case out, I observed that the session cookie
was NOT being re-read from a cookie-jar style of file. E.g. if I used
curl -c tmp.cky .../PDD/, followed by curl -b tmp.cky
...?filename=mbs..., it was ignoring the cookie in that file. Even
though the domain and path matched up just fine.

Is there a bug here? Possibly related to the -j/--junk-session-cookies
option? I did NOT specify that option, therefore I expected the session
cookie to be used.

Thanks,
- Kevin
Received on 2004-05-26