cURL / Mailing Lists / curl-users / Single Mail

curl-users

Re: Using list {} does not work with -z to download only newer files

From: Gustav Axelsson <gustav.axelsson_at_gmail.com>
Date: Fri, 21 Apr 2006 13:14:22 +0200

21 apr 2006 kl. 12.32 skrev Daniel Stenberg:

> On Fri, 21 Apr 2006, Gustav Axelsson wrote:
>
>> if I use a list with only one item (the date 2006-04-20) the file
>> is always downloaded despite the fact that the server does not
>> have a more recent file.
>
>> -z ~/Library/Xmltv/svt1.svt.se_#1.xml
>
> Uh, indeed. We don't support the #-parameters for the -z option so
> curl will get the existing date from the file using that _literal_
> name and mostly likely there is no such file. If the file doesn't
> exist, it switches off the conditional fetch again...
>
> This is still the same in the latest version! :-(
>
> It is certainly something we should fix. Anyone feels like grabbing
> the issue?

I thought it was a strange behaviour... I kind of found a workaround
but it also has problems:

Instead of using a list { } or two separate instances of curl, I
wrote the filename complete with -o and -z twice to download two
files with one instance of curl:
curl -R --user-agent tv.swedb.se.widget --compressed 'http://
tvguide.swedb.se/xmltv/svt1.svt.se_2006-04-20.xml.gz' -o ~/Library/
Xmltv/svt1.svt.se_2006-04-20.xml -z ~/Library/Xmltv/
svt1.svt.se_2006-04-20.xml 'http://tvguide.swedb.se/xmltv/
svt1.svt.se_2006-04-21.xml.gz' -o ~/Library/Xmltv/
svt1.svt.se_2006-04-21.xml -z ~/Library/Xmltv/svt1.svt.se_2006-04-21.xml

Doing it this way does respect the -z flag for both files since the
filenames are hard-coded.
But I need to do it for about 150 files... so I tried that too and it
seems that at least my version of curl does strange things when it
gets so many arguments.

I tried the 16 files I wanted for svt1.svt.se_YYYY-MM-DD.xml
(normally it's yesterday, today + 14 days) and while most of the
files respect -z and does not get downloaded if the dates on the
harddrive and server match, some do. It's not random either, if I run
the command twice, the same files get downloaded over and over again,
despite -z.

Below is the 16 files I want to check to see if they have been
updated. Currently only file 0,1,7,9 and 13 - 16 respect their -z
flags, the rest always gets downloaded. If I check one of them (for
example file 2) with a single curl call, then nothing is transferred
because of -z.

Yesterday when I tested 150 files with one call, most acctually
respected the -z flag but some didn't. But today it's not the same
files that get -z flagged as it was yesterday although repeated
invocations of curl today skips the same files, 0,1,7,9,13-16 so the
behaviour is static. Yesterday it also were certain files that never
respected -z despite running the command over and over again.

I do admit that I'm not going to do it this way but there might be
others that need to download several files with different names that
can't be calculated from a list and in theory this way is a "proper"
way to do it that *should* work, right?

The 16 files:
curl -R --user-agent tv.swedb.se.widget --compressed http://
tvguide.swedb.se/xmltv/svt1.svt.se_2006-04-19.xml.gz -o /Users/
gusax840/Library/Xmltv/svt1.svt.se_2006-04-19.xml -z /Users/gusax840/
Library/Xmltv/svt1.svt.se_2006-04-19.xml http://tvguide.swedb.se/
xmltv/svt1.svt.se_2006-04-20.xml.gz -o /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-20.xml -z /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-20.xml http://tvguide.swedb.se/xmltv/
svt1.svt.se_2006-04-21.xml.gz -o /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-21.xml -z /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-21.xml http://tvguide.swedb.se/xmltv/
svt1.svt.se_2006-04-22.xml.gz -o /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-22.xml -z /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-22.xml http://tvguide.swedb.se/xmltv/
svt1.svt.se_2006-04-23.xml.gz -o /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-23.xml -z /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-23.xml http://tvguide.swedb.se/xmltv/
svt1.svt.se_2006-04-24.xml.gz -o /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-24.xml -z /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-24.xml http://tvguide.swedb.se/xmltv/
svt1.svt.se_2006-04-25.xml.gz -o /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-25.xml -z /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-25.xml http://tvguide.swedb.se/xmltv/
svt1.svt.se_2006-04-26.xml.gz -o /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-26.xml -z /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-26.xml http://tvguide.swedb.se/xmltv/
svt1.svt.se_2006-04-27.xml.gz -o /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-27.xml -z /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-27.xml http://tvguide.swedb.se/xmltv/
svt1.svt.se_2006-04-28.xml.gz -o /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-28.xml -z /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-28.xml http://tvguide.swedb.se/xmltv/
svt1.svt.se_2006-04-29.xml.gz -o /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-29.xml -z /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-29.xml http://tvguide.swedb.se/xmltv/
svt1.svt.se_2006-04-30.xml.gz -o /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-30.xml -z /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-04-30.xml http://tvguide.swedb.se/xmltv/
svt1.svt.se_2006-05-01.xml.gz -o /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-05-01.xml -z /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-05-01.xml http://tvguide.swedb.se/xmltv/
svt1.svt.se_2006-05-02.xml.gz -o /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-05-02.xml -z /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-05-02.xml http://tvguide.swedb.se/xmltv/
svt1.svt.se_2006-05-03.xml.gz -o /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-05-03.xml -z /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-05-03.xml http://tvguide.swedb.se/xmltv/
svt1.svt.se_2006-05-04.xml.gz -o /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-05-04.xml -z /Users/gusax840/Library/Xmltv/
svt1.svt.se_2006-05-04.xml
Received on 2006-04-21