cURL problems (by design?) (fwd)

From: Daniel Stenberg via curl-users <curl-users_at_lists.haxx.se>
Date: Wed, 2 Sep 2026 09:39:22 +0200 (CEST)

Hello,

Some comments on curl functionality from a private email I received. Forwarded
with permission but anonymized.

---------- Forwarded message ----------
Date: Tue, 1 Sep 2026 09:51:40
To: daniel_at_haxx.se
Subject: cURL problems (by design?)

Dear Mister Stenberg,
Hi Daniel,

I'm surprized that cURL offers lots of flexibility at the source side, a.o. proposing globbing, which keeps being extended in new cURL releases --- and so much the better.
However, no similar flexible solutions are available at the target side.
Globbing is immensely powerful, and can be considered as a form of wildcard(s), because it allows to process a great many input data (images, i.e. URLs) without the need to script a way to loop through all possibilities.
And obviously: globbing will always be more efficient.
However, a backside is that one is obliged to stick to elements already present at the source side, which doesn't leave much room to obtain handy result files.

I ran into these limitations again recently, when trying to download the tiles (images) making up a big, high res image.
And once again I got frustrated by the difficulties to work around them, so this time I thought I'd share them.


This is the URL I want to process (which represents 195 images):

"https://dam.museabrugge.be/iiif/2/public%2F23525.tif/ [0-7168:512] , [0-6144:512] ,512,512/512,/0/default.jpg"

Yes indeed, all files have the same name "default.jpg" (but different paths, of course), which is the root cause of most of my cURL problems, but it's sth. we have to deal with, as this is how the images are shared online.
These are possible ways to process these images using globbing:

1.
cURL -s "<path>" -o "#1-#2.jpg"

This works; however, it is NOT possible to obtain the output format I would like to have.

Problem:
there is no way to define a Python-ish format for vars #1 and #2 (e.g. #1 %03d ), defining a length and adding zero-padding, which means that I run into trouble next, when all tiles must be stitched together to reconstruct the complete image.
I use ImageMagick to do so, and IM uses an alphabetical sort order (IDNK if that's caused by the shell, i.c. "cmd.exe", or if that's how IM is coded) which will put "0- 5 12.jpg" behind "0- 1 024", because it doesn't/can't take into consideration the length of the coordinates. (3-digit? 4-digit?)
Or more generally: because as in many (? most) user applications it is not possibe in IM to influence the processing order of the input files.


2.
One could also decide to avoid problems induced by naming the output files one-self, and stick to the source names instead. (= option " -O ").
In this example that choice wouldn't be very productive, because the command would result in only a single file, as opposed to 195 files.
Simply because cURL will always silently overwrite files with the same name. (contrary to default File Explorer behavior, which is to add a suffix " (n)" at the end of the file name, in front of the file extension. (cf. infra)
cURL's "skip" option doesn't help either, as it will only end up with the first file.

cURL does provide a solution, though, but IMHO it has not been very well thought through:

cURL -s --no-clobber -O "<path>"

Indeed, this option doesn't help for two main reasons:

- the option is limited to 100 files.
Why? The manual doesn't provide any clear reason, so it seems an arbitrary choice.
[ https://curl.se/docs/manpage.html#--no-clobber | https://curl.se/docs/manpage.html#--no-clobber ]
In that case I'd very much prefer to set a value myself, as a user. Or to let cURL use a default value (= current behavior), but which I can change if need be.
Even if the limitation has some clear technical rationale, the choice should be with the user, or he should at least be left the opportunity to change it. Quod non!
WIth the explanation in the Manual he'll be able to decide if he may run into certain problems (like not enough memory) and act accordingly.

IMHO it's the responsibility of the user to make sure that he's downloading a set of files which is realistic.
If there is a risk that he might make cURL crash by (unknowingly) processing a problematic source, cURL could still intervene by aborting after a certain number of files (with a clear, unambiguous warning or error).

- the 100 output files are (in this particular case): " default.jpg " to " default.jpg .99 ".
Which is nice, as it avoids the above mentioned overriding, but... why doesn't cURL do so in a more practical way?
I know have 99 files which I cannot easily open, because they don't have a standard extension, as Windows' File Explorer requires.
Both in order to open them with an image viewer ( by double-clicking them in File Explorer) , or to simply view them in File Explorer (when in mode "viewing as icons").
Now they appear as white rectangles, which has zero added value.

I don't understand why cURL can't simply create " default.jpg " to " default 99 .jpg " instead? (and as we are to it: with or without zero padding, at the discretion of the user)
It might even be preferable to allow users to define a dynamic output suffix for this specific use case, because that way the output order can be defined/fixed for later use. (including the possibility for zero-padding)

Best wishes,
[redacted]

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2026-09-02