Re: Corrupted download when using the resume option?!

From: Ray Satiro via curl-users <curl-users_at_cool.haxx.se>
Date: Mon, 15 Feb 2021 16:06:27 -0500

On 2/15/2021 3:01 AM, Mihail Kostira via curl-users wrote:
> The server supports resume when I connect from my machine. I can't
> access the corrupted file, as the IoT device has poor connection, I
> can only see the stdout of curl in the log output which is uploaded to
> a server.
>
> Kind regards, Mihail
>
>
> On Fri, Feb 12, 2021 at 12:02 PM <curl-users-request_at_cool.haxx.se
> <mailto:curl-users-request_at_cool.haxx.se>> wrote:
>
> Send curl-users mailing list submissions to
> curl-users_at_cool.haxx.se <mailto:curl-users_at_cool.haxx.se>
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
> <https://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users>
> or, via email, send a message with subject or body 'help' to
> curl-users-request_at_cool.haxx.se
> <mailto:curl-users-request_at_cool.haxx.se>
>
> You can reach the person managing the list at
> curl-users-owner_at_cool.haxx.se <mailto:curl-users-owner_at_cool.haxx.se>
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of curl-users digest..."
>
>
> Today's Topics:
>
>    1. Re: Corrupted download when using the resume option?! (Ray
> Satiro)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 12 Feb 2021 00:14:53 -0500
> From: Ray Satiro <raysatiro_at_yahoo.com <mailto:raysatiro_at_yahoo.com>>
> To: curl-users_at_cool.haxx.se <mailto:curl-users_at_cool.haxx.se>
> Subject: Re: Corrupted download when using the resume option?!
> Message-ID: <24ed09da-a625-ad78-b690-46b3b06c12a7_at_yahoo.com
> <mailto:24ed09da-a625-ad78-b690-46b3b06c12a7_at_yahoo.com>>
> Content-Type: text/plain; charset=utf-8; format=flowed
>
> On 2/10/2021 9:46 AM, Mihail Kostira via curl-users wrote:
> > I'm using curl to download a 10mb file over a poor connection.
> >
> > I use the '-C -' option in a loop to resume the download if it
> breaks
> > (see the snippet below).
> >
> > This works when I test locally and interrupt the download by
> exiting
> > the shell script - after restarting the script, it will resume the
> > download.
> >
> > However, in an IoT device in the field, midway through the
> transfer,
> > curl errors out with "curl: (18) transfer closed with 2674439 bytes
> > remaining to read". After that, the download is retried within the
> > loop and completes - but the file is corrupt. This has happened
> > multiple times (same device).
> >
> > Am I missing something, maybe I'm using an incompatible set of
> > options? Or is it possible that the poor connection results in a
> > corrupt transfer?
> >
> > downloadFileResume() {
> >     # $1  contains the URL to download
> >     local fileName=$(basename "$1")
> >     for (( i=0; i<=$numCurlRetries; i++ ))
> >     do
> >         # capture curl http code (-w %{http_code}) in stdout
> >         curlHttpCode=$(curl -kSLf -u
> "$REPO_USERNAME:$REPO_PASSWORD"
> > -w %{http_code} -O -C - "$1")
> >         local res=$?
> >         [ $res -eq 0 ] && return
> >         # curl gives error when attempting to resume a completely
> > downloaded file, it's a 416 error from the server
> >         [ $curlHttpCode -eq 416 ] && echo "Error 416, assuming
> file is
> > downloaded completely" && return
> >     done
> >     echo "failed to download $fileName"
> >     return 1
> > }
>
>
> Maybe the server does not support file resume? How is it
> corrupted, did
> you check the contents at the end?
>

Please don't top-post it makes the conversation hard to follow [1]. Use
-v for verbose output and append stderr to a file 2>>curlerr. That
should give you some idea of what is happening.


[1]: https://curl.se/mail/etiquette.html#Do_Not_Top_Post



-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2021-02-15