Re: Continue Download C++ Implementation

From: Tomas Berger via curl-library <curl-library_at_cool.haxx.se>
Date: Wed, 9 Dec 2020 10:00:54 +0100

> First - flush & close your file. It's almost certainly buffered and not all data may be actually written to disk when you invoke copy.
I did actually close my failed after easy_perform returned, it just missed it when I copied to code. I did however try to flush the file.
I tried after each write, and I tried before flush (not both at the same time though)

> Are you sure this problem relates to the "continue" semantics?
The main reason I believe it has to do with "continue" is that if I do it in one connection, It downloads perfectly. Both in the write callback and before the close after the file was finished/aborted

The way I abort the download is to flip a Boolean that makes my progress callback return 1
int progress(progressMeta* meta, curl_off_t totalDown, curl_off_t currentDown, curl_off_t totalUp, curl_off_t currentUp) {
const std::lock_guard<std::mutex> lk{meta->mtx};
if (meta->abort) {
return 1;
}
//does other stuff related to keeping track of progress
return 0;
}

> Compare the bytes of the file you wrote to the bytes of your reference file (ZIP).
> Where do they diverge? Do they diverge near your "join"? Elsewhere?

> If that doesn't help you could start debugging by dumping hex representation of the reference file and the one you ended up with. Find the first difference and go from there.
This is where it got interesting:
Contex for the split in hex, as i will compare hex dump;
write to [51062d] = ca
write to [51062e] = ec
write to [51062f] = c2
/Cut
/Continue
write to [51062e] = ec
write to [51062f] = c2
write to [510630] = dd

The first 3 line from the failed transfer's hex dump looks like this:
> 0000000 0000 0000 0000 0000 0000 0000 0000 0000
> *
> 0510620 0000 0000 0000 0000 0000 0000 0000 c2ec
The "*" is not an edit from my side, to compare, the first 3 lines from the successful transfer
> 0000000 4b50 0403 0014 0000 0008 6915 5183 e1ff
> 0000010 a5c6 a583 0152 74a2 0154 0007 001c 786d
> 0000020 6d49 6761 5565 0954 0300 d54a 5fc8 d54a

And the "0510620", line in question in the successful transfer looks like this:
> 0510620 a3d2 7c47 95e0 7595 7e7a 905d ca7a c2ec
And from there on out, they are the same.

This indicates to be that the continue from the offset is somewhat wrong, but what I am doing wrong, I don't know.
I'll do some own research in to continuing to write to files in C++, but gladly respond, if you know what is wrong.

Thank you both for the great debug tips! Completely forgot about hexdump/binary compare!
Tomas Berger
togtja_at_gmail.com (https://link.getmailspring.com/link/7D8F25C6-CD44-4182-B10E-9CD844857608_at_getmailspring.com/0?redirect=mailto%3Atogtja%40gmail.com&recipient=Y3VybC1saWJyYXJ5QGNvb2wuaGF4eC5zZQ%3D%3D)

On Dec 8 2020, at 5:13 pm, Tomalak Geret'kal via curl-library <curl-library_at_cool.haxx.se> wrote:
>
> On 08/12/2020 15:49, Tomas Berger via curl-library wrote:
>
> >
> > I have been trying to be able to continue a download from a half finished state, my write function is rather simple, but I have a couple of couts.
> > The idea is to download to a temporary file, and when the download is complete copy, and remove the tempfile
> > Relying on CURLOPT_RESUME_FROM_LARGE to help me continue from where I left of, with the size of what was downloaded before as its parameter
> > I also rewrite 2 bytes, so that my cout can make sure that the same offsets get the same bytes
> >
>
>
> > But the zip file can not be unzipped, and their sha256 sum is different, so something went wrong.
> > I assume I might have made an logical error, but I can see where.
> > I hope you guys can help me see where I have gone wrong.
>
> Compare the bytes of the file you wrote to the bytes of your reference file (ZIP).
>
> Where do they diverge? Do they diverge near your "join"? Elsewhere?
> Are you sure this problem relates to the "continue" semantics? Your output doesn't seem to prove that.
> Cheers
> -------------------------------------------------------------------
> Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html



-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Received on 2020-12-09