Re: Continue Download C++ Implementation

From: Tomas Berger via curl-library <curl-library_at_cool.haxx.se>
Date: Wed, 9 Dec 2020 10:22:53 +0100

Just for closure, in-case anybody else get in my position, the answer to my issue, is found here

https://stackoverflow.com/questions/15063985/opening-a-binary-output-file-stream-without-truncation (https://link.getmailspring.com/link/EE92048F-377C-4C04-9AE9-29AE9347FB42_at_getmailspring.com/0?redirect=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F15063985%2Fopening-a-binary-output-file-stream-without-truncation&recipient=Y3VybC1saWJyYXJ5QGNvb2wuaGF4eC5zZQ%3D%3D)
Or in short; I need to open my binary file with both input and output mode.
Thank you for pointing me in the right direction!
Tomas Berger
togtja_at_gmail.com (https://link.getmailspring.com/link/EE92048F-377C-4C04-9AE9-29AE9347FB42_at_getmailspring.com/1?redirect=mailto%3Atogtja%40gmail.com&recipient=Y3VybC1saWJyYXJ5QGNvb2wuaGF4eC5zZQ%3D%3D)

On Dec 9 2020, at 10:00 am, Tomas Berger <togtja_at_gmail.com> wrote:
> > First - flush & close your file. It's almost certainly buffered and not all data may be actually written to disk when you invoke copy.
> I did actually close my failed after easy_perform returned, it just missed it when I copied to code. I did however try to flush the file.
> I tried after each write, and I tried before flush (not both at the same time though)
>
> > Are you sure this problem relates to the "continue" semantics?
> The main reason I believe it has to do with "continue" is that if I do it in one connection, It downloads perfectly. Both in the write callback and before the close after the file was finished/aborted
>
> The way I abort the download is to flip a Boolean that makes my progress callback return 1
> int progress(progressMeta* meta, curl_off_t totalDown, curl_off_t currentDown, curl_off_t totalUp, curl_off_t currentUp) {
> const std::lock_guard<std::mutex> lk{meta->mtx};
> if (meta->abort) {
> return 1;
> }
> //does other stuff related to keeping track of progress
> return 0;
> }
>
> > Compare the bytes of the file you wrote to the bytes of your reference file (ZIP).
> > Where do they diverge? Do they diverge near your "join"? Elsewhere?
>
> > If that doesn't help you could start debugging by dumping hex representation of the reference file and the one you ended up with. Find the first difference and go from there.
> This is where it got interesting:
> Contex for the split in hex, as i will compare hex dump;
> write to [51062d] = ca
> write to [51062e] = ec
> write to [51062f] = c2
> /Cut
> /Continue
> write to [51062e] = ec
> write to [51062f] = c2
> write to [510630] = dd
>
> The first 3 line from the failed transfer's hex dump looks like this:
> > 0000000 0000 0000 0000 0000 0000 0000 0000 0000
> > *
> > 0510620 0000 0000 0000 0000 0000 0000 0000 c2ec
> The "*" is not an edit from my side, to compare, the first 3 lines from the successful transfer
> > 0000000 4b50 0403 0014 0000 0008 6915 5183 e1ff
> > 0000010 a5c6 a583 0152 74a2 0154 0007 001c 786d
> > 0000020 6d49 6761 5565 0954 0300 d54a 5fc8 d54a
>
> And the "0510620", line in question in the successful transfer looks like this:
> > 0510620 a3d2 7c47 95e0 7595 7e7a 905d ca7a c2ec
> And from there on out, they are the same.
>
> This indicates to be that the continue from the offset is somewhat wrong, but what I am doing wrong, I don't know.
> I'll do some own research in to continuing to write to files in C++, but gladly respond, if you know what is wrong.
>
> Thank you both for the great debug tips! Completely forgot about hexdump/binary compare!
> Tomas Berger
> togtja_at_gmail.com (https://link.getmailspring.com/link/EE92048F-377C-4C04-9AE9-29AE9347FB42_at_getmailspring.com/2?redirect=mailto%3Atogtja%40gmail.com&recipient=Y3VybC1saWJyYXJ5QGNvb2wuaGF4eC5zZQ%3D%3D)
>
>
>
>
>
>
>
>
> On Dec 8 2020, at 5:13 pm, Tomalak Geret'kal via curl-library <curl-library_at_cool.haxx.se> wrote:
> >
> > On 08/12/2020 15:49, Tomas Berger via curl-library wrote:
> >
> > >
> > > I have been trying to be able to continue a download from a half finished state, my write function is rather simple, but I have a couple of couts.
> > > The idea is to download to a temporary file, and when the download is complete copy, and remove the tempfile
> > > Relying on CURLOPT_RESUME_FROM_LARGE to help me continue from where I left of, with the size of what was downloaded before as its parameter
> > > I also rewrite 2 bytes, so that my cout can make sure that the same offsets get the same bytes
> > >
> >
> >
> > > But the zip file can not be unzipped, and their sha256 sum is different, so something went wrong.
> > > I assume I might have made an logical error, but I can see where.
> > > I hope you guys can help me see where I have gone wrong.
> >
> > Compare the bytes of the file you wrote to the bytes of your reference file (ZIP).
> >
> > Where do they diverge? Do they diverge near your "join"? Elsewhere?
> > Are you sure this problem relates to the "continue" semantics? Your output doesn't seem to prove that.
> > Cheers
> > -------------------------------------------------------------------
> > Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
> > Etiquette: https://curl.se/mail/etiquette.html
>



-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Received on 2020-12-09