My apologies if this ends up at wrong place in the mailing list. For some reason an email wasn't sent to me when a reply was made. As such, I was unable to directly reply to the previous response by Guenter.

I've tracked it down, or so I believe, to LWP+https being the cause here. I've managed to replicate the same issue on Debian 6.0 (Squeeze) as well as the OS X Lion system I run. As best I can tell, when LWP is downloading the file over HTTPS sometimes the request ends early. I was not able to replicate using curl, wget, or Google Chrome.

I tested this my commenting out the '$ua->mirror($url, $txt);' line, and replacing it with the following line:

    print $ua->get($url)->content . "\n"; exit(0);

The first time I ran the script the entire file was downloaded. However, it was a fluke, as the following request attempts varying amounts of the file contents were downloaded. And I was never able to print the full file again. I even tried playing with the "max_size" option in UserAgent. This had no affect, as expected, because the "max_size" option defaults to undef (unlimited). When switching to http, it works perfectly every single time.

I've tried to do some digging and have been unable to find a discussion regarding the change from HTTP to HTTPS. Additionally, I can't say I see any benefit of doing so as the information is publicly accessible. I would be interested in why it should not remain HTTP.

-Tim

---
Tim Heckman

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2012-03-26