cURL / Mailing Lists / curl-library / Single Mail

curl-library

Downloading unlimited file. Some clarification please.

From: Fadi Kahhaleh <fkahhaleh_at_hotmail.com>
Date: Tue, 3 Jan 2012 07:31:49 +0000

Hi, I was introduced to LibCURL by looking at an open source project. I am using LibCURL to download files from a local server
in my network. One gotcha about the file I am downloading is that it is endless!think of it as a dynamically generated file that contains information, so the file doesn't really physically exist as much as
the server makes it seamless to the client that is downloading it and you can go on and on downloading it. I have a strange behavior thus writing this email asking for your help! is there any option that i need to set? to make this work? I am noticing that after some period (which is random at best) my call back function starts to get small buffer sizes (i.e 300-ish bytes)compared to the a few hundered Kbytes i guess. or whatever the max download unit is set by default to. I am storing the bytes read in a Ring Buffer that is processed by another thread.Using 1 Mb of memory for the Ring Buffer and having a Mutex set/reset in both threads for thread-safe access. I tried reading a local file from my HDD using the above scenario (Threads, RingBuffer...etc) it worked well. So I can safely say that my implementation
works in that area. However once I go HTTP download I get this problem.I tried to download a fixed file size (but tried a large file approx. 350 Mb in size) and it caused the same behavior. Is there anything to consider? Setting Verbose mode I got this: * About to connect() to localhost port 6498 (#0)
* Trying 127.0.0.1... * connected
> GET /Some/Path/to/aPage HTTP/1.1
User-Agent: libcurl-agent/1.0
Host: localhost:6498
Accept: */** HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Cache-Control: no-cache, must-revalidate
< Last-Modified: Tue, 03 Jan 2012 06:45:43 GMT
< Content-Type: text/plain
< Pragma: no-cache
<
* Failed writing body (0 != 16384)
* Closing connection #0
* Failed writing received data to disk/application
* About to connect() to localhost port 6498 (#0)
* Trying 127.0.0.1... * connected
> GET /Some/Path/to/aPage HTTP/1.1
User-Agent: libcurl-agent/1.0
Host: localhost:6498
Accept: */** HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Cache-Control: no-cache, must-revalidate
< Last-Modified: Tue, 03 Jan 2012 06:45:43 GMT
< Content-Type: text/plain
< Pragma: no-cache
<
   
Regards,

 

Fadi .K
                                               

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2012-01-03