I totally agree with Alan
Had the same problem a while back and solved it just this way. Use the
callbacks only to what they are designated for, i.e. reading callback
reads the data, writing callback writes it. Don't forget that libcurl is
only using those functions to actually do its work :), so if you need to
do something with the data, only do that before / after the
curl_easy_perform()...

Alan Wolfe wrote:
> Actually i think i know what your issue is. Curl is single threaded,
> even if you use the multi interface. If you are doing a bunch of work
> in the callback function, that means that the main thread (the only
> thread) is doing that, and not processing data waiting on the line etc.
>
> Also, usually it's a bad idea to spend a lot of time in callbacks
> (whether we are talking about libcurl, another library, or even your
> operating system's APIs) and can often cause problems if you do.
>
> You could have your other data transfer happen on another thread, or
> you could write it to disk and fire up another process to send the
> data to wherever you are forwarding it on to.
>
> Hope this helps!
>
>
>
>
> On Sat, Jan 5, 2013 at 12:09 AM, ilavarasan M <m.ilavarasan_at_gmail.com
> <mailto:m.ilavarasan_at_gmail.com>> wrote:
>
> Hi,
>
> I'm really stuck up with this problem. can you please help out..?
> is there anything I need to check to solve this issue?
>
> Regards,
> Ilavaa
>
>
> On Fri, Dec 28, 2012 at 12:59 PM, ilavarasan M
> <m.ilavarasan_at_gmail.com <mailto:m.ilavarasan_at_gmail.com>> wrote:
>
> Hi,
>
> Thanks for your reply. I'm yet to investigate the network
> connection also I can't get the server side logs now. From my
> analyzis I found there will be no network problems happend
> because when i ran separate utility which downloads 60GB file
> successfully. In my previous program, I do the following process
>
> 1. Download the contents from remote url
> 2. In the call back function, data will be copied to another
> buffer which size is 10 MB.
> 3. And once the buffer is filled 10 MB i'll send to another
> location.
> 4. Until the buffer become empty, curl will not download any
> contents.
> 5. Once buffer emptied curl starts downloading again.
>
> I feel that, Curl is exited due to the sending to the another
> location time is high. Is there any time out or delay for curl
> call back function. I'm able to reproduce the issue by put
> sleep in the call back function. If there is no delay in
> downloading content, everything works fine. I have attaced my
> code.
>
> Is there any option I need to set for this delayed response?
>
> Thanks & Regards,
> Ilavaa.
>
> On Fri, Dec 21, 2012 at 6:02 PM, Daniel Stenberg
> <daniel_at_haxx.se <mailto:daniel_at_haxx.se>> wrote:
>
> On Fri, 21 Dec 2012, ilavarasan M wrote:
>
> I'm using the latest version 7.28.1 to download the
> files. Im trying to download the file which is 60 GB
> in size using cURL (C++). I'm able to get around 17 GB
> of content and suddenly cURL come out with return code
> as success and the http response code is 200. I try to
> debug and found that curl comes out when it receiving
> the buffer size less than 16K in the callback
> function. I'm getting random values like 4K or 13K. I
> can confirm the issue is because of this.
>
>
> No. That is just another sympthom, not the cause, of the
> problem. libcurl will provide you with as much data as it
> was able to read from the socket.
>
>
> Is there any way to fix this?
>
>
> We don't know what your problem is only based on this. We
> don't know what the problem is!
>
> If I were to investigate that problem I'd A) check the
> server side logs for when the failure occurs and B)
> wireshark the network and investigate the TCP action at
> the time of the transfer stop.
>
> So curl_easy_perform() returns OK for this transfer? Is
> that an old-style HTTP close connection to indicate end of
> transfer?
>
>
> can i set the CURL_MAX_WRITE_SIZE
>
>
> That's a compile-time value you can alter when you rebuild
> libcurl, but you changing that won't fix this problem...
>
> --
>
> / daniel.haxx.se <http://daniel.haxx.se>
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/list/listinfo/curl-library
> Etiquette: http://curl.haxx.se/mail/etiquette.html
>
>
>
>
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/list/listinfo/curl-library
> Etiquette: http://curl.haxx.se/mail/etiquette.html
>
>
>
>
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/list/listinfo/curl-library
> Etiquette: http://curl.haxx.se/mail/etiquette.html
>
>
> No virus found in this message.
> Checked by AVG - www.avg.com <http://www.avg.com>
> Version: 2012.0.2221 / Virus Database: 2637/5510 - Release Date: 01/04/13
>

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2013-01-05