Hi Ray,

Thanks for your detail response. For my application disabling
transfer-encoding seems to be a good option. Then I will do the decoding
of the chunked message in my wrapper itself.

- Ganesh

On 20-Jun-15 04:22 AM, Ray Satiro via curl-library wrote:
> On 6/19/2015 7:04 AM, Ganesh Nikam wrote:
>> The problem I am facing with curl is that, my write_callback is
>> getting called multiple times for single http response. As per the
>> documentation this callback can be called multiple times in some
>> cases. My query is in what scenarios it will be called multiple times
>> ? How curls decides it? Here is the text from documentation:
>>
>> /"The callback function will be passed as much data as possible in
>> all invokes, but you must not make any assumptions. It may be one
>> byte, it may be thousands. The maximum amount of body data that will
>> be passed to the write callback is defined in the curl.h header file:
>> CURL_MAX_WRITE_SIZE (the usual default is 16K)".
>> /
>> Some queries on this point:
>> 1. What do you mean by "as much as possible" here ? how libcurl
>> decides it ?
>> 2. In my case, my complete http response boby is of 2155 bytes. For
>> such small response also libcurl breaks the message in two callbacks.
>> First with 1295 byte and other is 860 byte. What could be the reasons for
>> calling callback two times for such small message size? My server
>> response is not going to be more than 5K.
>> 3. I want to get the complete http response in single write_callbck
>> function. But as per below thread it is not possible:
>> http://curl.haxx.se/mail/lib-2003-09/0174.html
>>
>> I have implemented wrapper class over the libcurl api. My
>> applications requirement is that, this wrapper class should return
>> complete http response to it. Now I would like to collect the
>> multiple chunks (received
>> from multiple write_callback function calls) together. For that I
>> want to know he complete size of the http response. is there is any
>> way to get the complete size of the http response ? I checked
>> easy_get_opt flags, but
>> I didn't find any relevant flag. Can you please suggest any way
>> to achieve this ?
>>
>> My http server sends response in chunked encoding. is there a way
>> to get the chunked response size ?
>>
>> libcurl version is 7.40 and the platform is Ubuntu 12.04
>
>
> libcurl passes a received chunk of (parsed) data up to
> CURL_MAX_WRITE_SIZE to your write callback. As noted in the
> documentation do not make any assumptions. "as much as possible" does
> not necessarily mean all received raw data even if that data is less
> than the max due to parsing. This is true in your case and other cases
> (like IMAP body I think may be passed to the write callback line-by-line).
>
> For example you make a request and get a reply like this that all
> arrives at once:
>
> ===================================================
> HTTP/1.1 200 OK
> Transfer-Encoding: chunked
>
> 3
> FOO
> 3
> BAR
> 0
>
> ===================================================
>
> libcurl in readwrite_data will call Curl_read to read the raw data [1]
> and it gets that whole thing. First it needs to handle the headers so
> it goes into parse-the-header-mode and then after that it can
> determine how to handle the body. In this case the encoding is chunked
> (as determined by the headers) and it will start to parse the body at
> the first '3' by parsing the chunks, which it does in
> Curl_httpchunk_read, which actually also writes away the data [2]. So:
>
> The first time the write callback is called with data "FOO", len 3
> The second time the write callback is called with data "BAR", len 3
>
> You do not need to know the complete body size to collect the multiple
> chunks together. As you receive the response write it somewhere, eg to
> memory [3]. As you can see in that memory example if you need more
> memory just reallocate. Considering your performance requirements you
> may have to do some adjustments.
>
> As to your question on content length, chunked encoding is used when
> the server does not know the length of content it will be returning at
> the time it starts the reply, but even if it does the content-length
> header isn't allowed. In other words you cannot get the complete size
> of the body until the transfer is complete. The only way that may be
> possible is in the most unlikely of configurations if your server
> sends a special header like content-length (but not content-length)
> but I don't know why anyone would do that (unless it was sending a
> really large file or something? still seems weird though).
>
>
> [1]:
> https://github.com/bagder/curl/blob/curl-7_43_0/lib/transfer.c#L422-L423
> [2]:
> https://github.com/bagder/curl/blob/curl-7_43_0/lib/http_chunks.c#L185-L205
> [3]: http://curl.haxx.se/libcurl/c/getinmemory.html
>
>
>
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/list/listinfo/curl-library
> Etiquette: http://curl.haxx.se/mail/etiquette.html

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2015-06-22