Steve Holme wrote:
> Hi Rich,
>
>> 1. Libcurl is returning message data line-by-line, with two
>> callbacks per line - one for the line data and the other for the
>> CRLF. This seems like strange behavior. I'd coded as if I were
>> getting the data off a TCP connection - might get one byte,
>> might get the whole message in one shot, might get anything
>> between. So I'm curious as to what the intent is here.
>
> Yes - that's by design.
>
> You could quite genuinely argue that it is strange behaviour and wouldn't it
> be better to pass data on block by block as it comes off the socket.
>
> However, there were problems with the existing CRLF.CRLF checking code that
> needed fixing and it was generally simpler for Daniel and I to implement
> this checking code this way. With this fix, I did question whether I should
> buffer the lines back up before passing them onto whatever application was
> using libcurl but I quite quickly came to the assumption that most
> applications, will probably buffer it up themselves... thus it seemed quite
> inefficient for libcurl to buffer up lines of data whilst checking for the
> dot and then for an application programmer to do roughly the same once
> received from libcurl.
>
> The reason for the two callbacks is that the checking code will send you all
> the data up to a CRLF, which it then has to buffer (if my memory serves me
> correctly) in case the CRLF.CRLF runs over the end of a packet into the
> next. The second callback is when sufficient data has been received by
> libcurl and it has realised that the line wasn't a CRLF.CRLF so it then
> passes the CRLF onto you.
>
> It should be possible to get libcurl to send the block of data it receives
> off the connection if it doesn't contain any part of the CRLF.CRLF less any
> part of that if it is received at the end of the packet, but both Daniel and
> I produced a few patches in an attempt to fix this, all of which failed the
> test harnesses - However, we are open to patches that are able to pass the
> data on in less calls it if you fancy the challenge ;-)

I pretty much figured this was the case. I had thought that "all" the code
had to do was implement a state machine to track the line ending cruft as it
passed the data to the caller, but then I realized that it must do the dot
destuffing (right?) so may have to modify the data in the case of a
CRLF..CRLF(?) Still, given that the CRLF at the end of the line IS part of
the data (#2), it should be possible to deliver it with the data. It is
start of the following line which can't be passed to the user before being
checked for .CRLF (end of data) or ..CRLF (pass as .CRLF), right?

I'm certainly not going to quibble about implementation details, given I'm
just a libcurl user at this point. As a user, I want to only express
gratitude to the developers and never even appear to demand anything.
Sometimes I can't help but comment though... ;) Currently, I'm off on
another, non-curl project for a while, but I do look forward to coming back
to this and taking a look at the code. (I seem to have +16 years of runtime
on you. Contrary to what Daniel said, I seem to have less time. My wife
and I must be doing something wrong - I know! It's the kids!!)

>> 2. Libcurl is dropping the final CRLF from the data.
>
> It's probably a combination of 1) me being slightly lazy and 2) me
> misinterpreted the spec :(
>
> I should have picked that up, but the existing code that had problems,
> stripped the final CRLF off so I maintained compatibility with that rather
> than questioning what the existing code was doing and fixing it. A fix
> should be fairly simple to do so I will have a go at that later ;-)

Sorry for lack of snippage, I couldn't figure out what to snip!

Cheers!
Rich
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2012-02-15