Hi John,

On Tue, 20 Aug 2013, Steve Holme wrote:

> I think I might have been a little premature with that statement as
> the "Written xxxx bytes, yyyyyy are left for transfer" is only output
> in the fetch response function and not during "transfer" mode
> which libcurl will be in if the data doesn't fit in one packet /
> response - so I think what I posted was the correct output.
>
> I will report my progress.

I got a little further last night, after we spoke, but I don't have a 100%
fix at the moment. However I wanted to post my findings to the list to see
if anyone has any ideas how to proceed with this...

The excess message and lack of last write happens because the bytecount /
max download values don't make sense to the transfer code (in transfer.c's
readwrite_data() line 623 onwards). In the imap code we have said that our
byte count is 1374 and that the max download is 262 bytes. Note only that
but when we get to transfer.c line 623 the nread is 282 bytes as the end of
the IMAP FETCH and IMAP response is tacked on to the end (20 characters in
all). So the code test to see if 1374 + 282 >= 262 which it is and then goes
on to calculate the excess and nread values - that later being 262 - 1374
which is incorrect :(

We can fix this by modifying the setup of the transfer (in imap.c at line
1435) to reset the bytecount to zero as follows:

      /* IMAP download */
      data->req.bytecount = 0;
      data->req.maxdownload = size;
      Curl_setup_transfer(conn, FIRSTSOCKET, size, FALSE, NULL, -1, NULL);

Unfortunately this then creates a second problem for some SSL based
connections in that the connection then appears to hang :(

This is caused by the main transfer loop as it says while(data_pending()) -
If I step into this code under the debugger then the underlying SSL function
tells me we have 20 bytes of data waiting.

If we change the while(data_pending()) to something like
while(data_pending() && k->keepon) to force the loop to exit, when it is
done receiving data, that then creates problem number 3 in imap_done() and
the pingpong layer.

imap_done() calls the pingpong state machine as it wants those lasts 20
bytes, however, Curl_socket_ready() in pingpong.c tells us there is no data
waiting and subsequently goes round the loop waiting for it :(

I'm not too sure why we are seeing this with only gmail (at present) and not
Exchange but both John and I have tested gmail and Exchange with Windows
builds of curl using both SChannel and OpenSSL - John has even tried a
Ubuntu build and verified the problem there as well.

It seems to be that Exchange sends more of the email content in a packet
rather the splitting it up like gmail does but then it sends the end of the
IMAP FETCH and the following response in a separate packet unlike gmail
which is adding onto the second packet of the email.

I don't know whether the imap code is unique within curl in that it wants to
do some pingpong stuff, perform a transfer and then carry on doing pingpong
stuff.

One fix would be to modify the imap_done() code to not wait for the response
- generally speaking if we have the size from the initial FETCH response,
have setup the transfer and completed it successfully then we know things
worked.

Another would be try and work out how to retrieve that extra data in the
transfer code and pass it down to the cache in the pingpong layer.

Any assistance / thoughts would be appreciated.

Kind Regards

Steve
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2013-08-21