Hi Steve,

sorry for the delay, I was quite busy both during the rest of the week
and on the weekend, so I could not experiment with this more. But now
I think I cracked it!

The behavior you observed, i.e. that Curl_socket_ready() returns 0 but
Curl_read() succeeds, was really weird. I was sure it couldn't work
like that, because nothing else socket-related could work otherwise,
but I just did not see the reason, even though it is pretty simple:
the SSL.

When secure sockets are used, the SSL library generally reads the
input data in different chunks than cURL. It can happen that the
library still has much data in its buffers after the initial connect
or when the transfer finishes. Curl_socket_ready() then naturally
returns 0, because there are really no data waiting on the socket
itself. But Curl_read(), which invokes SSL functions when called, will
still return data fine. And this is why it can all start behaving
strangely in a random fashion.

Fortunately, checking whether there are more data to read waiting in
the SSL library input buffers is easy, because we have a
backend-agnostic function Curl_ssl_data_pending(). A simple patch for
pingpong.c which addresses the issue is attached.

Please note that although I think this patch will solve the original
problem with downloading emails, I ended up with it because of a
different problem. Just compiling cURL on Win32 with SSL support over
SChannel would not work with imaps:// connections at all. It just got
stuck when connecting, after the initial SSL handshake. And I traced
it down to the same problem, Curl_socket_ready() called from
Curl_pp_statemach() returning 0, even though there surely was the
server greeting waiting to be received. Then I found it was already
ready in the SSL library and this is how I realized the SSL is to
blame here.

Related to this behavior I saw (connection stuck after SSL handshake),
I send you 3 more patches (for IMAP, POP3 and SMTP) which speed up
secure connection establishment. Currently the connection's handler
needs to be invoked one more time after the handshake is done, which
takes extra time (1 second timeout for not socket activity in "easy"
operation, possibly never in multi mode?). The change was to make sure
the PP state machine is invoked right away after the handshake, so it
can process e.g. an already received greeting message without delays.

Hope that helps,
Jiri

On Thu, Sep 5, 2013 at 10:55 PM, Steve Holme <steve_holme_at_hotmail.com> wrote:
> On Thu 5 Sep 2013, Steve Holme wrote:
>
>> Does select() always indicate that there is data waiting to be read
>> even after previous reads or is it a one-time notification?
>
> Modifying Curl_pp_statemach() to perform a read into a temporary buffer
> seems to solve the problem:
>
> if(Curl_pp_moredata(pp))
> /* We are receiving and there is data in the cache so just read it */
> rc = 1;
> else if(!pp->sendleft && !Curl_read(conn, sock,buf, sizeof(buf), &nread)
> && nread) {
> /* We have data waiting but may not receive a FDREAD from select() */
> rc = 1;
> pp->cache = malloc(nread);
> pp->cache_size = nread;
> memcpy(pp->cache, buf, nread);
> }
> else
> rc = Curl_socket_ready(pp->sendleft?CURL_SOCKET_BAD:sock, /* reading */
> pp->sendleft?sock:CURL_SOCKET_BAD, /* writing */
> interval_ms);
>
> I don't like this, hence why the code isn't production ready (no failed
> check on the malloc call) but I'm not sure how else to tackle it at the
> moment.
>
> Is there a way to test if there is data waiting without using select() /
> peek at the socket buffer?
>
> Kind Regards
>
> Steve
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/list/listinfo/curl-library
> Etiquette: http://curl.haxx.se/mail/etiquette.html

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html