Bugs item #1854175, was opened at 2007-12-19 16:24
Message generated for change (Comment added) made by bagder
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1854175&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: libcurl
Group: wrong behaviour
Status: Open
Resolution: None
Priority: 6
Private: No
Submitted By: Spacen Jasset (sjasset)
Assigned to: Daniel Stenberg (bagder)
Summary: Digest proxy auth with CURLOPT_READFUNCTION fails / behavior

Initial Comment:
We are using curl 7.17.0 on various platforms.

We set:

curl_easy_setopt(curl, CURLOPT_PROXYAUTH, CURLAUTH_ANY);

and use a CURLOPT_READFUNCTION.

We use squid or ISA server as a proxy with Digest authentication enabled.

There are two things that happen number (2) depends on timing.

(1)
We make a largeish size post (> 64K) of misc (binary) data using the read callback.
Curl recieves a 407 from squid.
Curl sends us a CURLIOCMD_RESTARTREAD which we honor
Curl then starts sending the post data again on the same connection

Curl then re-tries with digest authentication and succeeds.

The problem here is that curl rewinds the data stream and yet still sends all the data -- plus the bit it sent before the rewind. See case1.log

(2) Like (1) except the proxy server closes the connection as soon as it realises that curl is sending more post data than it should, the request sequence then ends in a failure. This is harder to reproduce, and in fact is far more prevalent on an slow or loaded machine. See case2.log

Because we see (2) quite often I have compiled a patch, which passes the unit tests I have run. The patch may not be in the correct place, and may not be proper as I don't have a full understanding of how the flags work.

RCS file: /cvsroot/3rdParty/curl-7.17.0/lib/transfer.c,v
retrieving revision 1.1
diff -u -r1.1 transfer.c
--- transfer.c 26 Sep 2007 14:05:39 -0000 1.1
+++ transfer.c 18 Dec 2007 17:20:25 -0000
@@ -656,6 +656,8 @@
                   infof(data, "Keep sending data to get tossed away!\n");
                   k->keepon |= KEEP_WRITE;
                 }
+ else
+ k->keepon &= ~KEEP_WRITE;
               }
 #endif /* CURL_DISABLE_HTTP */

See with_patch.log

----------------------------------------------------------------------

>Comment By: Daniel Stenberg (bagder)
Date: 2008-01-08 11:20

Message:
Logged In: YES
user_id=1110
Originator: NO

Ok, with #1850730 just closed, is the issue still happening for you?

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-12-21 14:23

Message:
Logged In: YES
user_id=1110
Originator: NO

Ok, so let's focus on 1850730 first then. Can you provide us with a better
trace log of what is sent and received for that bug? That bug is still
awaiting further input from you.

----------------------------------------------------------------------

Comment By: Spacen Jasset (sjasset)
Date: 2007-12-21 13:58

Message:
Logged In: YES
user_id=1400532
Originator: YES

Hmm. Bug 1850730 isn't the same, but actually they are closely related,
and this bug can't be reproduced until 1850730 is eliminated, which we are
doing by rewinding the stream ourselfs when it reaches the end. See the
attached test program, it reproduces bug #1850730. If then you eliminate
bug 1850730 then it will also reproduce this bug (1854175)
File Added: test_program.tar.gz

----------------------------------------------------------------------

Comment By: Spacen Jasset (sjasset)
Date: 2007-12-21 12:11

Message:
Logged In: YES
user_id=1400532
Originator: YES

I will produce some debug logs.

>And no, the "2007-12-20 15:59" patch doesn't make sense since in
>lib/transfer.c:1432 (in current CVS), just before the Curl_readrewind()
>function is called, that bit is already cleared.

This is true, but the flag isn't cleared when calling perhapsrewind in
http.c. This is the problem. perhapsrewind is called from
Curl_http_auth_act, which selects an authentication method and detects a
407 and rewinds the steam but doesn't clear the flag

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-12-20 23:27

Message:
Logged In: YES
user_id=1110
Originator: NO

Isn't this bug report just a variation/duplicate of #1850730 ?

And no, the "2007-12-20 15:59" patch doesn't make sense since in
lib/transfer.c:1432 (in current CVS), just before the Curl_readrewind()
function is called, that bit is already cleared.

I'm looking at your two logs from case 1 and case 2 but I have a very hard
time to understand them. Can you use DEBUGFUNCTION instead and do complete
logs similar to the debug.c example
(http://curl.haxx.se/lxr/source/docs/examples/debug.c) just so that I can
see both incoming and outgoing headers and data.

----------------------------------------------------------------------

Comment By: Spacen Jasset (sjasset)
Date: 2007-12-20 15:59

Message:
Logged In: YES
user_id=1400532
Originator: YES

We're not using the following patch, which is more reasonable. KEEP_WRITE
is cleared when a rewind happens. This may or may not be better placed in
the perhapsrewind function of http.c

RCS file: /cvsroot/3rdParty/curl-7.17.0/lib/transfer.c,v
retrieving revision 1.1
diff -u -r1.1 transfer.c
--- transfer.c 26 Sep 2007 14:05:39 -0000 1.1
+++ transfer.c 20 Dec 2007 12:01:08 -0000
@@ -237,6 +237,7 @@
      (data->set.httpreq == HTTPREQ_POST_FORM))
     ; /* do nothing */
   else {
+ data->reqdata.keep.keepon &= ~KEEP_WRITE;
     if(data->set.ioctl_func) {
       curlioerr err;
 

----------------------------------------------------------------------

Comment By: Spacen Jasset (sjasset)
Date: 2007-12-19 17:23

Message:
Logged In: YES
user_id=1400532
Originator: YES

Unfortunatly, the patch introduces serious problems when not using a proxy
and so is not useful. Why it seemingly passed the unit tests I am not
sure.

The original problem still remains.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1854175&group_id=976
Received on 2008-01-08