Bugs item #3538625, was opened at 2012-06-28 02:31
Message generated for change (Comment added) made by marcelraad
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=3538625&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: libcurl
Group: wrong behaviour
Status: Open
Resolution: Fixed
Priority: 5
Private: No
Submitted By: Marcel Raad (marcelraad)
Assigned to: Daniel Stenberg (bagder)
Summary: NTLM proxy authentication broken?

Initial Comment:
I use a proxy server that requires NTLM authentication (Microsoft Forefront Threat Management Gateway) and the cURL easy interface with the CONNECT method. CURLOPT_CONNECT_ONLY and CURLOPT_HTTPPROXYTUNNEL are set to true. I only allow NTLM and plain authentication in my code. Until libcurl 7.24.0, NTLM authentication with SSPI was working correctly.
Since commit 41b02378342322aa8e264260057502f4d7493239 ("CONNECT: made generically not per-protocol", libcurl 7.25.0), the proxy always returns "407 Proxy Authentication required". When I use plain authentication, the connection works.

I have the following options set in my libcurl build:
USE_WINDOWS_SSPI
CURL_STATICLIB
HTTP_ONLY
CURL_DISABLE_COOKIES
CURL_DISABLE_IMAP
CURL_DISABLE_POP3
CURL_DISABLE_SMTP
CURL_DISABLE_GOPHER

----------------------------------------------------------------------

>Comment By: Marcel Raad (marcelraad)
Date: 2012-07-06 15:34

Message:
I've tested today's snapshot (20120706) and it works perfectly again, and
so does the new support for unicode user names with SSPI. Thank you!

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2012-07-05 03:29

Message:
I've just now pushed code that seems to fix the problem for me. The
specific fix for the code is this:
https://github.com/bagder/curl/commit/ed7bfeee7aac5ffb8edc074fc704ce888f12ad0f

but I did a series of further fixes too to make sure that we don't
introduce this problem again in the future

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2012-07-04 07:20

Message:
Thanks, I tried that but it doesn't completely solve the problem - I re-run
all the NTLM test cases now and pay extra attention to closed connections.
I'm on it and will provide my suggested patch in another day or two.

----------------------------------------------------------------------

Comment By: Marcel Raad (marcelraad)
Date: 2012-07-03 23:41

Message:
Thank you for looking into this! Here is a unified diff:

 lib/http_proxy.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/http_proxy.c b/lib/http_proxy.c
index 01502fd..95c2abb 100644
--- a/lib/http_proxy.c
+++ b/lib/http_proxy.c
@@ -69,6 +69,7 @@ CURLcode Curl_proxy_connect(struct connectdata *conn)
     prot_save = conn->data->state.proto.generic;
     memset(&http_proxy, 0, sizeof(http_proxy));
     conn->data->state.proto.http = &http_proxy;
+ conn->bits.close = FALSE;
     result = Curl_proxyCONNECT(conn, FIRSTSOCKET,
                                conn->host.name, conn->remote_port);
     conn->data->state.proto.generic = prot_save;
@@ -264,6 +265,7 @@ CURLcode Curl_proxyCONNECT(struct connectdata *conn,
        2) we're in multi-mode and we didn't block - it's either an error
or we
        now have some data waiting.
        In any case, the tunnel_connecting phase is over. */
+ conn->tunnel_state[sockindex] = TUNNEL_INIT;
 
     { /* BEGIN NEGOTIATION PHASE */
       size_t nread; /* total size read */

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2012-07-03 13:50

Message:
Thanks for this, I've now come to understand that this problem exists for
me as well on Linux. The problem has been that the NTLM test cases for this
have been faulty so they didn't detect the broken connection in the midst
of the NTLM authentication - which with a real proxy of course doesn't
work!

I'll use your hint to try things out. Can you post a diff with your fix so
that I don't misunderstand how you made it?

----------------------------------------------------------------------

Comment By: Marcel Raad (marcelraad)
Date: 2012-07-03 00:18

Message:
Both CURLOPT_TIMEOUT and CURLPOPT_CONNECT_TIMEOUT are set to 5 seconds.

I have finally found the cause for the timeout: no response is set to the
second 407 from the proxy because with commit
41b02378342322aa8e264260057502f4d7493239, a "conn->bits.tunnel_connecting =
FALSE" was removed before "BEGIN NEGOTIATION PHASE" in Curl_proxyCONNECT
(http_proxy.c). Thus, the "if(TUNNEL_INIT ==
conn->tunnel_state[sockindex])" block at the beginning of the loop will
only be executed once.

If I add back a "conn->tunnel_state[sockindex] = TUNNEL_INIT" (in addition
to the "conn->bits.close = FALSE" in Curl_proxy_connect before the call to
Curl_proxyCONNECT), NTLM authentication works again.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2012-07-02 14:18

Message:
And you have no special timeout set? Can you set a break-point in there or
otherwise get the values of the relevant variables that are used to trigger
the timeout condition?

----------------------------------------------------------------------

Comment By: Marcel Raad (marcelraad)
Date: 2012-06-29 04:23

Message:
Adding "conn->bits.close = FALSE" before the call to Curl_proxyCONNECT
fixes the problem for revisions between
41b02378342322aa8e264260057502f4d7493239 and
805788e0434f4f09d8049c51000af604efb800ed, but
c83de6d07625b813e3bbc31f9a0827c3a0007355 ("CONNECT: fix multi interface
regression") broke that too. Now I get a timeout.

The libcurl verbous output is:

Working (805788e0434f4f09d8049c51000af604efb800ed):

* About to connect() to proxy 192.168.7.19 port 8080 (#0)
* Trying 192.168.7.19...
* connected
* Connected to 192.168.7.19 (192.168.7.19) port 8080 (#0)
* Establish HTTP proxy tunnel to XXX:443
> CONNECT XXX:443 HTTP/1.1

Host: XXX:443

User-Agent: XXX

Proxy-Connection: Keep-Alive

< HTTP/1.1 407 Proxy Authentication Required ( Forefront TMG requires
authorization to fulfill the request. Access to the Web Proxy filter is
denied. )

< Via: 1.1 XXX

< Proxy-Authenticate: Negotiate

< Proxy-Authenticate: Kerberos

< Proxy-Authenticate: NTLM

< Connection: close

< Proxy-Connection: close

< Pragma: no-cache

< Cache-Control: no-cache

< Content-Type: text/html

< Content-Length: 722

<

* Ignore 722 bytes of response-body
* Received HTTP code 407 from proxy after CONNECT
* About to connect() to proxy 192.168.7.19 port 8080 (#0)
* Trying 192.168.7.19...
* connected
* Connected to 192.168.7.19 (192.168.7.19) port 8080 (#0)
* Establish HTTP proxy tunnel to XXX:443
* Proxy auth using NTLM with user 'XXX'
> CONNECT XXX:443 HTTP/1.1

Host: XXX:443

Proxy-Authorization: NTLM
TlRMTVNTUAABAAAAt4II4gAAAAAAAAAAAAAAAAAAAAAGAbEdAAAADw==

User-Agent: XXX

Proxy-Connection: Keep-Alive

< HTTP/1.1 407 Proxy Authentication Required ( Access is denied. )

< Via: 1.1 XXX

< Proxy-Authenticate: NTLM
TlRMTVNTUAACAAAAFAAUADgAAAA1goni3nAXUZ3kvzAAAAAAAAAAANIA0gBMAAAABgGxHQAAAA9SAE8AUwBTAE0AQQBOAEkAVABIAAIAFABSAE8AUwBTAE0AQQBOAEkAVABIAAEAFgBTAFIAVgAwADgAUgAyAC0ASQBTAEEABAAkAGkAbgB0AC4AcgBvAHMAcwBtAGEAbgBpAHQAaAAuAGMAbwBtAAMAPABzAHIAdgAwADgAcgAyAC0ASQBTAEEALgBpAG4AdAAuAHIAbwBzAHMAbQBhAG4AaQB0AGgALgBjAG8AbQAFACQAaQBuAHQALgByAG8AcwBzAG0AYQBuAGkAdABoAC4AYwBvAG0ABwAIAPi+NoLnVc0BAAAAAA==

< Connection: Keep-Alive

< Proxy-Connection: Keep-Alive

< Pragma: no-cache

< Cache-Control: no-cache

< Content-Type: text/html

< Content-Length: 0

<

* Establish HTTP proxy tunnel to XXX:443
* Proxy auth using NTLM with user 'XXX'
> CONNECT XXX:443 HTTP/1.1

Host: XXX:443

Proxy-Authorization: NTLM
TlRMTVNTUAADAAAAGAAYAGwAAAAYABgAhAAAAAAAAABYAAAACAAIAFgAAAAMAAwAYAAAABAAEACcAAAANYKI4gYBsR0AAAAP1wgkLIYnVBbIe55UQx24zXIAYQBhAGQARABFAFYALQA0ADIABfam7z1nOc8AAAAAAAAAAAAAAAAAAAAAqzwvBFcRlcYI//6ne/cduTcI6mfNW9QRkX+Fjq96PtMsg7AzFqFMFQ==

User-Agent: XXX

Proxy-Connection: Keep-Alive

< HTTP/1.1 200 Connection established

< Via: 1.1 XXX

< Connection: Keep-Alive

< Proxy-Connection: Keep-Alive

<

* Proxy replied OK to CONNECT request
* Connection #0 to host 192.168.7.19 left intact

Not working (c83de6d07625b813e3bbc31f9a0827c3a0007355):

* About to connect() to proxy 192.168.7.19 port 8080 (#0)
* Trying 192.168.7.19...
* connected
* Connected to 192.168.7.19 (192.168.7.19) port 8080 (#0)
* Establish HTTP proxy tunnel to XXX:443
> CONNECT XXX:443 HTTP/1.1

Host: XXX:443

User-Agent: XXX

Proxy-Connection: Keep-Alive

< HTTP/1.1 407 Proxy Authentication Required ( Forefront TMG requires
authorization to fulfill the request. Access to the Web Proxy filter is
denied. )

< Via: 1.1 XXX

< Proxy-Authenticate: Negotiate

< Proxy-Authenticate: Kerberos

< Proxy-Authenticate: NTLM

< Connection: close

< Proxy-Connection: close

< Pragma: no-cache

< Cache-Control: no-cache

< Content-Type: text/html

< Content-Length: 722

<

* Ignore 722 bytes of response-body
* Received HTTP code 407 from proxy after CONNECT
* About to connect() to proxy 192.168.7.19 port 8080 (#0)
* Trying 192.168.7.19...
* connected
* Connected to 192.168.7.19 (192.168.7.19) port 8080 (#0)
* Establish HTTP proxy tunnel to XXX:443
* Proxy auth using NTLM with user 'XXX'
> CONNECT XXX:443 HTTP/1.1

Host: XXX:443

Proxy-Authorization: NTLM
TlRMTVNTUAABAAAAt4II4gAAAAAAAAAAAAAAAAAAAAAGAbEdAAAADw==

User-Agent: XXX

Proxy-Connection: Keep-Alive

< HTTP/1.1 407 Proxy Authentication Required ( Access is denied. )

< Via: 1.1 XXX

< Proxy-Authenticate: NTLM
TlRMTVNTUAACAAAAFAAUADgAAAA1goni/6p5/iqD8+cAAAAAAAAAANIA0gBMAAAABgGxHQAAAA9SAE8AUwBTAE0AQQBOAEkAVABIAAIAFABSAE8AUwBTAE0AQQBOAEkAVABIAAEAFgBTAFIAVgAwADgAUgAyAC0ASQBTAEEABAAkAGkAbgB0AC4AcgBvAHMAcwBtAGEAbgBpAHQAaAAuAGMAbwBtAAMAPABzAHIAdgAwADgAcgAyAC0ASQBTAEEALgBpAG4AdAAuAHIAbwBzAHMAbQBhAG4AaQB0AGgALgBjAG8AbQAFACQAaQBuAHQALgByAG8AcwBzAG0AYQBuAGkAdABoAC4AYwBvAG0ABwAIAChP4DPmVc0BAAAAAA==

< Connection: Keep-Alive

< Proxy-Connection: Keep-Alive

< Pragma: no-cache

< Cache-Control: no-cache

< Content-Type: text/html

< Content-Length: 0

<

* Proxy CONNECT aborted due to timeout
* Closing connection #0

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2012-06-28 08:05

Message:
That sounds like a plausible reason. Can you try adding it back in your
version and see if it fixes the problem? I personally can't try it out...

----------------------------------------------------------------------

Comment By: Marcel Raad (marcelraad)
Date: 2012-06-28 02:48

Message:
The problem seems to be that the TCP connection is not being reused
anymore. Prior to this revision, the two CONNECT requests with NTLM
information were sent on the same TCP connection. Now they are sent on
seperate TCP connections.
Perhaps the problem is that "conn->bits.close = FALSE" was set before the
proxy authentication when the Curl_proxyCONNECT call was in
Curl_http_connect, which is now missing in Curl_protocol_connect?

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=3538625&group_id=976
Received on 2012-07-07