Before I start, I know I owe some updates to the threaded resolve patch, and another answer to Daniel about the persistent connections issue I reported, but as you can see below I've been hit by a barrage of people giving cURL a run for its money in the app I'm working on at VMware.

So, this is a flashback to this conversation:
http://www.mail-archive.com/curl-library@cool.haxx.se/msg00572.html

The app that I'm working on connects to a server via SSL to download a bunch of XML and checksums. Then, it finds a file to download in that XML, but then "downgrades" to HTTP to download that file. (This is to save ourselves from paying much more money to Akamai for increased SSL traffic for large, large payloads.)

I recently received a bug involving proxy NTLM authentication as set up on a Red Hat machine with a Squid/Samba NTLM/Active Directory setup. Basically, we connect to the server and download everything over SSL. Connections are reused, transfers complete, and all is well with the world.

Once that initial download stage quiesces, I verified that /proc/PID/fd does not contain any open sockets, yet the connection cache is still full of connections that are marked with 'inuse = TRUE'. (Which means that ConnectionKillOne never manages to reap any connection from the cache.)

So we try to connect to download the file, the subsequent HTTP connection doesn't fit into the cache and is immediately closed (url.c:4914 in CVS.) However, NTLM authentication needs to work this way in cURL:

--> GET http://foo.ourdomain.com/... HTTP/1.0 (no credentials -- CURLAUTH_ANY initial probe for methods)
<-- HTTP/1.0 407... Proxy-Connection: close
[connection severed]
--> GET http://foo.ourdomain.com/... HTTP/1.0 (NTLMSSP_NEGOTIATE)
<-- HTTP/1.0 407... NTLMSSP_CHALLENGE .. Proxy-connection: keep-alive
--> (reused) GET http://foo.ourdomain.com/... HTTP/1.0 (NTLMSSP_AUTH with appropriate credentials)

What I'm seeing in my app, however, is this:

--> GET http://foo.ourdomain.com/... HTTP/1.0 (no credentials -- CURLAUTH_ANY initial probe for methods)
<-- HTTP/1.0 407... Proxy-Connection: close
[connection severed]
--> GET http://foo.ourdomain.com/... HTTP/1.0 (NTLMSSP_NEGOTIATE)
<-- HTTP/1.0 407... NTLMSSP_CHALLENGE .. Proxy-connection: keep-alive
[connection severed due to 'connection cache full']
--> GET http://foo.ourdomain.com/... HTTP/1.0 (no credentials -- CURLAUTH_ANY initial probe)
<-- HTTP/1.0 407... NTLMSSP_CHALLENGE .. Proxy-connection: keep-alive
[connection severed due to 'connection cache full']
--> GET http://foo.ourdomain.com/... HTTP/1.0 (no credentials -- CURLAUTH_ANY initial probe)
... ad nauseum ...

Additionally, this causes cURL to falsely call our progress function and say that the file I asked for is 100% done downloading, which leads me to believe that cURL is confused!

Solutions/questions:
1) If all the connections have actually been severed on the socket level, why is the connection cache still full? Should it be empty by the point where all sockets have been closed and traffic has ceased? Or does the connection cache hold symbolic connections (i.e. cookie and authentication state preserving but not socket preserving?)
1.5) Our program uses the multi API with the socket_action workflow and it seems like we could plausibly responsible for keeping cURL state consistent with what sockets are actually open or closed. Could this be the source of the problem?
2) Assuming #1 is working as designed, we need to preserve NTLM state even if the initial NEGOTIATE results in a closed connection after the response comes in. I think the state machine might need another value to represent this so the state can be copied over to the new connection.

I think I'm talking out my butt in #2, but I think I've explained my predicament.

Any thoughts?

The whole thing wouldn't be a problem if the connection cache weren't always full, or if the NTLM subsystem had better support for a disconnection right after the CHALLENGE comes back. url.c:2639 looks like it contains code that will help to preserve the NTLM state over different connections but perhaps it isn't enough as-is?

Thanks...
-Josh
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2009-09-30