cURL / Mailing Lists / curl-library / Single Mail

curl-library

RE: NTLM, HTTP 100 Continue, and IIS 6 / .NET 1.1

From: Xiuping Hu <xhu_at_aventail.com>
Date: Thu, 25 Mar 2004 17:25:36 -0800

Good finding. This is the same result as I posted month ago, I have the
patch that is passed test on IIS 5.0/.NET and IIS 5.0/OWA2k.

Attached is the patch, it is patched against release 7.11.1.

Index: http.c
===================================================================
RCS file: /usr/aventail/prodroot/appliance/vendor/curl/src/lib/http.c,v
retrieving revision 1.1.1.7
diff -u -w -r1.1.1.7 http.c
--- http.c 2004/03/22 19:37:38 1.1.1.7
+++ http.c 2004/03/26 01:17:06
@@ -18,7 +18,7 @@
  * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY
OF ANY
  * KIND, either express or implied.
  *
- * $Id: http.c,v 1.1.1.7 2004/03/22 19:37:38 dyouatt Exp $
+ * $Id: http.c,v 1.1.1.3.24.3 2004/03/24 23:32:51 xhu Exp $
 
************************************************************************
***/
 
 #include "setup.h"
@@ -93,6 +93,7 @@
 #include "url.h"
 #include "share.h"
 #include "http.h"
 
 #define _MPRINTF_REPLACE /* use our functions only */
 #include <curl/mprintf.h>
@@ -1340,14 +1341,14 @@
 
     if(!req_buffer)
       return CURLE_OUT_OF_MEMORY;
-
+/* NEW CHANGE: move password to the bottom of the list. */
     /* add the main request stuff */
     result =
       add_bufferf(req_buffer,
                   "%s " /* GET/HEAD/POST/PUT */
                   "%s HTTP/%s\r\n" /* path + HTTP version */
- "%s" /* proxyuserpwd */
- "%s" /* userpwd */
                   "%s" /* range */
                   "%s" /* user agent */
                   "%s" /* cookie */
@@ -1356,14 +1357,13 @@
                   "%s" /* accept */
                   "%s" /* accept-encoding */
                   "%s" /* referer */
- "%s",/* transfer-encoding */
+ "%s"/* transfer-encoding */
+ "%s" /* proxyuserpwd */
+ "%s",/* userpwd */
 
                 request,
                 ppath,
                 httpstring,
- (conn->bits.httpproxy && conn->allocptr.proxyuserpwd)?
- conn->allocptr.proxyuserpwd:"",
- conn->allocptr.userpwd?conn->allocptr.userpwd:"",
                 (conn->bits.use_range && conn->allocptr.rangeline)?
                 conn->allocptr.rangeline:"",
                 (data->set.useragent && *data->set.useragent &&
conn->allocptr.uagent)?
@@ -1375,7 +1375,10 @@
                 (data->set.encoding && *data->set.encoding &&
conn->allocptr.accept_encoding)?
                 conn->allocptr.accept_encoding:"", /* 08/28/02 jhrg */
                 (data->change.referer &&
conn->allocptr.ref)?conn->allocptr.ref:"" /* Referer: <data> <CRLF> */,
- te
+ te,
+ (conn->bits.httpproxy && conn->allocptr.proxyuserpwd)?
+ conn->allocptr.proxyuserpwd:"",
+ conn->allocptr.userpwd?conn->allocptr.userpwd:""
                 );
 
     if(result)
@@ -1543,15 +1546,29 @@
       }
       break;
 
+ /* NEW CHANGE: Load Content-Length and 100 Continue after NTLM
authenticated. */
     case HTTPREQ_PUT: /* Let's PUT the data to the server! */
-
- if((data->set.infilesize>0) && !conn->bits.upload_chunky)
+ {
+ bool header_load = FALSE;
+ if((data->state.authwant == CURLAUTH_NONE) ||
+ (data->state.authwant == CURLAUTH_BASIC) ||
+ (strncmp(request,"POST",4)))
+ {
+ header_load = TRUE;
+ }
+ else if(authdone)
+ {
+ header_load = TRUE;
+ }
+ if( (header_load) && (data->set.infilesize>0) &&
!conn->bits.upload_chunky)
         /* only add Content-Length if not uploading chunked */
         add_bufferf(req_buffer,
                     "Content-Length: %" FORMAT_OFF_T "\r\n", /* size */
                     data->set.infilesize );
 
- if(!checkheaders(data, "Expect:")) {
+ if(authdone && !checkheaders(data, "Expect:")) {
         /* if not disabled explicitly we add a Expect: 100-continue
            to the headers which actually speeds up post operations (as
            there is one packet coming back from the web server) */
@@ -1579,7 +1596,7 @@
       if(result)
         return result;
       break;
-
+ }
     case HTTPREQ_POST:
       /* this is the simple POST, using x-www-form-urlencoded style */

Hope this can help.

Thanks,

Xiuping

 

> -----Original Message-----
> From: curl-library-bounces_at_cool.haxx.se [mailto:curl-library-
> bounces_at_cool.haxx.se]
> Sent: Thursday, March 25, 2004 4:48 PM
> To: libcurl development
> Subject: Re: NTLM, HTTP 100 Continue, and IIS 6 / .NET 1.1
>
> Excellent news -- working with MS-Support (which actually is
> incredible) -- we have found the source of the problem, as well as the
> solution (which I have tested with a hand-coded telnet session).
>
> Summary: Libcurl does not properly implement handling of POST requests
> with Content-Length: X and Expect: 100-Continue headers, which leads
to
> the sending of unexpected and invalid data to IIS 6.
>
> Details: The HTTP parser for IIS 6 was heavily refactored to improve
> performance by implementing HTTP 100 support as we discussed
> previously:
>
> >> This is obviously inefficient if your POST body is large, so it
seems
> >> that
> >> in newer versions of libcurl this has been optimized, and libcurl
> >> instead
> >> DOES NOT send the POST body, opting instead to send an "Expect:
> >> 100-continue" header, and only passing on the POST body when the
> >> server
> >> responds with "HTTP 100 Continue".
> >
> > Correct. This seems to be the best way to deal with this. It also
has
> > the
> > added benefit that if the server doesn't require any authentication
> > libcurl
> > won't attempt to perform any.
>
> IIS 6 is now optimized, according to the rules of the HTTP RFC:
>
> From RFC 2616:
> - Upon receiving a request which includes an Expect request-header
> field with the "100-continue" expectation, an origin server MUST
> either respond with 100 (Continue) status and continue to read
> from the input stream, or respond with a final status code. The
> origin server MUST NOT wait for the request body before sending
> the 100 (Continue) response. If it responds with a final status
> *******
> code, it MAY close the transport connection or it MAY continue
> *******
> to read and discard the rest of the request. It MUST NOT
> *******
> perform the requested method if it returns a final status code.
>
> - A client MUST NOT send an Expect request-header field (section
> *******
> 14.20) with the "100-continue" expectation if it does not intend
> *******
> to send a request body.
> *******
>
> The relevant sections have **** at the end....
>
> So, looking back at our trace that errored out:
>
> > --------- libcurl 7.11.1 talking to IIS 6.0/.NET 1.1
---------------
> > 010.000.001.101.50729-065.161.004.200.00080: POST
> > /mediabinwebservice/MediaBinServer.asmx HTTP/1.1
> > Host: mediabin.interwoven.com
> > Pragma: no-cache
> > Accept: */*
> > User-Agent:MediaBin Mac Native Client
> > Content-Type:text/xml; charset=utf-8
> > SOAPAction:"http://www.mediabin.com/GetMediaBinServerName"
> > Content-Length: 308
> > Expect: 100-continue
> >
> >
> > 065.161.004.200.00080-010.000.001.101.50729: HTTP/1.1 401
Unauthorized
> > Content-Length: 1656
> > Content-Type: text/html
> > Server: Microsoft-IIS/6.0
> > WWW-Authenticate: Negotiate
> > WWW-Authenticate: NTLM
> > WWW-Authenticate: Basic realm="mediabin.interwoven.com"
> > X-Powered-By: ASP.NET
> > Date: Wed, 24 Mar 2004 05:12:03 GMT
> >
> > <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
> > "http://www.w3.org/TR/html4/strict.dtd">
> > <HTML>
> > <!-- a very big HTML page -->
> > </HTML>
> >
>
> ...at this point in the request, IIS 6 has decided to respond with a
> FINAL status. Now, according to the RFC, it is still expecting 308
> bytes of POST request body... so the next 308 bytes it gets are "read
> and discarded" according to the RFC. And starting with the 309th byte,
> IIS 6 is expecting the NEXT request. Well that turns out to be in the
> middle of our HTTP headers of our next request, whose first 308 bytes
> were discarded by IIS. Thus, IIS expects POST or GET at this point in
> the stream but instead gets something right in the middle of the
> SOAPAction header. Aha!
>
> > 010.000.001.101.50729-065.161.004.200.00080: POST
> > /mediabinwebservice/MediaBinServer.asmx HTTP/1.1
> > Authorization: NTLM TlRMTVNTUAABAAAAAgIAAAAAAAAgAAAAAAAAACAAAAA=
> > Host: mediabin.interwoven.com
> > Pragma: no-cache
> > Accept: */*
> > User-Agent:MediaBin Mac Native Client
> > Content-Type:text/xml; charset=utf-8
> > SOAPAction:"http://www.mediabin.com/GetMediaBinServerName"
> > Content-Length: 308
> > Expect: 100-continue
> >
> >
> > 065.161.004.200.00080-010.000.001.101.50729: HTTP/1.1 400 Bad
Request
> > Content-Type: text/html
> > Date: Wed, 24 Mar 2004 05:12:03 GMT
> > Connection: close
> > Content-Length: 35
> >
> > <h1>Bad Request (Invalid Verb)</h1>
>
> And of course, yes, given all of this debugging we now know that IIS
> definitely saw an invalid verb.
>
> SOOO........ the fix?
>
> It's quite simple I think... there are two parts though, since we have
> to deal with "AnyAuth" probes...
>
> Let's deal with the simple case first...
> 1) --ntlm (no probe)
> > POST /mediabin/default.asp HTTP/1.1
> > Authorization: NTLM TlRMTVNTUAABAAAAAgIAAAAAAAAgAAAAAAAAACAAAAA=
> > Host: mediabin.interwoven.com
> > Pragma: no-cache
> > Accept: */*
> > User-Agent:MediaBin Mac Native Client
> > Content-Type:text/xml; charset=utf-8
> > Content-Length: 0
> > Expect: 100-continue
> >
>
> since we are sending a post request and NTLM is a 3-way handshake,
both
> client and server know that the POST with the NTLM type-1 request
> doesn't need the POST body. SO, we set the Content-Length to 0....
>
> > HTTP/1.1 401 Unauthorized
> > Content-Length: 1539
> > Content-Type: text/html
> > Server: Microsoft-IIS/6.0
> > WWW-Authenticate: NTLM
> > TlRMTVNTUAACAAAAAAAAADgAAAACAgAC9qFBg/
> > 0Kj08AAAAAAAAAAAAAAAA4AAAABQLODgAAAA8=
> > X-Powered-By: ASP.NET
> > Date: Fri, 26 Mar 2004 00:28:11 GMT
> >
> > <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
> > "http://www.w3.org/TR/html4/strict.dtd">
> > <HTML>
> > <!-- big HTML page -->
> > </HTML>
>
> The server then responds with a FINAL HTTP 401, with the NTLM
challenge
> header... according to the RFC, it must now "read and discard" the
rest
> of the request. Since Content-Length was 0, IIS knows that there's
> nothing left of that request and thus it's ready for the next request.
>
> Now, the client can send the next request, with the NTLM response in
> the HTTP headers. Since this is the last step in the handshake, we can
> go ahead and submit the entire POST body along with the correct
> Content-Length.
>
> > POST /mediabinwebservice/MediaBinServer.asmx HTTP/1.1
> > Authorization: NTLM
> >
TlRMTVNTUAADAAAAGAAYAE0AAAAAAAAAZQAAAAAAAABAAAAADQANAEAAAAAAAAAATQAAAAA
> > AAABlAAAAAYIAAGFkbWluaXN0cmF0b3KmBCTJa4n481uTDMKbdDBS2mmqUV3ybaQ=
> > Host: 10.0.1.108
> > Pragma: no-cache
> > Accept: */*
> > User-Agent:MediaBin Mac Native Client
> > Content-Type:text/xml; charset=utf-8
> > SOAPAction:"http://www.mediabin.com/GetMediaBinServerName"
> > Content-Length: 308
> >
> > <?xml version="1.0" encoding="utf-8"?>
> > <soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> > xmlns:xsd="http://www.w3.org/2001/XMLSchema"
> > xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
> > <soap:Body>
> > <GetMediaBinServerName xmlns="http://www.mediabin.com" />
> > </soap:Body>
> > </soap:Envelope>HTTP/1.1 401 Unauthorized
> >
> [ this would have worked (ie not 401) if I had sent the proper NTLM
> response, but I couldn't do it by hand. the thing to notice is that it
> responded as expected, and didn't give us "invalid verb" HTTP 400 ]
>
> 2) AnyAuth - this case is slightly more complicated as we have to deal
> with the probe. The required course of action will actually get
> slightly less efficient if there is NO authentication, but since that
> shouldn't happen too often, it isn't a big deal.
>
> Basically, it's the same as above... the initial PROBE should have a
> Content-Length of 0 and then see the server response...
>
> > POST /mediabin/default.asp HTTP/1.1
> > Host: mediabin.interwoven.com
> > Pragma: no-cache
> > Accept: */*
> > User-Agent:MediaBin Mac Native Client
> > Content-Type:text/xml; charset=utf-8
> > Content-Length: 0
> > Expect: 100-continue
> >
> > HTTP/1.1 200 OK
> >
>
> We send the probe and see the response... in fact I am wondering if
> instead we should probe with a HEAD request... yes that works!
>
> > HEAD /mediabin/default.asp HTTP/1.1
> > Host: mediabin.interwoven.com
> > Pragma: no-cache
> > Accept: */*
> > User-Agent:MediaBin Mac Native Client
> > Content-Type:text/xml; charset=utf-8
> > Content-Length: 0
> > Expect: 100-continue
> >
> > HTTP/1.1 200 OK
>
> ok... so we then inspect the response code. if it's a 200, then we
know
> there is no authentication and the we re-submit the ACTUAL request.
>
> if the response is a 401, then we start the challenge/response in the
> same manner as described above.
>
> --------
>
> Hmm... now that I think about it, why don't we do all "handshaking"
and
> such with HEAD requests entirely? Then we don't need to mess with
> content-length issues until we know we're sending the "final" request
> (the request that should give us the desired response)? That may fix
> this issue without having to "fake" the content length header and deal
> with deciding whether or not to send the POST body at different
> times... also should work more generally than with POST/GET etc...
>
> ---------
>
> Whew. That's all! Please everyone digest and comment. Once there's a
> consensus I can look at how to patch this, unless someone else wants
to
> go for it...
>
> Alan Pinstein

Received on 2004-03-26