thanks.. I did try changing the order of the user/pass headers but that
didn't help. I'll try the entire patch and we'll see.

alan

On Mar 25, 2004, at 8:25 PM, Xiuping Hu wrote:

> Good finding. This is the same result as I posted month ago, I have the
> patch that is passed test on IIS 5.0/.NET and IIS 5.0/OWA2k.
>
> Attached is the patch, it is patched against release 7.11.1.
>
> Index: http.c
> ===================================================================
> RCS file: /usr/aventail/prodroot/appliance/vendor/curl/src/lib/http.c,v
> retrieving revision 1.1.1.7
> diff -u -w -r1.1.1.7 http.c
> --- http.c 2004/03/22 19:37:38 1.1.1.7
> +++ http.c 2004/03/26 01:17:06
> @@ -18,7 +18,7 @@
> * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY
> OF ANY
> * KIND, either express or implied.
> *
> - * $Id: http.c,v 1.1.1.7 2004/03/22 19:37:38 dyouatt Exp $
> + * $Id: http.c,v 1.1.1.3.24.3 2004/03/24 23:32:51 xhu Exp $
>
> ***********************************************************************
> *
> ***/
>
> #include "setup.h"
> @@ -93,6 +93,7 @@
> #include "url.h"
> #include "share.h"
> #include "http.h"
>
> #define _MPRINTF_REPLACE /* use our functions only */
> #include <curl/mprintf.h>
> @@ -1340,14 +1341,14 @@
>
> if(!req_buffer)
> return CURLE_OUT_OF_MEMORY;
> -
> +/* NEW CHANGE: move password to the bottom of the list. */
> /* add the main request stuff */
> result =
> add_bufferf(req_buffer,
> "%s " /* GET/HEAD/POST/PUT */
> "%s HTTP/%s\r\n" /* path + HTTP version */
> - "%s" /* proxyuserpwd */
> - "%s" /* userpwd */
> "%s" /* range */
> "%s" /* user agent */
> "%s" /* cookie */
> @@ -1356,14 +1357,13 @@
> "%s" /* accept */
> "%s" /* accept-encoding */
> "%s" /* referer */
> - "%s",/* transfer-encoding */
> + "%s"/* transfer-encoding */
> + "%s" /* proxyuserpwd */
> + "%s",/* userpwd */
>
> request,
> ppath,
> httpstring,
> - (conn->bits.httpproxy && conn->allocptr.proxyuserpwd)?
> - conn->allocptr.proxyuserpwd:"",
> - conn->allocptr.userpwd?conn->allocptr.userpwd:"",
> (conn->bits.use_range && conn->allocptr.rangeline)?
> conn->allocptr.rangeline:"",
> (data->set.useragent && *data->set.useragent &&
> conn->allocptr.uagent)?
> @@ -1375,7 +1375,10 @@
> (data->set.encoding && *data->set.encoding &&
> conn->allocptr.accept_encoding)?
> conn->allocptr.accept_encoding:"", /* 08/28/02 jhrg */
> (data->change.referer &&
> conn->allocptr.ref)?conn->allocptr.ref:"" /* Referer: <data> <CRLF> */,
> - te
> + te,
> + (conn->bits.httpproxy && conn->allocptr.proxyuserpwd)?
> + conn->allocptr.proxyuserpwd:"",
> + conn->allocptr.userpwd?conn->allocptr.userpwd:""
> );
>
> if(result)
> @@ -1543,15 +1546,29 @@
> }
> break;
>
> + /* NEW CHANGE: Load Content-Length and 100 Continue after NTLM
> authenticated. */
> case HTTPREQ_PUT: /* Let's PUT the data to the server! */
> -
> - if((data->set.infilesize>0) && !conn->bits.upload_chunky)
> + {
> + bool header_load = FALSE;
> + if((data->state.authwant == CURLAUTH_NONE) ||
> + (data->state.authwant == CURLAUTH_BASIC) ||
> + (strncmp(request,"POST",4)))
> + {
> + header_load = TRUE;
> + }
> + else if(authdone)
> + {
> + header_load = TRUE;
> + }
> + if( (header_load) && (data->set.infilesize>0) &&
> !conn->bits.upload_chunky)
> /* only add Content-Length if not uploading chunked */
> add_bufferf(req_buffer,
> "Content-Length: %" FORMAT_OFF_T "\r\n", /* size
> */
> data->set.infilesize );
>
> - if(!checkheaders(data, "Expect:")) {
> + if(authdone && !checkheaders(data, "Expect:")) {
> /* if not disabled explicitly we add a Expect: 100-continue
> to the headers which actually speeds up post operations (as
> there is one packet coming back from the web server) */
> @@ -1579,7 +1596,7 @@
> if(result)
> return result;
> break;
> -
> + }
> case HTTPREQ_POST:
> /* this is the simple POST, using x-www-form-urlencoded style */
>
> Hope this can help.
>
> Thanks,
>
> Xiuping
>
>
>
>
>
>> -----Original Message-----
>> From: curl-library-bounces_at_cool.haxx.se [mailto:curl-library-
>> bounces_at_cool.haxx.se]
>> Sent: Thursday, March 25, 2004 4:48 PM
>> To: libcurl development
>> Subject: Re: NTLM, HTTP 100 Continue, and IIS 6 / .NET 1.1
>>
>> Excellent news -- working with MS-Support (which actually is
>> incredible) -- we have found the source of the problem, as well as the
>> solution (which I have tested with a hand-coded telnet session).
>>
>> Summary: Libcurl does not properly implement handling of POST requests
>> with Content-Length: X and Expect: 100-Continue headers, which leads
> to
>> the sending of unexpected and invalid data to IIS 6.
>>
>> Details: The HTTP parser for IIS 6 was heavily refactored to improve
>> performance by implementing HTTP 100 support as we discussed
>> previously:
>>
>>>> This is obviously inefficient if your POST body is large, so it
> seems
>>>> that
>>>> in newer versions of libcurl this has been optimized, and libcurl
>>>> instead
>>>> DOES NOT send the POST body, opting instead to send an "Expect:
>>>> 100-continue" header, and only passing on the POST body when the
>>>> server
>>>> responds with "HTTP 100 Continue".
>>>
>>> Correct. This seems to be the best way to deal with this. It also
> has
>>> the
>>> added benefit that if the server doesn't require any authentication
>>> libcurl
>>> won't attempt to perform any.
>>
>> IIS 6 is now optimized, according to the rules of the HTTP RFC:
>>
>> From RFC 2616:
>> - Upon receiving a request which includes an Expect request-header
>> field with the "100-continue" expectation, an origin server MUST
>> either respond with 100 (Continue) status and continue to read
>> from the input stream, or respond with a final status code. The
>> origin server MUST NOT wait for the request body before sending
>> the 100 (Continue) response. If it responds with a final status
>> *******
>> code, it MAY close the transport connection or it MAY continue
>> *******
>> to read and discard the rest of the request. It MUST NOT
>> *******
>> perform the requested method if it returns a final status code.
>>
>> - A client MUST NOT send an Expect request-header field (section
>> *******
>> 14.20) with the "100-continue" expectation if it does not intend
>> *******
>> to send a request body.
>> *******
>>
>> The relevant sections have **** at the end....
>>
>> So, looking back at our trace that errored out:
>>
>>> --------- libcurl 7.11.1 talking to IIS 6.0/.NET 1.1
> ---------------
>>> 010.000.001.101.50729-065.161.004.200.00080: POST
>>> /mediabinwebservice/MediaBinServer.asmx HTTP/1.1
>>> Host: mediabin.interwoven.com
>>> Pragma: no-cache
>>> Accept: */*
>>> User-Agent:MediaBin Mac Native Client
>>> Content-Type:text/xml; charset=utf-8
>>> SOAPAction:"http://www.mediabin.com/GetMediaBinServerName"
>>> Content-Length: 308
>>> Expect: 100-continue
>>>
>>>
>>> 065.161.004.200.00080-010.000.001.101.50729: HTTP/1.1 401
> Unauthorized
>>> Content-Length: 1656
>>> Content-Type: text/html
>>> Server: Microsoft-IIS/6.0
>>> WWW-Authenticate: Negotiate
>>> WWW-Authenticate: NTLM
>>> WWW-Authenticate: Basic realm="mediabin.interwoven.com"
>>> X-Powered-By: ASP.NET
>>> Date: Wed, 24 Mar 2004 05:12:03 GMT
>>>
>>> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
>>> "http://www.w3.org/TR/html4/strict.dtd">
>>> <HTML>
>>> <!-- a very big HTML page -->
>>> </HTML>
>>>
>>
>> ...at this point in the request, IIS 6 has decided to respond with a
>> FINAL status. Now, according to the RFC, it is still expecting 308
>> bytes of POST request body... so the next 308 bytes it gets are "read
>> and discarded" according to the RFC. And starting with the 309th byte,
>> IIS 6 is expecting the NEXT request. Well that turns out to be in the
>> middle of our HTTP headers of our next request, whose first 308 bytes
>> were discarded by IIS. Thus, IIS expects POST or GET at this point in
>> the stream but instead gets something right in the middle of the
>> SOAPAction header. Aha!
>>
>>> 010.000.001.101.50729-065.161.004.200.00080: POST
>>> /mediabinwebservice/MediaBinServer.asmx HTTP/1.1
>>> Authorization: NTLM TlRMTVNTUAABAAAAAgIAAAAAAAAgAAAAAAAAACAAAAA=
>>> Host: mediabin.interwoven.com
>>> Pragma: no-cache
>>> Accept: */*
>>> User-Agent:MediaBin Mac Native Client
>>> Content-Type:text/xml; charset=utf-8
>>> SOAPAction:"http://www.mediabin.com/GetMediaBinServerName"
>>> Content-Length: 308
>>> Expect: 100-continue
>>>
>>>
>>> 065.161.004.200.00080-010.000.001.101.50729: HTTP/1.1 400 Bad
> Request
>>> Content-Type: text/html
>>> Date: Wed, 24 Mar 2004 05:12:03 GMT
>>> Connection: close
>>> Content-Length: 35
>>>
>>> <h1>Bad Request (Invalid Verb)</h1>
>>
>> And of course, yes, given all of this debugging we now know that IIS
>> definitely saw an invalid verb.
>>
>> SOOO........ the fix?
>>
>> It's quite simple I think... there are two parts though, since we have
>> to deal with "AnyAuth" probes...
>>
>> Let's deal with the simple case first...
>> 1) --ntlm (no probe)
>>> POST /mediabin/default.asp HTTP/1.1
>>> Authorization: NTLM TlRMTVNTUAABAAAAAgIAAAAAAAAgAAAAAAAAACAAAAA=
>>> Host: mediabin.interwoven.com
>>> Pragma: no-cache
>>> Accept: */*
>>> User-Agent:MediaBin Mac Native Client
>>> Content-Type:text/xml; charset=utf-8
>>> Content-Length: 0
>>> Expect: 100-continue
>>>
>>
>> since we are sending a post request and NTLM is a 3-way handshake,
> both
>> client and server know that the POST with the NTLM type-1 request
>> doesn't need the POST body. SO, we set the Content-Length to 0....
>>
>>> HTTP/1.1 401 Unauthorized
>>> Content-Length: 1539
>>> Content-Type: text/html
>>> Server: Microsoft-IIS/6.0
>>> WWW-Authenticate: NTLM
>>> TlRMTVNTUAACAAAAAAAAADgAAAACAgAC9qFBg/
>>> 0Kj08AAAAAAAAAAAAAAAA4AAAABQLODgAAAA8=
>>> X-Powered-By: ASP.NET
>>> Date: Fri, 26 Mar 2004 00:28:11 GMT
>>>
>>> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
>>> "http://www.w3.org/TR/html4/strict.dtd">
>>> <HTML>
>>> <!-- big HTML page -->
>>> </HTML>
>>
>> The server then responds with a FINAL HTTP 401, with the NTLM
> challenge
>> header... according to the RFC, it must now "read and discard" the
> rest
>> of the request. Since Content-Length was 0, IIS knows that there's
>> nothing left of that request and thus it's ready for the next request.
>>
>> Now, the client can send the next request, with the NTLM response in
>> the HTTP headers. Since this is the last step in the handshake, we can
>> go ahead and submit the entire POST body along with the correct
>> Content-Length.
>>
>>> POST /mediabinwebservice/MediaBinServer.asmx HTTP/1.1
>>> Authorization: NTLM
>>>
> TlRMTVNTUAADAAAAGAAYAE0AAAAAAAAAZQAAAAAAAABAAAAADQANAEAAAAAAAAAATQAAAAA
>>> AAABlAAAAAYIAAGFkbWluaXN0cmF0b3KmBCTJa4n481uTDMKbdDBS2mmqUV3ybaQ=
>>> Host: 10.0.1.108
>>> Pragma: no-cache
>>> Accept: */*
>>> User-Agent:MediaBin Mac Native Client
>>> Content-Type:text/xml; charset=utf-8
>>> SOAPAction:"http://www.mediabin.com/GetMediaBinServerName"
>>> Content-Length: 308
>>>
>>> <?xml version="1.0" encoding="utf-8"?>
>>> <soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xmlns:xsd="http://www.w3.org/2001/XMLSchema"
>>> xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
>>> <soap:Body>
>>> <GetMediaBinServerName xmlns="http://www.mediabin.com" />
>>> </soap:Body>
>>> </soap:Envelope>HTTP/1.1 401 Unauthorized
>>>
>> [ this would have worked (ie not 401) if I had sent the proper NTLM
>> response, but I couldn't do it by hand. the thing to notice is that it
>> responded as expected, and didn't give us "invalid verb" HTTP 400 ]
>>
>> 2) AnyAuth - this case is slightly more complicated as we have to deal
>> with the probe. The required course of action will actually get
>> slightly less efficient if there is NO authentication, but since that
>> shouldn't happen too often, it isn't a big deal.
>>
>> Basically, it's the same as above... the initial PROBE should have a
>> Content-Length of 0 and then see the server response...
>>
>>> POST /mediabin/default.asp HTTP/1.1
>>> Host: mediabin.interwoven.com
>>> Pragma: no-cache
>>> Accept: */*
>>> User-Agent:MediaBin Mac Native Client
>>> Content-Type:text/xml; charset=utf-8
>>> Content-Length: 0
>>> Expect: 100-continue
>>>
>>> HTTP/1.1 200 OK
>>>
>>
>> We send the probe and see the response... in fact I am wondering if
>> instead we should probe with a HEAD request... yes that works!
>>
>>> HEAD /mediabin/default.asp HTTP/1.1
>>> Host: mediabin.interwoven.com
>>> Pragma: no-cache
>>> Accept: */*
>>> User-Agent:MediaBin Mac Native Client
>>> Content-Type:text/xml; charset=utf-8
>>> Content-Length: 0
>>> Expect: 100-continue
>>>
>>> HTTP/1.1 200 OK
>>
>> ok... so we then inspect the response code. if it's a 200, then we
> know
>> there is no authentication and the we re-submit the ACTUAL request.
>>
>> if the response is a 401, then we start the challenge/response in the
>> same manner as described above.
>>
>> --------
>>
>> Hmm... now that I think about it, why don't we do all "handshaking"
> and
>> such with HEAD requests entirely? Then we don't need to mess with
>> content-length issues until we know we're sending the "final" request
>> (the request that should give us the desired response)? That may fix
>> this issue without having to "fake" the content length header and deal
>> with deciding whether or not to send the POST body at different
>> times... also should work more generally than with POST/GET etc...
>>
>> ---------
>>
>> Whew. That's all! Please everyone digest and comment. Once there's a
>> consensus I can look at how to patch this, unless someone else wants
> to
>> go for it...
>>
>> Alan Pinstein
>
> <http.c.diff>
Received on 2004-03-26