curl-library
Re: NTLM, HTTP 100 Continue, and IIS 6 / .NET 1.1
Date: Thu, 25 Mar 2004 19:48:18 -0500
Excellent news -- working with MS-Support (which actually is
incredible) -- we have found the source of the problem, as well as the
solution (which I have tested with a hand-coded telnet session).
Summary: Libcurl does not properly implement handling of POST requests
with Content-Length: X and Expect: 100-Continue headers, which leads to
the sending of unexpected and invalid data to IIS 6.
Details: The HTTP parser for IIS 6 was heavily refactored to improve
performance by implementing HTTP 100 support as we discussed
previously:
>> This is obviously inefficient if your POST body is large, so it seems
>> that
>> in newer versions of libcurl this has been optimized, and libcurl
>> instead
>> DOES NOT send the POST body, opting instead to send an "Expect:
>> 100-continue" header, and only passing on the POST body when the
>> server
>> responds with "HTTP 100 Continue".
>
> Correct. This seems to be the best way to deal with this. It also has
> the
> added benefit that if the server doesn't require any authentication
> libcurl
> won't attempt to perform any.
IIS 6 is now optimized, according to the rules of the HTTP RFC:
From RFC 2616:
- Upon receiving a request which includes an Expect request-header
field with the "100-continue" expectation, an origin server MUST
either respond with 100 (Continue) status and continue to read
from the input stream, or respond with a final status code. The
origin server MUST NOT wait for the request body before sending
the 100 (Continue) response. If it responds with a final status
*******
code, it MAY close the transport connection or it MAY continue
*******
to read and discard the rest of the request. It MUST NOT
*******
perform the requested method if it returns a final status code.
- A client MUST NOT send an Expect request-header field (section
*******
14.20) with the "100-continue" expectation if it does not intend
*******
to send a request body.
*******
The relevant sections have **** at the end....
So, looking back at our trace that errored out:
> --------- libcurl 7.11.1 talking to IIS 6.0/.NET 1.1 ---------------
> 010.000.001.101.50729-065.161.004.200.00080: POST
> /mediabinwebservice/MediaBinServer.asmx HTTP/1.1
> Host: mediabin.interwoven.com
> Pragma: no-cache
> Accept: */*
> User-Agent:MediaBin Mac Native Client
> Content-Type:text/xml; charset=utf-8
> SOAPAction:"http://www.mediabin.com/GetMediaBinServerName"
> Content-Length: 308
> Expect: 100-continue
>
>
> 065.161.004.200.00080-010.000.001.101.50729: HTTP/1.1 401 Unauthorized
> Content-Length: 1656
> Content-Type: text/html
> Server: Microsoft-IIS/6.0
> WWW-Authenticate: Negotiate
> WWW-Authenticate: NTLM
> WWW-Authenticate: Basic realm="mediabin.interwoven.com"
> X-Powered-By: ASP.NET
> Date: Wed, 24 Mar 2004 05:12:03 GMT
>
> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
> "http://www.w3.org/TR/html4/strict.dtd">
> <HTML>
> <!-- a very big HTML page -->
> </HTML>
>
...at this point in the request, IIS 6 has decided to respond with a
FINAL status. Now, according to the RFC, it is still expecting 308
bytes of POST request body... so the next 308 bytes it gets are "read
and discarded" according to the RFC. And starting with the 309th byte,
IIS 6 is expecting the NEXT request. Well that turns out to be in the
middle of our HTTP headers of our next request, whose first 308 bytes
were discarded by IIS. Thus, IIS expects POST or GET at this point in
the stream but instead gets something right in the middle of the
SOAPAction header. Aha!
> 010.000.001.101.50729-065.161.004.200.00080: POST
> /mediabinwebservice/MediaBinServer.asmx HTTP/1.1
> Authorization: NTLM TlRMTVNTUAABAAAAAgIAAAAAAAAgAAAAAAAAACAAAAA=
> Host: mediabin.interwoven.com
> Pragma: no-cache
> Accept: */*
> User-Agent:MediaBin Mac Native Client
> Content-Type:text/xml; charset=utf-8
> SOAPAction:"http://www.mediabin.com/GetMediaBinServerName"
> Content-Length: 308
> Expect: 100-continue
>
>
> 065.161.004.200.00080-010.000.001.101.50729: HTTP/1.1 400 Bad Request
> Content-Type: text/html
> Date: Wed, 24 Mar 2004 05:12:03 GMT
> Connection: close
> Content-Length: 35
>
> <h1>Bad Request (Invalid Verb)</h1>
And of course, yes, given all of this debugging we now know that IIS
definitely saw an invalid verb.
SOOO........ the fix?
It's quite simple I think... there are two parts though, since we have
to deal with "AnyAuth" probes...
Let's deal with the simple case first...
1) --ntlm (no probe)
> POST /mediabin/default.asp HTTP/1.1
> Authorization: NTLM TlRMTVNTUAABAAAAAgIAAAAAAAAgAAAAAAAAACAAAAA=
> Host: mediabin.interwoven.com
> Pragma: no-cache
> Accept: */*
> User-Agent:MediaBin Mac Native Client
> Content-Type:text/xml; charset=utf-8
> Content-Length: 0
> Expect: 100-continue
>
since we are sending a post request and NTLM is a 3-way handshake, both
client and server know that the POST with the NTLM type-1 request
doesn't need the POST body. SO, we set the Content-Length to 0....
> HTTP/1.1 401 Unauthorized
> Content-Length: 1539
> Content-Type: text/html
> Server: Microsoft-IIS/6.0
> WWW-Authenticate: NTLM
> TlRMTVNTUAACAAAAAAAAADgAAAACAgAC9qFBg/
> 0Kj08AAAAAAAAAAAAAAAA4AAAABQLODgAAAA8=
> X-Powered-By: ASP.NET
> Date: Fri, 26 Mar 2004 00:28:11 GMT
>
> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
> "http://www.w3.org/TR/html4/strict.dtd">
> <HTML>
> <!-- big HTML page -->
> </HTML>
The server then responds with a FINAL HTTP 401, with the NTLM challenge
header... according to the RFC, it must now "read and discard" the rest
of the request. Since Content-Length was 0, IIS knows that there's
nothing left of that request and thus it's ready for the next request.
Now, the client can send the next request, with the NTLM response in
the HTTP headers. Since this is the last step in the handshake, we can
go ahead and submit the entire POST body along with the correct
Content-Length.
> POST /mediabinwebservice/MediaBinServer.asmx HTTP/1.1
> Authorization: NTLM
> TlRMTVNTUAADAAAAGAAYAE0AAAAAAAAAZQAAAAAAAABAAAAADQANAEAAAAAAAAAATQAAAAA
> AAABlAAAAAYIAAGFkbWluaXN0cmF0b3KmBCTJa4n481uTDMKbdDBS2mmqUV3ybaQ=
> Host: 10.0.1.108
> Pragma: no-cache
> Accept: */*
> User-Agent:MediaBin Mac Native Client
> Content-Type:text/xml; charset=utf-8
> SOAPAction:"http://www.mediabin.com/GetMediaBinServerName"
> Content-Length: 308
>
> <?xml version="1.0" encoding="utf-8"?>
> <soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:xsd="http://www.w3.org/2001/XMLSchema"
> xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
> <soap:Body>
> <GetMediaBinServerName xmlns="http://www.mediabin.com" />
> </soap:Body>
> </soap:Envelope>HTTP/1.1 401 Unauthorized
>
[ this would have worked (ie not 401) if I had sent the proper NTLM
response, but I couldn't do it by hand. the thing to notice is that it
responded as expected, and didn't give us "invalid verb" HTTP 400 ]
2) AnyAuth - this case is slightly more complicated as we have to deal
with the probe. The required course of action will actually get
slightly less efficient if there is NO authentication, but since that
shouldn't happen too often, it isn't a big deal.
Basically, it's the same as above... the initial PROBE should have a
Content-Length of 0 and then see the server response...
> POST /mediabin/default.asp HTTP/1.1
> Host: mediabin.interwoven.com
> Pragma: no-cache
> Accept: */*
> User-Agent:MediaBin Mac Native Client
> Content-Type:text/xml; charset=utf-8
> Content-Length: 0
> Expect: 100-continue
>
> HTTP/1.1 200 OK
>
We send the probe and see the response... in fact I am wondering if
instead we should probe with a HEAD request... yes that works!
> HEAD /mediabin/default.asp HTTP/1.1
> Host: mediabin.interwoven.com
> Pragma: no-cache
> Accept: */*
> User-Agent:MediaBin Mac Native Client
> Content-Type:text/xml; charset=utf-8
> Content-Length: 0
> Expect: 100-continue
>
> HTTP/1.1 200 OK
ok... so we then inspect the response code. if it's a 200, then we know
there is no authentication and the we re-submit the ACTUAL request.
if the response is a 401, then we start the challenge/response in the
same manner as described above.
--------
Hmm... now that I think about it, why don't we do all "handshaking" and
such with HEAD requests entirely? Then we don't need to mess with
content-length issues until we know we're sending the "final" request
(the request that should give us the desired response)? That may fix
this issue without having to "fake" the content length header and deal
with deciding whether or not to send the POST body at different
times... also should work more generally than with POST/GET etc...
---------
Whew. That's all! Please everyone digest and comment. Once there's a
consensus I can look at how to patch this, unless someone else wants to
go for it...
Alan Pinstein
Received on 2004-03-26