curl-library
Re: NTLM, HTTP 100 Continue, and IIS 6 / .NET 1.1
Date: Fri, 26 Mar 2004 11:24:32 +0100 (CET)
On Thu, 25 Mar 2004 apinstein_at_mac.com wrote:
> Excellent news -- working with MS-Support (which actually is incredible) --
> we have found the source of the problem, as well as the solution (which I
> have tested with a hand-coded telnet session).
Very good work!
> Summary: Libcurl does not properly implement handling of POST requests with
> Content-Length: X and Expect: 100-Continue headers, which leads to the
> sending of unexpected and invalid data to IIS 6.
Let's at least say that IIS and libcurl have different opinions about what
"proper" handling of this means.
> So, looking back at our trace that errored out:
>
> > 010.000.001.101.50729-065.161.004.200.00080: POST
> > /mediabinwebservice/MediaBinServer.asmx HTTP/1.1
> > Content-Length: 308
> > Expect: 100-continue
> > 065.161.004.200.00080-010.000.001.101.50729: HTTP/1.1 401 Unauthorized
> > Server: Microsoft-IIS/6.0
> > WWW-Authenticate: NTLM
> > Date: Wed, 24 Mar 2004 05:12:03 GMT
> >
> > <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
>
> ...at this point in the request, IIS 6 has decided to respond with a FINAL
> status. Now, according to the RFC, it is still expecting 308 bytes of POST
> request body...
What specific section in the RFC are reading when you say that the client is
expected to continue to the POST even though no 100 continue arrived? My
interpretation of the 100-continue is that we require the 100 code reponse
before we send anything, and if we don't get it we don't send anything.
You quoted this section:
- A client MUST NOT send an Expect request-header field (section
14.20) with the "100-continue" expectation if it does not intend
to send a request body.
And I think that paragraph only confirms my view of things. The client *does*
intend to send the body, it just needs clearance first.
> so the next 308 bytes it gets are "read and discarded" according to the RFC.
This is painfully true.
Argh, I don't see the point in having this behavior defined like this! It
makes no sense at all at it makes operation such as this slower.
But do note that the RFC states "it MAY continue to read and discard the rest
of the request". MAY being the keyword. It doesn't _have to_ and I claim it is
stupid to do so. Especially since it also says "It MUST NOT perform the
requested method if it returns a final status code."
But of course, having "MAY" in the spec like that leads to what we see here
and I bet there will be server that reads that MAY the other way...
> And starting with the 309th byte, IIS 6 is expecting the NEXT request. Well
> that turns out to be in the middle of our HTTP headers of our next request,
> whose first 308 bytes were discarded by IIS. Thus, IIS expects POST or GET
> at this point in the stream but instead gets something right in the middle
> of the SOAPAction header. Aha!
This makes it perfectly clear why the server thinks we are being stupid.
> Let's deal with the simple case first...
> 1) --ntlm (no probe)
> > POST /mediabin/default.asp HTTP/1.1
> > Authorization: NTLM TlRMTVNTUAABAAAAAgIAAAAAAAAgAAAAAAAAACAAAAA=
>
> since we are sending a post request and NTLM is a 3-way handshake, both
> client and server know that the POST with the NTLM type-1 request doesn't
> need the POST body. SO, we set the Content-Length to 0....
Right, this would also work for Digest. But we must not forget that we did get
into problems when we tried to POST entirely without Content-Length: (which
was my initial way of doing this), which make me suspect that there might be
serves out there that will dislike getting "Content-Length: 0" POSTs too. This
is of course only a suspicion of mine, I have not seen nor heard of this.
We also need to provide the exact same logic for PUT.
> 2) AnyAuth - this case is slightly more complicated as we have to deal with
> the probe. The required course of action will actually get slightly less
> efficient if there is NO authentication, but since that shouldn't happen too
> often, it isn't a big deal.
>
> Basically, it's the same as above... the initial PROBE should have a
> Content-Length of 0 and then see the server response...
>
> > POST /mediabin/default.asp HTTP/1.1
> > Host: mediabin.interwoven.com
> > Pragma: no-cache
> > Accept: */*
> > User-Agent:MediaBin Mac Native Client
> > Content-Type:text/xml; charset=utf-8
> > Content-Length: 0
> > Expect: 100-continue
> >
> > HTTP/1.1 200 OK
> >
>
> We send the probe and see the response... in fact I am wondering if instead
> we should probe with a HEAD request... yes that works!
Yes, I believe the "anyauth" case must be converted to use HEAD in order to
avoid doing duplicate POSTs (if there is no auth required) - which indeed
could mess things up.
> ok... so we then inspect the response code. if it's a 200, then we know
> there is no authentication and the we re-submit the ACTUAL request.
>
> if the response is a 401, then we start the challenge/response in the
> same manner as described above.
Yeps!
> Hmm... now that I think about it, why don't we do all "handshaking" and such
> with HEAD requests entirely? Then we don't need to mess with content-length
> issues until we know we're sending the "final" request (the request that
> should give us the desired response)? That may fix this issue without having
> to "fake" the content length header and deal with deciding whether or not to
> send the POST body at different times... also should work more generally
> than with POST/GET etc...
I'm not sure it is plausible to use HEAD for the negotiation. Digest for
example uses the request keyword itself in the hash.
I suspect that some servers will require different authentication for POST
than for HEAD on a specific URL. For "anyauth" I don't think we can avoid
HEAD, but for a specific auth I think we should at least try to make a
solution with the "real" request keyword.
> Whew. That's all! Please everyone digest and comment. Once there's a
> consensus I can look at how to patch this, unless someone else wants to go
> for it...
This was indeed quite a bite to digest. Very good research and report!
I think I need to let my brain bounce around all this info a little while
before I can comment anything further.
I am of course interested in getting a patch that makes this work for you.
--
Daniel Stenberg -- http://curl.haxx.se -- http://daniel.haxx.se
Dedicated custom curl help for hire: http://haxx.se/curl.html
Received on 2004-03-26