Bugs item #1703444, was opened at 2007-04-19 10:19
Message generated for change (Comment added) made by barklund
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1703444&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: documentation
Group: None
Status: Closed
Resolution: Fixed
Priority: 3
Private: No
Submitted By: Morten Barklund (barklund)
Assigned to: Daniel Stenberg (bagder)
Summary: -L documentation does not reflect actual behaviour

Initial Comment:
When curl'ing a URL with a --data content and the URL returns a location header along with a 3xx response code, the new URL is curl'ed without the --data but with all other headers intact.

This follows the specification of the rfc, but it is not reflected in the man-page, which states:

       -L/--location
              (HTTP/HTTPS) If the server reports that
              the requested page has moved to a different
              location (indicated with a Location: header
              and a 3XX response code) this option
              will make curl redo the request on the
              new place. If used together with
              -i/--include or -I/--head, headers from all
              requested pages will be shown. When
              authentication is used, curl only sends
              its credentials to the initial host. If a
              redirect takes curl to a different host, it
              won't be able to intercept the
              user+password. See also --location-trusted
              on how to change this. You can limit the
              amount of redirects to follow by using the
              --max-redirs option.

It should somehow be reflected, that if first request is POST, second (and consecutive) request(s) will be GET-requests without the body of the first.

Example is a simple PHP-page like:

 <?php
 header("Location: ".$_SERVER['REQUEST_URI']);
 header("X-method: ".$_SERVER['REQUEST_METHOD']);
 header("X-post-body: ".file_get_contents("php://input"));
 ?>

When invoked like:

  curl -i --data 'test=boing' --max-redirs 1 -L http://host/path/method_test.php

Response is correctly:

 HTTP/1.1 302 Found
 Location: /path/method_test.php
 X-method: POST
 X-post-body: test=boing
 Content-Length: 0
 Content-Type: text/html; charset=ISO-8859-1

 HTTP/1.1 302 Found
 Location: /path/method_test.php
 X-method: GET
 X-post-body:
 Content-Length: 0
 Content-Type: text/html; charset=ISO-8859-1

 curl: (47) Maximum (1) redirects followed

(irrelevant headers stripped)

Thanks for a great tool anyways :)

-- 
Morten Barklund
----------------------------------------------------------------------
>Comment By: Morten Barklund (barklund)
Date: 2007-04-23 10:14
Message:
Logged In: YES 
user_id=989204
Originator: YES
Sure, no problem. Thanks for the fast feedback anyways :)
----------------------------------------------------------------------
Comment By: Daniel Stenberg (bagder)
Date: 2007-04-23 10:08
Message:
Logged In: YES 
user_id=1110
Originator: NO
I decided that was far too much details for it to be what most people want
to see in the curl man page. I added a description telling which response
codes that change the method, and mentioned that the others don't.
In-depth details such as _why_ they act the way they do are better put
elsewhere as most users won't (need to) care about that.
An "--rfc-compliant" kind of flag has been discussed before but so far
nobody ever bothered to implement one. But that's also besides the scope of
this bug report, which by the way already is closed! ;-)
If you want to follow-up on anything of this, I suggest you subscribe to
the curl-users mailing list and bring the discussion there instead!
----------------------------------------------------------------------
Comment By: Morten Barklund (barklund)
Date: 2007-04-23 10:01
Message:
Logged In: YES 
user_id=989204
Originator: YES
To include your comment on breaking the RFC, one could include it as:
 If initial request is POST, the redirected request will
 be sent as GET (without request body) if response code
 is
 - 301 (by convention, directly violating specification),
 - 302 (by convention, somewhat violating specification) or
 - 303 (by specification).
 305 and 307 is redirected without changing request method.
 Violating the RFC is done to mimick most other user
 agents (including all larger browser) and because it as a
 sideeffect of this is the expected behaviour of some
 serverside scripts. RFC violations can be turned off by
 explicitly setting the --rfc-compliant flag
And thus introduce such a flag (there are probably many more cases in
which the rfc is violated to follow conventions).
-- 
Morten Barklund
----------------------------------------------------------------------
Comment By: Daniel Stenberg (bagder)
Date: 2007-04-19 12:15
Message:
Logged In: YES 
user_id=1110
Originator: NO
Yes, that seems to be the correct description. Thanks!
I find that to be a very large chunk of info that is hard to add to the
-L/--location paragraph nicely.
The 301 and 302 actions could also be mentioned to violate the spec
(rfc2616) because curl does what browsers do in order to 1) be able to
mimic their operations fully and 2) because lots of web side
servers/scripts assume that behavior.
But again, I'm not a very good manual writer so I'm not really sure how to
add all this very detailed in-depth info to that user-oriented manual.
----------------------------------------------------------------------
Comment By: Morten Barklund (barklund)
Date: 2007-04-19 11:13
Message:
Logged In: YES 
user_id=989204
Originator: YES
Fair enough, you (and your kind) has studied this way more than me :)
Documentation could then be:
 If initial request is POST, the redirected request will
 be sent as GET (without request body) if response code
 is
 - 301 (by convention, directly violating specification),
 - 302 (by convention, somewhat violating specification) or
 - 303 (by specification).
 305 and 307 is redirected without changing request method.
Would that summarize things properly? :)
-- 
Morten Barklund
----------------------------------------------------------------------
Comment By: Daniel Stenberg (bagder)
Date: 2007-04-19 11:07
Message:
Logged In: YES 
user_id=1110
Originator: NO
Allow me to quote the libcurl source code again, regarding the treatment
of 301
     * Warning: Because most of importants user agents do this obvious
RFC2616   
     * violation, many webservers expect this misbehavior. So these
servers      
     * often answers to a POST request with an error page.  To be sure
that      
     * libcurl gets the page that most user agents would get, libcurl has
to     
     * force GET:                                                         
      
So yes, libcurl does this.
----------------------------------------------------------------------
Comment By: Morten Barklund (barklund)
Date: 2007-04-19 11:03
Message:
Logged In: YES 
user_id=989204
Originator: YES
I see that when changing response code, different results happen (as you
describe). As far is I can tell, 301, 302 and 303 converts POST to GET, 304
does not support a location header and 305 and 307 does not convert POST to
GET.
But the rfc says (10.3.2):
      Note: When automatically redirecting a POST request after
      receiving a 301 status code, some existing HTTP/1.0 user agents
      will erroneously change it into a GET request.
That actually makes curl an "erroneous HTTP/1.0 user agent" :)
Example:
 curl -i --data 'test=boing' --max-redirs 1 -L
http://host/path/method_test.php
 HTTP/1.1 301 Flaf
 Location: /path/method_test.php
 X-method: POST
 X-post-body: test=boing
 Content-Length: 0
 Content-Type: text/html; charset=ISO-8859-1
 HTTP/1.1 301 Flaf
 Location: /path/method_test.php
 X-method: GET
 X-post-body:
 Content-Length: 0
 Content-Type: text/html; charset=ISO-8859-1
 curl: (47) Maximum (1) redirects followed
301 SHOULD NOT convert POST to GET given the RFC 2616 section 10.3.2 as
far as I can tell.
Documentation for this would be something along the lines of:
 If initial request is POST, the redirected request will be sent as GET
(without request body)
 if response code is 302 (by convention) or 303 (by specification).
Thanks :)
-- 
Morten Barklund
----------------------------------------------------------------------
Comment By: Daniel Stenberg (bagder)
Date: 2007-04-19 10:26
Message:
Logged In: YES 
user_id=1110
Originator: NO
(lib)curl does not unconditionally switch from POST to GET. It depends
entirely on the response code. In your case you got a 302 back, and I feel
it appropriate to quote this (somewhat large) comment below from the source
code regarding the behavior you see. The quote originates from RFC2616. But
anyway, if you can think if a way to explain this in the -L documentation
section, please help me out...
    /* (From 10.3.3)                                                      
      
                                                                          
      
    Note: RFC 1945 and RFC 2068 specify that the client is not allowed    
      
    to change the method on the redirected request.  However, most        
      
    existing user agent implementations treat 302 as if it were a 303     
      
    response, performing a GET on the Location field-value regardless     
      
    of the original request method. The status codes 303 and 307 have     
      
    been added for servers that wish to make unambiguously clear which    
      
    kind of reaction is expected of the client.                           
      
                                                                          
      
    (From 10.3.4)                                                         
      
                                                                          
      
    Note: Many pre-HTTP/1.1 user agents do not understand the 303         
      
    status. When interoperability with such clients is a concern, the     
      
    302 status code may be used instead, since most user agents react     
      
    to a 302 response as described here for 303.                          
      
  
----------------------------------------------------------------------
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1703444&group_id=976