Re: POST data goes into second packet.
Date: Tue, 13 May 2003 02:36:56 -0500
You just have to grit your teeth and grind through all the iterations. Oh, and if you get a "frames" page, do a trial fetch on *all* the frame contents pages - I found one server that sent a title bar with cookies and expected the main window to use those cookies...
This has worked with secure servers as well as non-secure servers, without needing a network scanner, though there have been times where a scanner such as ettercap might have been useful.
Kaye-Smith Adam wrote:
> As you have suggested, POSTable forms can be very complex in that they may pass cookies or go via redirects etc. - I have no problem with following this path if it is all unencrypted - but with this case I am working on , all communications are done in SSL & therefore I can see none of the contents below the TCP layer. This makes it very difficult to track its progress. My strategy is to caputure the form & modify the POST action to go to an http url rather than an https - this allows me to see the contents of the initial post but ofcoarse the server does not respond and I get no further exmaple of the communications that would have occured if I had used SSL.
> Are there are other techniques for working out what is in the SSL packet ? Could I use my own certificate & therefore somehow decrypt the packet ?
> -----Original Message-----
> From: Ralph Mitchell [mailto:rmitchell_at_eds.com]
> Sent: Tuesday, 13 May 2003 3:31 PM
> To: curl-users_at_lists.sourceforge.net
> Subject: Re: POST data goes into second packet.
> Daniel Stenberg wrote:
> > On Mon, 12 May 2003, Kaye-Smith Adam wrote:
> > > I am trying to replicate what occurs when I logon on to a website through a
> > > browser.
> > >
> > > With curl I am using the -d to post the variables and -H to changes headers
> > > values & this works as expected but when I monitor what packets are
> > > exchanged (via the ethereal network analyser) when the browser provides its
> > > posted values, the browser puts its variables in a packet that is seperate
> > > to the initial post packet. ie the Host:, User-agent:, Accept: headers are
> > > in the first http packet & the next http packet has just the postdata & the
> > > headers Type: & Size. I am not sure if this critical to why my curl based
> > > form entry is not working but how do I get curl to emulate the browser by
> > > defining exactly what goes in what packets in the initial POST sequemce (
> > > which includes a second packet as stated above).
> > I doubt it very much, this is the reason for your script's failure.
> > You can't emulate the browser's behavior to that level with curl. The only
> > valid reason I can think of, would be if the remote server is utterly stupid
> > imlemented and can't be fixed.
> > One way to try it out, would be to setup a proxy in between and let curl
> > operate through that one, as it'll change how the packets are sent out from
> > the proxy to the remote server.
> The browser may also be doing several other things *before* you get the login
> page. Did you try starting ethereal before pointing your browser to the page
> you're trying to fetch?
> sometimes embedded hidden text fields) to track the login, and it's usually
> necessary to follow the whole cookie trail. I've had pages that have gone
> through several redirects (back to themselves...) setting a different cookie
> each time.
> This has happened to me often enough that I have a standard set of options that
> I use for every script:
> CURLOPTS="-s -S -L -b cookies -c cookies -m 90 --connect-time 45"
> and then I add in output filename, proxy info where necessary, on the actual
> command line:
> curl -o xxxx.html $CURLOPTS http://host.domain.com
> Ralph Mitchell
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
Received on 2003-05-13