All I can really suggest is that you start by pointing curl at the first server page, just as you would with a browser, and take it from there. Download that first page, following redirects, saving cookies, and perhaps displaying the headers too. Examine the page to make sure it isn't doing anything weird with javascript, extract the form variables and try posting to the form action page. Watch out for onSubmit in the form tag, that might do something with javascript, such as fiddle with the form variables.

The only time this approach has broken down for me is when the page used javascript to edit the form variables, then it constructed a new url from the edited values in the form and set "location.href=newurl" to magically transport itself to the new location.

You just have to grit your teeth and grind through all the iterations. Oh, and if you get a "frames" page, do a trial fetch on *all* the frame contents pages - I found one server that sent a title bar with cookies and expected the main window to use those cookies...

This has worked with secure servers as well as non-secure servers, without needing a network scanner, though there have been times where a scanner such as ettercap might have been useful.

Ralph

Kaye-Smith Adam wrote:

> As you have suggested, POSTable forms can be very complex in that they may pass cookies or go via redirects etc. - I have no problem with following this path if it is all unencrypted - but with this case I am working on , all communications are done in SSL & therefore I can see none of the contents below the TCP layer. This makes it very difficult to track its progress. My strategy is to caputure the form & modify the POST action to go to an http url rather than an https - this allows me to see the contents of the initial post but ofcoarse the server does not respond and I get no further exmaple of the communications that would have occured if I had used SSL.
>
> Are there are other techniques for working out what is in the SSL packet ? Could I use my own certificate & therefore somehow decrypt the packet ?
>
> AdamKS
>
> -----Original Message-----
> From: Ralph Mitchell [mailto:rmitchell_at_eds.com]
> Sent: Tuesday, 13 May 2003 3:31 PM
> To: curl-users_at_lists.sourceforge.net
> Subject: Re: POST data goes into second packet.
>
> Daniel Stenberg wrote:
>
> > On Mon, 12 May 2003, Kaye-Smith Adam wrote:
> >
> > > I am trying to replicate what occurs when I logon on to a website through a
> > > browser.
> > >
> > > With curl I am using the -d to post the variables and -H to changes headers
> > > values & this works as expected but when I monitor what packets are
> > > exchanged (via the ethereal network analyser) when the browser provides its
> > > posted values, the browser puts its variables in a packet that is seperate
> > > to the initial post packet. ie the Host:, User-agent:, Accept: headers are
> > > in the first http packet & the next http packet has just the postdata & the
> > > headers Type: & Size. I am not sure if this critical to why my curl based
> > > form entry is not working but how do I get curl to emulate the browser by
> > > defining exactly what goes in what packets in the initial POST sequemce (
> > > which includes a second packet as stated above).
> >
> > I doubt it very much, this is the reason for your script's failure.
> >
> > You can't emulate the browser's behavior to that level with curl. The only
> > valid reason I can think of, would be if the remote server is utterly stupid
> > imlemented and can't be fixed.
> >
> > One way to try it out, would be to setup a proxy in between and let curl
> > operate through that one, as it'll change how the packets are sent out from
> > the proxy to the remote server.
>
> The browser may also be doing several other things *before* you get the login
> page. Did you try starting ethereal before pointing your browser to the page
> you're trying to fetch?
>
> In my experience, pages that have POSTable login forms generally use cookies (or
> sometimes embedded hidden text fields) to track the login, and it's usually
> necessary to follow the whole cookie trail. I've had pages that have gone
> through several redirects (back to themselves...) setting a different cookie
> each time.
>
> This has happened to me often enough that I have a standard set of options that
> I use for every script:
>
> CURLOPTS="-s -S -L -b cookies -c cookies -m 90 --connect-time 45"
>
> and then I add in output filename, proxy info where necessary, on the actual
> command line:
>
> curl -o xxxx.html $CURLOPTS http://host.domain.com
>
> Ralph Mitchell

-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com
Received on 2003-05-13