cURL / Mailing Lists / curl-users / Single Mail

curl-users

RE: POST data goes into second packet.

From: David Withnall <david_withnall_at_health.qld.gov.au>
Date: Wed, 14 May 2003 06:43:26 +1000

If you're using windows there is a program available called HTTPWatch. http://www.simtec.ltd.uk
It's a plugin for IE that allows you to see all traffic to/from the browser. Headers, Cookies, Post, Get etc.

I've found it handy, but it's not free.

Cheers
D.

------------------------------------------------------------------------------
David Withnall Ph: 07 3406 8079
Biomedical Engineer Mb: 0405 131 087
Asset Management and Technical Audit Program
Biomedical Technology Services
Queensland Health

>>> Adam.Kaye-Smith_at_nrm.qld.gov.au 13/05/03 4:30:51 pm >>>

As you have suggested, POSTable forms can be very complex in that they may pass cookies or go via redirects etc. - I have no problem with following this path if it is all unencrypted - but with this case I am working on , all communications are done in SSL & therefore I can see none of the contents below the TCP layer. This makes it very difficult to track its progress. My strategy is to caputure the form & modify the POST action to go to an http url rather than an https - this allows me to see the contents of the initial post but ofcoarse the server does not respond and I get no further exmaple of the communications that would have occured if I had used SSL.

Are there are other techniques for working out what is in the SSL packet ? Could I use my own certificate & therefore somehow decrypt the packet ?

AdamKS

-----Original Message-----
From: Ralph Mitchell [mailto:rmitchell_at_eds.com]
Sent: Tuesday, 13 May 2003 3:31 PM
To: curl-users_at_lists.sourceforge.net
Subject: Re: POST data goes into second packet.

Daniel Stenberg wrote:

> On Mon, 12 May 2003, Kaye-Smith Adam wrote:
>
> > I am trying to replicate what occurs when I logon on to a website through a
> > browser.
> >
> > With curl I am using the -d to post the variables and -H to changes headers
> > values & this works as expected but when I monitor what packets are
> > exchanged (via the ethereal network analyser) when the browser provides its
> > posted values, the browser puts its variables in a packet that is seperate
> > to the initial post packet. ie the Host:, User-agent:, Accept: headers are
> > in the first http packet & the next http packet has just the postdata & the
> > headers Type: & Size. I am not sure if this critical to why my curl based
> > form entry is not working but how do I get curl to emulate the browser by
> > defining exactly what goes in what packets in the initial POST sequemce (
> > which includes a second packet as stated above).
>
> I doubt it very much, this is the reason for your script's failure.
>
> You can't emulate the browser's behavior to that level with curl. The only
> valid reason I can think of, would be if the remote server is utterly stupid
> imlemented and can't be fixed.
>
> One way to try it out, would be to setup a proxy in between and let curl
> operate through that one, as it'll change how the packets are sent out from
> the proxy to the remote server.

The browser may also be doing several other things *before* you get the login
page. Did you try starting ethereal before pointing your browser to the page
you're trying to fetch?

In my experience, pages that have POSTable login forms generally use cookies (or
sometimes embedded hidden text fields) to track the login, and it's usually
necessary to follow the whole cookie trail. I've had pages that have gone
through several redirects (back to themselves...) setting a different cookie
each time.

This has happened to me often enough that I have a standard set of options that
I use for every script:

CURLOPTS="-s -S -L -b cookies -c cookies -m 90 --connect-time 45"

and then I add in output filename, proxy info where necessary, on the actual
command line:

curl -o xxxx.html $CURLOPTS http://host.domain.com

Ralph Mitchell

-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

************************************************************************
The information in this e-mail together with any attachments is
intended only for the person or entity to which it is addressed
and may contain confidential and/or privileged material.

Any form of review, disclosure, modification, distribution
and/or publication of this e-mail message is prohibited.

If you have received this message in error, you are asked to
inform the sender as quickly as possible and delete this message
and any copies of this message from your computer and/or your
computer system network.
************************************************************************

**********************************************************************
This e-mail, including any attachments sent with it, is confidential
and for the sole use of the intended recipient(s). This confidentiality
is not waived or lost if you receive it and you are not the intended
recipient(s), or if it is transmitted/ received in error.

Any unauthorised use, alteration, disclosure, distribution or review
of this e-mail is prohibited. It may be subject to a statutory duty of
confidentiality if it relates to health service matters.

If you are not the intended recipient(s), or if you have received this
e-mail in error, you are asked to immediately notify the sender by
telephone or by return e-mail. You should also delete this e-mail
message and destroy any hard copies produced.
**********************************************************************

This message: [ Message body ]
Next message: Perry, Kevin: "Trying not to use LD_LIBRARY_PATH"
Previous message: Ralph Mitchell: "Re: (OT sort of)SMS using PHP"
Maybe in reply to: Kaye-Smith Adam: "POST data goes into second packet."

Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]