I've written on this list before regarding the use of a debug version of WinInet.dll for the purposes of tracing requests and responses when browsing "manually" using Internet Explorer. This technique is one of the easiest to use when trying to peek behind the scenes of https:// communication, since it can't easily be recorded by a network sniffer (such as Ethereal).

I'm sending this note to provide a quick update on that technique, and to mention a new one I just became familiar with.

The new tool I just learned about is called ieHTTPHeaders, and can be downloaded from http://www.blunck.info/iehttpheaders.html. It installs itself as a new Explorer Bar within internet explorer, and will display for you all HTTP requests and response headers (but NOT response BODIES) that occur while that explorer bar is open. This allows you to see exactly which URLs are requested while browsing a site you're hoping to automate with curl. This type of log file is extremely valuable in knowing what sequence of URLs you'll need for your curl script.

NOTE: if you try the installer and nothing happens, try uninstalling, then rebooting, then installing again right away...

Here's an update on using the debug-enabled WinInet.dll:

The initial set of instructions I had provided for using the debug version of WinInet.dll involved replacing that file (in the SYSTEM folder) with a special debug-enabled version. The problem is that Windows File Protection makes this difficult to do under Win2K and WinXP. To get around that problem:

 a] Place the debug enabled version of WinInet.dll into the
    same folder as internet explorer
    (usually c:\Program Files\Internet Explorer).

 b] Create an empty (0-byte) file named "iexplore.exe.local",
    and place that into the same folder.

 c] Using a command prompt, set the environment variable
    WININETLOG, like this:
> SET WININETLOG=1

 d] Launch iexplore.exe from the same command prompt, like this:
> IEXPLORE.EXE "http://your.url.here/"

 e] Look for the file named WININET.LOG to be created on your desktop.

Here's a small perl script which may prove useful in parsing through one of these logfiles. To use it, save it into a file (named e.g. parse.pl). Then, at the command line, "pipe" your wininet.log file through it, like this:

> TYPE wininet.log | perl.exe parse.pl

  local $b = 0; # "break" - set anytime we encounter hex lines,
    # cleared at the end of a section
  local $l = 0; # line number of the input logfile
  local $s = ""; # concatenated decoded string representing
    # one section of hex
  local $sl = ""; # previous decoded string (so we can just
    # print one copy of any SSL-encrypted string)

  while (<>) {
    $l++;
    # find hex lines: 8-char address, following by 16 hex chars, then the ascii equivalent
    # previous regex: /(?: [\w ]{2}){8}[- ](?:[\w ]{2} ){8}..(.*)$/
    if ( / ([0-9a-f]{8}) ([0-9a-f \-]{47}) (.{1,16})$/ ) {
      # skip over some common hex lines that we don't care about
      # 8 .'s is usually the winsock version (1000.1000)
      # 4 .'s is usually the length of the TCP frame

      if (!($3 eq "...." or $3 eq "........")) {
        for ( split /[- ]/, $2 ) { $s .= chr(hex($_)) unless $_ eq "07" }
        $b = 1;
      }

    } elsif ($b) {
      print "$s\n\n" unless ($sl eq $s);
      $sl = $s; $s = ""; $b = 0; $i = 1;
    }
  }

  exit;

-- Kevin

-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
Received on 2003-10-16