Hi,

I'm a novice of libcurl. Now I'm trying to utilize it to develop an https client. Basically, just to automate the procedure of login to an
https website, and then downloading a file from it. I did a small test at first and figured out there could be a hidden bug within libcurl, because

it doesn't seem to handle https redirect properly. (At first I tought it could be within curlpp wrapper, but then I figured out it shouldn't,

as curlpp is only a C++ wrapper of libcurl. The redirect stuff is handled by libcurl. Please correct me if I am wrong.)

Here is what I did.

I downloaded the curlpp package (a C++ wrapper of libcurl, version0.3.1), compiled it and installed it w/o any problem.

My box has RedHat Linux 2.4.21-4 EL installed. It's i686 machine.

Here is the code:

==={ Source Code # 1}======================

#include <stdio.h>

#include <stdlib.h>

#include <string>

#include <string.h>

#include <fstream>

#include "curl.hpp"
#include "easy.hpp"
#include "types.hpp"

#define HTTPS_URL https://username:password@my.xxxxx.net/XXX/XXX/XXX.jsp

int main()
{
   CURL *curl;
   CURLcode res;
   char *useragent = "";

   char *cookiejar = "cookies.jar";

   curl = curl_easy_init();

   if(curl) {

     curl_easy_setopt(curl, CURLOPT_URL, HTTPS_URL);

     curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
     curl_easy_setopt(curl, CURLOPT_UNRESTRICTED_AUTH, 1);

     curl_easy_setopt(curl, CURLOPT_USERAGENT, useragent);
     curl_easy_setopt(curl, CURLOPT_COOKIEJAR, cookiejar);
     curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);

     res = curl_easy_perform(curl);
     curl_easy_cleanup(curl);
   }

   return 0;
}
======{ End of Souce code #1}=====================

And here is the output I obtained.

======{ Verbose/Trace Info. #1}==========================

* About to connect() to my.xxxxx.net:443

* Connected to my.xxxxx.net (xx.xxx.xxx.xx) port 443

* SSL connection using EDH-RSA-DES-CBC3-SHA

* Server certificate:

* subject: /C=US/ST=XXXXXXX/L=XXXXX/O=XXXXX/OU=eBusiness/OU=Terms of use at www.verisign.com/RPA (c)01/CN=my.XXXXX.net

* start date: 200X-XX-XX 00:00:00 GMT

* expire date: 200X-XX-XX 23:59:59 GMT

* common name: my.xxxxx.net (matched)

* issuer: /O=VeriSign Trust Network/OU=VeriSign, Inc./OU=VeriSign International Server CA - Class 3/OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign

> GET /login/log_log_page.jsp?CTAuthMode=BASIC&amp;ct_orig_uri=%2XXX%XXX.jsp HTTP/1.1

Authorization: Basic XXXXXXXXXXXXXXXXXXXX==

Host: my.xxxxx.net

Pragma: no-cache

Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*

HTTP/1.1 200 OK

Date: Thu, 07 Oct 2004 20:04:52 GMT

Server: HP Apache-based Web Server/1.3.26 (Unix) mod_ssl/2.8.9 OpenSSL/0.9.6c

Set-Cookie: JSESSIONID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; path=/

Transfer-Encoding: chunked

Content-Type: text/html

.. then followed by the ascii text format of login.jsp page, but not the page I'm trying to fetch :(

======={ End of Trace Info #1 }==========================

As I have RedHat linux installed, so there is already a bundled curl (7.10.6), which is under /usr/bin.

I tried using that curl to download the file:

$curl https://username:password@my.xxxxx.net/XXX/XXX/XXX.jsp

and it worked perfectly! (I even don't need to put -L option in the command line. I got the file I want.

Here is the version info for the command-line curl I have on my box.

========{ command-line curl version }========================

$ /usr/bin/curl -V

curl 7.10.6 (i386-redhat-linux-gnu) libcurl/7.10.6 OpenSSL/0.9.7a ipv6 zlib/1.1.4

Protocols: ftp gopher telnet dict ldap http file https ftps

Features: IPv6 SSL libz NTLM

========{ End of command-line curl version }===================

Again, here is the strace info I got for using command line curl:

========{ Strace Info #2: for command line curl }=============

* About to connect() to my.xxxxx.net:443

* Connected to my.xxxxx.net (xx.xxx.xxx.xx) port 443

* SSL connection using EDH-RSA-DES-CBC3-SHA

* Server certificate:

* subject: /C=US/ST=XXXXXXX/L=XXXXX/O=XXXXX/OU=XXXXX/OU=Terms of use a

t www.verisign.com/RPA (c)01/CN=my.xxxxx.net

* start date: 200X-XX-XX 00:00:00 GMT

* expire date: 200X-XX-XX 23:59:59 GMT

* common name: my.xxxxx.net (matched)

* issuer: /O=VeriSign Trust Network/OU=VeriSign, Inc./OU=VeriSign International Serve

r CA - Class 3/OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign

> GET /XXXX/XXXXX.jsp HTTP/1.1

Authorization: Basic XXXXXXXXXXXXXXXXXXX==

User-Agent: curl/7.10.6 (i386-redhat-linux-gnu) libcurl/7.10.6 OpenSSL/0.9.7a ipv6 zlib/1.1.4

Host: my.xxxxx.net

Pragma: no-cache

Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*

< HTTP/1.1 200 OK

< Date: Wed, 06 Oct 2004 00:31:43 GMT

< Server: HP Apache-based Web Server/1.3.26 (Unix) mod_ssl/2.8.9 OpenSSL/0.9.6c

< Set-Cookie: CTSESSION=XXXXXXXXXXXXXX; domain=.xxxxx.net; path=/

< Set-Cookie: BIGipServersslmbea_xxxxx_http=XXXXXXXXX.XXXXX.0000; path=/

< Set-Cookie: JSESSIONID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; path=/

< Cache-control: no-cache="set-cookie"

< Transfer-Encoding: chunked

< Content-Type: text/html

<!doctype html public "-//w3c//dtd html 4.0 transitional//en" "http://www.w3.org/TR/html4/lo

ose.dtd">

<HTML> // <= This is the file I want!

..

========{ End of Strace Info #2: for command line curl }=============

I also tried to turn on the cookies and force to set-up user-agent while I'm sending the request, but it didn't work, either.

I'm thinking that the problem could be within libcurl, because the curl seems to be able to handle https web rediction properly...

I understand that curl and libcurl are totally different things, curl is a command-line tool. While if curl can handle https redirect w/o problem, I assume that libcurl should be able to do something similar.

I suppose if curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1) works, then after the authentication (which already happened properly), it should automatically redirect to the https webpage I intended to fetch (but it didn't). i.e. the process should look like:

o Given --

https://username:password@my.xxxxx.net/MyTargetFile.jsp (note that

the target file is not login.jsp, which certainly has the same domain name (my.xxxxx.net/) as MyTargetFile.jsp.

o Authentication Succeeded -- I got RSA authentication approved (detail was listed in the above)

o Redirect to page MyTargetFile.jsp and download MyTargetFile (Unfortunately, this never happened while I tried using libcurl/curlpp in my code). What I got is the ascii format of login.jsp :-( That's the whole story.

Any suggestion would be greatly appreciated. Thanks in advance.

Iris
Received on 2004-10-11