cURL / Mailing Lists / curl-library / Single Mail

curl-library

RE: FW: HTTP GET using easycurl - HTTP chunks in HTTP response.

From: Rangoli Mathur <Rangoli.Mathur_at_wnco.com>
Date: Wed, 13 Dec 2006 18:24:48 -0600

> -----Original Message-----
> From: Rangoli Mathur
> Sent: Wednesday, December 13, 2006 6:17 PM
> To: Rangoli Mathur
> Subject: RE: FW: HTTP GET using easycurl - HTTP chunks in
> HTTP response.
>
> From: Dan Fandrich <dan_at_coneharvesters.com>
> Date: 2006-12-14
>
> On Wed, Dec 13, 2006 at 05:08:54PM -0600, Rangoli Mathur wrote:
> > I tried setting the HTTP version to 1.0 and I see all chunks being
> > coagulated.
>
>
> More likely, the server doesn't send chunked data in the 1.0
> case. See
> if there's a Transfer-Encoding: chunked header.
>

Yes, the Transfer Encoding is chunked header.

>
> > If I don't set the HTTP version, in which case it uses HTTP
> 1.1, I see
> > ([http-chunk-length] [http-chunk])* in the response.
> >
> > MOREOVER, for the 1st chunk after HTTP Header, I don't see the
> > http-chunk-length at all. I see http-chunk-length for
> subsequent chunks.
>
>
> That does look like a bug. Does the curl command-line tool do
> the same on
> the same URL? If so, can you post the output of --trace? It might be
> that the server isn't sending chunked data properly, and that
> would help to
> tell. Did you say which version of libcurl and which platform
> you're using?
>

I am using libcurl 7.16.0 on Solaris.
I tried the curl command line tool doesn't pose this problem. The
response from curl coagulated all the HTTP chunks, (I made sure the
transfer encoding is still chunked).
So, the issue, then lies in libcurl?

The command executed was:
curl -x proxy.swacorp.com:8080 -i -G -d "www.yahoo.com" --url
"www.yahoo.com" --trace trace.out >a.out

I am attaching trace.out and a.out.

>
> >>> Dan
> Received on Thu Dec 14 00:52:43 2006
>
>
> > -----Original Message-----
> > From: curl-library-bounces_at_cool.haxx.se
> > [mailto:curl-library-bounces_at_cool.haxx.se] On Behalf Of
> Rangoli Mathur
> > Sent: Wednesday, December 13, 2006 5:09 PM
> > To: libcurl development
> > Subject: RE: FW: HTTP GET using easycurl - HTTP chunks in
> > HTTP response.
> >
> > Here are details of what I was trying:
> >
> >
> > The URL I am using is http://www.yahoo.com Code snippet:
> > //////////////////////////////////////////
> >
> >
> > curl_global_init(CURL_GLOBAL_ALL );
> > curlHandle = curl_easy_init();
> > curl_easy_setopt(curlHandle, CURLOPT_NOPROGRESS, 1);
> > curl_easy_setopt(curlHandle, CURLOPT_WRITE_FUNCTION, write_data);
> > curl_east_setopt(curlHandle, CURLOPT_HEADER, 1);
> > curl_easy_setopt(curlHandle, CURLOPT_URL,
> "http://www.yahoo.com");
> > //Trying with http 1.0
> > curl_easy_setopt(curlHandle, CURLOPT_HTTP_VERSION,
> > CURL_HTTP_VERSION_1_0);
> > int i = curl_easy_perform(curlHandle);
> >
> > //////////////////////////////////////////////
> >
> >
> > I tried setting the HTTP version to 1.0 and I see all chunks being
> > coagulated.
> > If I don't set the HTTP version, in which case it uses HTTP
> > 1.1, I see
> > ([http-chunk-length] [http-chunk])* in the response.
> >
> > MOREOVER, for the 1st chunk after HTTP Header, I don't see the
> > http-chunk-length at all. I see http-chunk-length for
> > subsequent chunks.
> >
> > //Excerpts of the response I receive from libcurl are:
> > ////////////////////////////////////////////////////
> > HTTP/1.1 200 OK^M
> > Date: Wed, 13 Dec 2006 22:54:34 GMT^M
> > P3p: policyref="http://p3p.yahoo.com/w3c/p3p.xml", CP="CAO
> DSP COR CUR
> > ADM DEV TAI
> > PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi
> PUBi IND PHY
> > ONL UNI PUR
> > FIN COM NAV INT DEM CNT STA POL HEA PRE GOV"^M
> > Cache-Control: private^M
> > Vary: User-Agent^M
> > Content-Type: text/html; charset=utf-8^M
> > Via: 1.1 ics_server.swacorp.com (ICS 2.3.0.0.16)^M
> > Proxy-Connection: keep-alive^M
> > Transfer-Encoding: chunked^M
> > ^M
> > <html><head>
> > <title>Yahoo!</title>
> > <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
> > <meta http-equiv="PICS-Label" content='(PICS-1.1
> > "http://www.icra.org/ratingsv02.h
> > tml" l r (cz 1 lz 1 nz 1 oz 1 vz 1) gen true for
> > "http://www.yahoo.com"
> > r (cz 1 lz
> > 1 nz 1 oz 1 vz 1) "http://www.rsac.org/ratingsv01.html" l r
> > (n 0 s 0 v
> > 0 l 0) gen
> > true for "http://www.yahoo.com" r (n 0 s 0 v 0 l 0))'>
> > <base
> > href="http://www.yahoo.com/_ylh=X3oDMTFkOWw2OHFqBF9TAzI3MTYxND
> > kEcGlkAzEx
> > NjYw
> > NDk2NzAEdGVzdAMwBHRtcGwDaW5kZXgtdA--/" target="_top">
> > <style type="text/css">
> > a{color:#16387c;}
> > a:link,a:visited{text-decoration:none;}
> > a:hover{text-decoration:underline;}
> > </style>
> > <style type="text/css" media="all">
> > #p{width:310px;}
> > form{margin:0;}
> > </style>
> > </head>
> > <body link="#16387c" vlink="#16387c">
> > <center>
> > <table cellpadding="0" cellspacing="0" border="0" bgcolor="#EEF3F6"
> > width="760">
> > <tr><td colspan="3">
> > <table cellpadding="0" cellspacing="0" border="0" width="0"><tr><td
> > width="0" heig
> > ht="7"><spacer type="block" width="0" height="7"></td></tr></table>
> > </td></tr>
> > <tr>
> > <td width="10" height="^M
> > 5B4^M
> > 0" rowspan="2"><spacer type="block" width="10" height="0"></td>
> > <td height="56" valign="top">
> > <img src="http://us.a1.yimg.com/us.yimg.com/i/ww/beta/y3.gif"
> > width="232" height="
> > 44" alt="Yahoo!" title="Yahoo">
> > </td>
> > ......
> > .....
> > .....
> > 5B4^M
> > /fdbg.gif" nowrap="nowrap"><font face="arial" size="-1"><a
> > href="r/m1">My Mail</a>
> > </font></td></tr></table></td>
> > <td width="5" height="1" bgcolor="#dde8ea"
> > background="http://us.i1.yimg.com/us.yi
> > mg.com/i/ww/tbl/fdbg.gif"><spacer type="block" width="5"
> > height="1"></td>
> > <td width="1" height="100%" bgcolor="#586B7A" nowrap><spacer
> > type="block" width="1
> > " height="100%"></td>
> > </tr><tr>
> > <td width="100%" height="1" bgcolor="#586B7A" colspan="5"
> > nowrap><spacer
> > type="blo
> > ck" width="100%" height="1"></td>
> > </tr></table>
> > </td>
> > </tr></table>
> > </td>
> > </tr>
> > <tr><td colspan="3">
> > <table cellpadding="0" cellspacing="0" border="0" width="0"><tr><td
> > width="0" heig
> > ht="9"><spacer type="block" width="0" height="9"></td></tr></table>
> > </td></tr>
> > </table>
> > <table cellpadding="0" cellspacing="0" border="0" width="0"><tr><td
> > width="0" heig
> > ht="5"><spacer type="block" width="0" height="5"></td></tr></table>
> >
> > <table cellpadding="0" cellspacing="0" border="0" width="760">
> > <tr>
> > <td align="right">
> > <table cellpadding="0" cellspacing="0" border="0">
> > <tr>
> > <td><font face="arial" size="-1"><a href="r/l6"><b>Sign
> > In</b></a></font></td>
> > <td width="5" height="0"><spacer type="block" width="5"
> > height="0"></td>
> > <td bgcolor="#888888" width="2"><spacer type="block" width="2"
> > height="0"></td>
> > <td width="5" height="0"><spacer type="block" width="5"
> > height="0"></td>
> > <td><font face="arial" size="-1"><a href="r/m7"><b>Sign
> > Up</b></a></font></td>
> > <td width="10" height="0"><spacer type="block"^M
> > 498^M
> > ........
> > .........
> > ........
> > AE^M
> > y</a>.
> > </font></td></tr></table>
> > </td></tr></table>
> > </center>
> > </body>
> > </html>
> > 
> > ^M
> > 0^M
> > ^M
> > y</a>.
> > </font></td></tr></table>
> > </td></tr></table>
> > </center>
> > </body>
> > </html>
> > 
> > ^M
> > 0^M
> > ^M
> >
> >
> > > -----Original Message-----
> > > From: curl-library-bounces_at_cool.haxx.se
> > > [mailto:curl-library-bounces_at_cool.haxx.se] On Behalf Of
> Dan Fandrich
> > > Sent: Wednesday, December 13, 2006 4:30 PM
> > > To: libcurl development
> > > Subject: Re: FW: HTTP GET using easycurl - HTTP chunks in
> > > HTTP response.
> > >
> > > On Wed, Dec 13, 2006 at 02:47:50PM -0600, Rangoli Mathur wrote:
> > > > If I have defined a function: write_data and I have used
> > > > Curl_easy_setopt(curlHandle, CURLOPT_WRITE_FUNCTION,
> write_data).
> > > >
> > > > For HTTP1.1 chunked transfer encoding, Does the
> > write_data function
> > > > get called once for every http chunk there is?
> > >
> > > Not necessarily. You shouldn't rely on that behaviour, even
> > > it it appears to be the case.
> > >
> > > > If so, for the body, can I safely assume that every write
> > > would begin
> > > > with the number of bytes to read in hex, followed by those
> > > many bytes?
> > >
> > > libcurl should be taking care of un-chunking the data for
> > > you. If not, then there is likely a problem with the web
> > > server, or, less likely, libcurl itself. Send us a debug log
> > > of the transaction where you're seeing this if you'd like
> > > more details.
> > >
> > > >>> Dan
> > > --
> > > http://www.MoveAnnouncer.com The web change of
> > > address service
> > > Let webmasters know that your web site has moved
> > >
> >
> >

application/octet-stream attachment: trace.out

application/octet-stream attachment: a.out

Received on 2006-12-14

This message: [ Message body ]
Next message: Dan Fandrich: "Re: FW: HTTP GET using easycurl - HTTP chunks in HTTP response."
Previous message: Dan Fandrich: "Re: FW: HTTP GET using easycurl - HTTP chunks in HTTP response."
Maybe in reply to: Rangoli Mathur: "FW: HTTP GET using easycurl - HTTP chunks in HTTP response."
Next in thread: Dan Fandrich: "Re: FW: HTTP GET using easycurl - HTTP chunks in HTTP response."
Reply: Dan Fandrich: "Re: FW: HTTP GET using easycurl - HTTP chunks in HTTP response."
Reply: Daniel Stenberg: "RE: FW: HTTP GET using easycurl - HTTP chunks in HTTP response."

Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]