doh!

daniel, this looks like a good problem to solve for the next release :)

cheers
domenico

----- Forwarded message from Eric Cooper <ecc_at_cmu.edu> -----

Date: Thu, 26 May 2005 21:10:36 -0400
From: Eric Cooper <ecc_at_cmu.edu>
Subject: Bug#310948: curl fails when HTTP response headers contain null bytes

Package: curl
Version: 7.13.2-2
Severity: important

Some broken HTTP servers return response headers containing null bytes.
For example:
    http://ftp.pl.debian.org/debian/dists/sarge/main/binary-i386/Packages.gz
(at least until the webmaster fixes it).

When curl is parsing header lines, it reads a buffer and then looks
for the end-of-line by doing
            k->end_ptr = strchr (k->str_start, '\n');
in lib/transfer.c

But this is incorrect if the buffer contains embedded nulls -- the
strchr search will terminate too soon. Then more data gets appended
to the header, the true CRLF termination is missed, and curl returns
insanely long headers which are actually part of the response data.

Here is a simple test program to reproduce the problem:

---- cut here ----
#include <stdio.h>
#include <stdlib.h>
#include <curl/curl.h>

void
get_headers(char *url)
{
    CURL *handle;
    const int TRUE = 1;

    handle = curl_easy_init();

    // write headers to stdout
    curl_easy_setopt(handle, CURLOPT_HEADERFUNCTION, fwrite);
    curl_easy_setopt(handle, CURLOPT_WRITEHEADER, stdout);

    // exit when the data arrives
    curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, exit);

#if DEBUG
    curl_easy_setopt(handle, CURLOPT_VERBOSE, TRUE);
#endif

    curl_easy_setopt(handle, CURLOPT_URL, url);
    curl_easy_setopt(handle, CURLOPT_HTTPGET, TRUE);
    curl_easy_perform(handle);
}

int
main(int argc, char *argv[])
{
    if (argc != 2) {
        fprintf(stderr, "Usage: %s URL\n", argv[0]);
        exit(1);
    }
    get_headers(argv[1]);
    return 0;
}
---- cut here ----

Run
    ./ctest http://ftp.pl.debian.org/debian/dists/sarge/main/binary-i386/Packages.gz > foo"
and examine foo with hexdump or emacs to see the problem.

Here is a patch that seems to fix it, but I've only tested it with the
site above.

---- patch ----
diff -ur curl-7.13.2/lib/transfer.c curl-7.13.2-ecc/lib/transfer.c
--- curl-7.13.2/lib/transfer.c 2005-02-16 09:31:33.000000000 -0500
+++ curl-7.13.2-ecc/lib/transfer.c 2005-05-26 20:48:40.000000000 -0400
@@ -351,11 +351,20 @@
             size_t rest_length;
             size_t full_length;
             int writetype;
+ int i;
 
             /* str_start is start of line within buf */
             k->str_start = k->str;
 
- k->end_ptr = strchr (k->str_start, '\n');
+ /* we can't use strchr or index to find the end of line because
+ broken HTTP servers might put null bytes in the headers */
+ k->end_ptr = 0;
+ for (i = 0; i < nread; i++) {
+ if (k->str_start[i] == '\n') {
+ k->end_ptr = k->str_start + i;
+ break;
+ }
+ }
 
             if (!k->end_ptr) {
               /* Not a complete header line within buffer, append the data to
---- end patch ----

----- End forwarded message -----

-----[ Domenico Andreoli, aka cavok
 --[ http://people.debian.org/~cavok/gpgkey.asc
   ---[ 3A0F 2F80 F79C 678A 8936 4FEE 0677 9033 A20E BC50
Received on 2005-05-27