cURL / Mailing Lists / curl-library / Single Mail


Re: Need to maintain statistics on a page as it is downloading and in real time

From: Daniel Stenberg <>
Date: Mon, 17 Dec 2001 09:56:26 +0100 (MET)

On Sat, 15 Dec 2001, Mike and Melissa Malloy wrote:

[description cut off]

> I am not sure how to pass this information from a callback(
> CURLOPT_WRITEFUNCTION ) to the thread that initiated the request.

The callback function you provide will get a "stream" pointer in the forth
argument. That pointer is what you set CURLOPT_FILE to be. libcurl will not
touch it.

You would then possibly set CURLOPT_FILE to point to a struct of your choice,
and in the callback function you typecast the void pointer to your struct
pointer and *tadaa* you'll have whatever data you want passed between your
application and the callback function.

> I am also unsure how to stop retrieving a file once the request is made.

You return an error from the callback function (return 0 for example). Or you
can set a progress callback and return a non-zero value from that one (you
could have the progress callback receive the same struct pointer as your
write callback gets). The progress callback approach is useful since that
will enable you to abort the function even when no data is being received, as
the write callback isn't called then.

> If I can get this to work, the final hurdle I see is really a C programming
> problem. If the data sent to the callback has only half of the data I am
> looking for, a string search on that data and the data in the next callback
> will may not match the string, but a concatenation of both data parcels
> will:
> Search for string "hello"
> request n data: xxxxxxxxxxxxxxxx....xxhel
> request n+1 data: loxxxxxxxxxxxxxx....xxxxx

You could also get them like this:

n : xxxxxhe
n+1: l
n+2: l
n+3: oxxxxxx

> My plan is to concatenate data blocks into a buffer until the string is
> found and then perform a memmove of the buffer, starting with character
> after the string is found to the beginning of the buffer. If anyone knows
> of a better approach or why this approach wouldn't work, I would very much
> like to know.

First, you only need to "concatenate" or consider the following (n+1) block
if the current (n) block ends with a "valid" part of your search string. If
the block ends with "xxxxhex" then you can just ignore this block.

If this is a plain string, you can just count the number of matched
characters in the block. There's no need to actually store the block data.


 n: Wow! There's 2 letters matching in the end, we store this fact

 n+1: ah, last block had matching letters, we continue from that index to
      see if we're still on a match. Ah, yes, we are. We're now on 3 matching

 n+2: we're on a matching string, with this match we're now on 4 matching

 n+3: we're on a matching string, we matched yet another letter and *wham*
      we have a full string match (it is enough to know that we matched X
      letters when the search string is X letters long).

Of course, if the code at any point would fail to match, the counter should
go back to zero and you should start over looking for the beginning of a
search string.

If the string is a static string, possible rather long, there might be an
idea to consider doing the search Boyer-Moore style as that is a rather
clever way to not check every byte. This is far beyond the scope of libcurl
though... :-)

    Daniel Stenberg -- curl groks URLs --
Received on 2001-12-17