On Sat, Sep 20, 2008 at 02:22:34PM +0400, Alexey A. Rybak wrote:
> curl --max-filesize 1024 anyhost

Works here! - for servers that actually send a Content-Length header.

> But from the "physical" point of view I can't understand why we just
> can't stop downloading when the size of HTTP response becomes greater
> than the limit? Is there any possibility to do this?

You can easily do this with CURLOPT_RANGE, i.e. sending a Content-Range
header with your request. Some servers may ignore that header, so you might
also have to count how many bytes your callback function has received so
far, and abort the download once you have received enough data. (The Range
header, if recognized, has the advantage that your crawler can re-use the
HTTP connection after the data was received.)

BTW, the CURLOPT_RANGE documentation is a bit inaccurate; AFAIK the "X-Y"
means that the byte at offset Y is included.

Cheers,

  Richard

-- 
  __   _
  |_) /|  Richard Atterer     |  GnuPG key: 888354F7
  | \/¯|  http://atterer.net  |  08A9 7B7D 3D13 3EF2 3D25  D157 79E6 F6DC 8883 54F7
  ¯ '` ¯