> NOTE: you mailed this to the curl-main mailing list and to the admin address
> of the libcurl list. There's no point in mailing the admin address with
> normal list posts.

Sorry about that.

> [ getting lots of web pages explained ]

[...]

> Well, we can't actually know anything since we don't know anything about your
> script, your bandwidth, the site you're talking about and its bandwidth.

The bandwith from my end is a medium one. I mean, it's not a very good
one, but not a bad one either. I know there is a limit for 1GB per month,
but the traffic in a month, is at least a few times more than 1 GB. I
don't know the exact bandwith right now (before I ask our sysadmin), but
the real bandwith is around 64K/sec I think.

> Many site owners considers it a bad thing to "suck" every web page from
> his/hers site, especially if done with a very high speed (no pause between
> the fetches).

Well, my script extracts information from every page it gets from that
server (www.blackvoices.com). And the time between get requests is at
least 1 sec. I will add "sleep(60)" between HTTP requests though and see
if this works. But I had the same problem (downloading a page, in a
loop) with the other server I wanted to crawl (www.findmymate.com). I will
try your sugestion and see what happens.

> Also, if the remote site has a limited bandwidth while you have
> much better, there's indeed the possibility that you starve out other
> visitors from the site while you're downloading the (entire) site.
>
> If the server was messed up by your script, then it surely must be a silly
> server, but then there are no laws against that and many a server out there
> are truly stupid.
>
> You might get more popular if you make sure your script doesn't work quite as
> fast as it can. Most search engines, for example, only make one request a
> second during harvesting.
>
>

-- 
Nick...
_______________________________________________________________________
"It's nice to be important, but it's more important to be nice !"
						SCOOTER
http://terra.vl.ro/nick/ <-- My personal web site (info about me&my life)
http://terra.vl.ro/nick/resume.html <-- My resume/CV.
_______________________________________________________________________