Hello;

I am new to the Curl module.

What I have been looking for is a way to semi duplicate what WGet does.

In case you do not know WGet basically can copy a website to your computer.
But I use it strictly for downloading files.

WGet also has the ability to obtain files of whatever type you wish to get.
And it parses web pages looking for links to other pages and/or the requested
file to download.

It parses by first downloading the file. Then parsing and then removing the file
if it is not the requested type.

Simply put I would like to see if I could use Curl to immitate only some of the
WGet functionality.

What I have is a list of URL's. I want to parse them for requested file types.
While parsing I would also like to recursively parse any other URL links found.

I would like to set a level of depth, so that the parsing would not go on forever
(links containing links, containing links, etc.....)

I was thinking that I would like to obtain all the page URL's first. That way I could
build up a list doing the recursive thing until the level was reached or no more links.

Then I would parse those for the file URL's. Once I had all of the file URL's then I
would just ftp them. Maybe Curl has a tool for this too?

I also want to put measures so that I do not reparse the same link over and over.
(link of subpage that goes to home page, link of hompage goes to subpage).

I dont want to use WGet because I want to handle the output in a slightly different way.

Any comments and/or links to example code would be apreciated.

RJ
Received on 2004-08-17