Arthur Rodrigues wrote:
>
> I'm running a multi-threaded webCrawler using Curl, but eventually,
after
> downloading about 600 sites
> it starts return error 6 or error 23 even for existent sites. I'm
working
> on Gentoo 2.16.2.
>
> For those sites returning error, I easy access them via browser.
>
> Do you have any ideas of what might be happening?
>

Two words immediately come to mind: memory leak.

Which version of curl are you using?
Any HTTPS connections?
Can you reproduce the problem if you hit the same site instead of
crawling over 600 different ones?

-Gary
Received on 2008-04-23