Hi Todd,

I'm trying to build the same app (c++ crawler with
curl/curlpp) but I am struggling with the threaded
stuff.

Do you mind sharing the code ? I'll be happy to
provide any testing info or feedback you want

Regards,
Jose

 --- Todd Fisher <taf2_at_lehigh.edu> escribió: > Hi,
> I'm writting a multithread web crawler each
> thread does basically
> the following:
> 1. create an easycurl handle
> 2. setopts _NOSIGNAL(1), _NOPROGRES(1),
> _WRITEFUNCTION(myfunc),
> _FAILONERROR(1), _FOLLOWLOCATION(1),
> _MAXREDIRS(3),
> _USERAGENT("mycrawl+email", and _TIMEOUT(3)
> 3. enter bounded download loop
> 4. select a URL from my list of known urls
> 5. setopt _FILE(memlocation), _URL(url to download)
> 6. call curl_perform
> 7. parse content in memlocation to populate list of
> known urls
> 8. continue looping until bound max
> 9. out side of bounded loop cleanup curl handle
>
>
> everything works great except when i set the
> _TIMEOUT option in step 2.
> When i do that
> I get a core dump, that always points to domain name
> resolver somewhere
> in the curl library.
>
> However, if I don't set the timeout the process will
> not crash but my
> worker threads will hang
> in the curl perform function for upwards of 200
> seconds.
>
> Any ideas? This behavior usually starts to show up
> after 500+ downloads...
>
>
> -todd
>
>
>
>
-------------------------------------------------------
> This SF.net email is sponsored by: SlickEdit Inc.
> Develop an edge.
> The most comprehensive and flexible code editor you
> can use.
> Code faster. C/C++, C#, Java, HTML, XML, many more.
> FREE 30-Day Trial.
> www.slickedit.com/sourceforge
>

___________________________________________________
Yahoo! Móviles
Personaliza tu móvil con tu logo y melodía favorito
en http://moviles.yahoo.es

-------------------------------------------------------
This SF.net email is sponsored by: SlickEdit Inc. Develop an edge.
The most comprehensive and flexible code editor you can use.
Code faster. C/C++, C#, Java, HTML, XML, many more. FREE 30-Day Trial.
www.slickedit.com/sourceforge
Received on 2003-02-23