I wrote a crawler using pycurl which can crawl 5M urls per day on Amazon EC2 (small). This crawler uses single handle in threaded mode and is well commented.
I want to keep this as open source and was wondering if this could be added into Pycurl Official Examples.
Moreover, what is max known speed achieved by a cURL crawler? I am trying to develop a crawler (cURL based) optimized to a level where bandwidth can be the only bottleneck/limit. 
Thanksd_r_a_g_o_s
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
Received on 2012-03-02