On Thu, Feb 04, 2010 at 12:46:19PM -0500, my name wrote:
> I've modified retriever-multi.py to constantly fetch URLs from a database
> and do some work on it. I'm able to push out roughly 5MB/s despite having
> over 80mbps at my disposal. Is there any way to get better performance out
> of this? I'm thinking I should implement cStringIO rather than writing to a
> file and re-reading it in.

Have you done any performance analysis on this code? Python2.6 comes
with a pretty decent C-based profiler. Using the profier might give you
some insight into where your program is spending its time. On a basic
level, though, it would help to figure out the general area where your
performance is poor. 80mbps is 10MB/s but unless your destination is on
the same uncongested network as the source, it may be hard to get the
max theoretical throughput. Is the program bound by the disk, CPU, or
network? If you can figure out the answer to that question, it will
give you a better idea of what part of your code to change.

-j

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
Received on 2010-02-05