Re: PyCurl as a high-performance client

From: Daniel Stenberg <>
Date: Thu, 11 Dec 2008 23:32:29 +0100 (CET)

On Thu, 11 Dec 2008, wrote:

> With a slightly-modified set to 20 concurrent connections
> retrieving the same 100KB file, it generates slightly over 5 MEGABITS per
> second, and the CPU is maxed out.

That sounds terribly wrong.

> Is PyCurl the right tool for this job?

I think it is.

> Is PyCurl's concurrent sessions the right way to do this, or might it work
> better with multiple threads each having a single session?

I'm not sure exactly how the performance will differ between using the multi
interface with 20 transfers and 20 threads doing one transfer each, but I
can't imagine it being noticably difference with that low number of transfers.

> If you want more info, let me know, but the main idea of the program is too
> retrieve MANY files as fast as possible, over and over again.

curl-loader is a tool that does similar stuff, using the curl multi interface
(although not the python binding) and I know they've reached very high speed.

I think your problem lies either in your implementation/usage of pycurl, or
possibly a bug in pycurl's interface against libcurl.

Received on 2008-12-11