Re: Using pycurl with streaming python interfaces?

From: <>
Date: Thu, 4 Dec 2008 14:25:14 -0800

Hi Daniel,

> Quite clearly your biggest "problem" in my view seems to be that urllib's
> and libcurl's (and thus pycurl's) interfaces are somewhat different and
> your existing code is baked around the nature of urllib and not really that
> libcurl/pycurl has any particular flaws.

I did not mean to insinuate that any of my interface problems were
caused by a particular flaw in libcurl.

Part of our re-write will decouple the packaging client's internal
algorithms from the logic used for content retrieval.

> Allow me to throw in my opinions on this - me being a libcurl guy
> primarily, and not really a pycurl person by the concepts should still
> apply no matter language:
> - Getting data line-by-line from a site is an inefficient way of downloading
> whatever you want. It's much more clever to get more data at once and then
> have a front-end that can traverse the downloaded data line-by-line.

I agree that this is suboptimal. We're planning on changing most of
these cases to use the easy and multi interfaces in pretty
straight-forward ways. It should give us a performance win in some
cases, and will simplify our code in other places.

Urllib doesn't support http pipelining, and connection re-use is tricky.
I'm looking forward to taking advantage of both of these features in

> - libcurl provides its multi interface as a non-blocking alternative that
> allows the caller, the application, to decide when to act and thus you can
> make the line-by-line function more or less exactly like how I understand
> your current code does it without additional threads. I actually once wrote
> up an fopen() style function (that was subsequently enhanced by others) that
> shows how it could be done (the example is in C using the native libcurl
> API):

Thanks for the pointer to the fopen example. I had been considering doing
something similar. Python provides StringIO/cStringIO in its standard
library. It's a slighly clumsier rendition of the buffered interface
from your example.

Thanks for answering my questions. Your response was very helpful.

Received on 2008-12-04