Hi Daniel,

> Quite clearly your biggest "problem" in my view seems to be that urllib's
> and libcurl's (and thus pycurl's) interfaces are somewhat different and
> your existing code is baked around the nature of urllib and not really that
> libcurl/pycurl has any particular flaws.

I did not mean to insinuate that any of my interface problems were
caused by a particular flaw in libcurl.

Part of our re-write will decouple the packaging client's internal
algorithms from the logic used for content retrieval.

> Allow me to throw in my opinions on this - me being a libcurl guy
> primarily, and not really a pycurl person by the concepts should still
> apply no matter language:
>
> - Getting data line-by-line from a site is an inefficient way of downloading
> whatever you want. It's much more clever to get more data at once and then
> have a front-end that can traverse the downloaded data line-by-line.

I agree that this is suboptimal. We're planning on changing most of
these cases to use the easy and multi interfaces in pretty
straight-forward ways. It should give us a performance win in some
cases, and will simplify our code in other places.

Urllib doesn't support http pipelining, and connection re-use is tricky.
I'm looking forward to taking advantage of both of these features in
libcurl.

> - libcurl provides its multi interface as a non-blocking alternative that
> allows the caller, the application, to decide when to act and thus you can
> make the line-by-line function more or less exactly like how I understand
> your current code does it without additional threads. I actually once wrote
> up an fopen() style function (that was subsequently enhanced by others) that
> shows how it could be done (the example is in C using the native libcurl
> API):
>
> http://curl.haxx.se/lxr/source/docs/examples/fopen.c

Thanks for the pointer to the fopen example. I had been considering doing
something similar. Python provides StringIO/cStringIO in its standard
library. It's a slighly clumsier rendition of the buffered interface
from your example.

        http://www.python.org/doc/2.4.4/lib/module-StringIO.html

Thanks for answering my questions. Your response was very helpful.

-j
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
Received on 2008-12-04