Re: urlgrabber and pycurl

From: Seth Vidal <>
Date: Tue, 4 Aug 2009 08:00:19 -0400 (EDT)

On Tue, 4 Aug 2009, Daniel Stenberg wrote:

> On Mon, 3 Aug 2009, Seth Vidal wrote:
>> I've done a first release of urlgrabber using pycurl as a backend. The
>> urlgrabber api is unchanged but what happens in the back has changed a bit.
>> I was posting it here since I thought it might be handy for some folks.
> Cool!
> Is there any reasoning for this added support mentioned anywhere for me to
> read? I mean why the urllib stuff was deemed not good enough.
> I'm really not a python guy, but as a libcurl hacker I'm very interested in
> reasons for usage and converting from other alternatives!

two major reasons driving it:
1. fips certification that curl has when linked to nss - urllib/m2crypto
doesn't have it

2. dealing with http proxies and accessing https sites - urllib can't
really do it - curl can.

minor reasons:
- all devel on urllib is pretty much stopped - if we want things fixed we
either havs to abstract them and fix it or we have to get the fix into
upstream python which can be time consuming.

- to see if it was doable or not.

- multi-file download support. - I've not implemented that yet to
CurlMulti() but it is part of a plan.


Received on 2009-08-04