"Guillaume Arluison" <ga_at_lacde.net> writes:

> Ok it wasnt enough clear before.
> I must admit I dont really understand why using a web server for that (the
> example is exagerated of course so I dont know when you ll need this but it
> makes more sense) maybe because it isnt a choice !
> I dont know if you control the targeted webserver but the simplest solution
> would be a cgi on this one that cuts/split/add the bits from your file(s)
> before sending it...

The task at hand is a rsync reimplementation that works off any (sane,
so no IIs v5) http server. Intended use is to download debian
packages, cd and dvd isos from any of the various http mirrors while
still benefiting from rsyncs error correction and update capabilities.

Normaly this means I don't have any image and need the full file or I
already have an image that is very much complete and only needs a few
chunks. For a dvd image a few chunks might still be a sizeable amount,
~1000 chunks for a weekly image.

> If you dont really need all the clever features of curl and do only simple
> gets, a simple program using sockets may be sufficient instead of using the
> curl library ?

Having all the extra features like proxy, ssl, user/pass, error
detection and reconnects and such certainly wouldn't hurt.

> Depends on what do you need of course and I didnt test this pipelining
> feature before in a simple open socket program (beware that your webserver
> must handle this as well, IIS v5 has some bugs around that actually) but
> coding a basic version like :
>
> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> #now connect to the web server on port 80
> s.connect((domain, 80))
> s.send('GET ' + filePathName + ' HTTP/1.1\n')
> s.send('Host: '+domain+'\n') # to add if we use HTTP/1.1
> s.send('\n')
> while 1:
> data=s.recv(1024)
> l=len(data)
> if l==0: break
> # put some send headers here too, check size to cut your answers...
> s.close()
>
> is not too difficult the problem is of course the number of features you
> want to add (handle errors/timeouts with alarms etc...) but it is still no
> too complex is it ?

Surely not, I can just rewrite libcurl with a few less features. :)

But I'm very sure adding pipelining support to libcurl will be the
easier task.

MfG
        Goswin

PS: I recently noticed someone else implementing nearly exactly what I
am doing and I will most likely just join his project. Google for zsync.
Received on 2004-11-26