High Performance libcurl
Hiper is the name of the sub-project that intends to bring high performance adjustments and additions to the libcurl multi interface.
The aim is to
- provide a better API for extremely large amounts of simultaneous transfers (the c10k problem). By better API, we mean both better suited to deal with the amount of connections (i.e no artificial limits like select() imposes etc), better performance-wise by avoiding expensive looping over all file descriptors to check for actions and to allow users to pick an underlying event system of their own choice.
- add HTTP pipelining support.
- (NEW) provide a zero copy interface that reduces the amount of data copies needed when transferring data to/from memory.
The order above is the order of preference and implementation.
I've also decided not to:
- Make any parts of this new API aware of any thread API or similar, it remains thread agnostic but of course perfectly good to use in (many) threads.
- Offer any API to help applications to dispatch transfer jobs between many threads. A high-volume transfer application could very well have N threads with a multi handle in each, using the Hiper API, but we do not offer any built-in support to dispatch transfers to the various threads. This is indeed room for future improvements.
Schedule
Daniel intends to work more or less full-time with the Hiper project during December 2005. Leftovers will be done part-time in 2006 with possibly some further full-time periods during spring 2006.
I'll get back with more specific (release) dates later on. See also the roadmap.
Funding
Daniel receives funding from the Swedish foundation IIS for the purpose of implementing this.
Project Involvement
The plan is to keep the process and progress as open and public as I possibly can, to allow anyone and everyone who feels like contributing or just watching to do so. This is still curl, this is still open source and I still value and appreciate all user feedback and comments.
Progress reports will be posted and discussions will be held on the regular curl-library mailing list. I will also post "quarterly" reports in Swedish to IIS (as mandated by the contract I've signed with them as a condition for me to receive the money), and I will make those reports available here online as well.
Source Code Repository
For the sake of being able to be 100% public and open, I will create a branch in the CVS repository so that the Hiper experiments and developments won't interfere with the "normal" libcurl development.
The name of the tag will be announced when this has been done.