On 11/7/05, Daniel Stenberg <daniel_at_haxx.se> wrote:
> On Mon, 7 Nov 2005, Cory Nelson wrote:
>
> > Sorry for the late reply, been busy...
>
> No worries at all. This project won't be "done" in several months anyway, and
> I won't get up to full speed on it until December.
>
> > Here is a small sample that should give you the idea. Unfortunately I don't
> > know *nix APIs very well, so I'm unable to guess the best API that would
> > work for the both of them.
>
> You don't have to have that knowledge. But I would rather like us to discuss
> around and based on the current API proposal and the psuedo code app as I've
> published on the site, as it has been developed over time through discussions
> to become a fairly generic approach that would fit "random" event systems.
>
> > I'd make something like:
> >
> > // called on completion, error, etc..
> > typedef void (*hiper_callback)(int status, CURL *easy);
> >
> > // creates the handler.
> > hiper_handle hiper_create(hiper_callback);
> > void hiper_destroy(hiper_handle);
> >
> > // pushes an easy handle onto the hyper handle.
> > void hiper_push(hiper_handle, CURL *easy);
> >
> > // blocks until all easy handles are done processing.
> > void hiper_wait(hiper_handle);
> >
> > The big change is making the wait function optional- blocking doesn't belong
> > in a high perf app. I'm assuming that function currently processes all the
> > sockets- I believe that should all be done in the background for the user so
> > CURL can make proper use of SMP if it's available. And maybe add another
> > function to pull the current status of an easy handle.
> >
> > What are your thoughts?
>
> First I sould say that I don't know much about how to design this to make it
> work as good as possible in an SMP environment, and I haven't given that so
> much consideration so far.
>
> A few things about your proposal:
>
> I don't see how that is a fully working API. My idea is to extend/modify the
> current multi interface, which thus allows simultaneous transfers in the same
> thread. I have no plans to add any kind of multi-thread awareness in libcurl,
> but of course i should still function correctly when used multi-threaded.

The current multi interface of having the user wait for events is not
an efficient design when it comes to IOCP.

IOCP uses threading to get full scalabilty on SMP systems. It could
certainly be done from one thread, but then you would be losing many
of the benefits. This is aimed at being high performance so libcurl
(imho) should definately add the minimal threading awareness for IOCP,
if only for this hiper API.

The API I gave was complete. You open a hiper handle and push your
easy handles into it. The hiper handle would open one or more threads
in the background and callback when something significant happens to
an easy handle. Simple as that. If you wanted to wait until all the
easy handles are finished, you can call the wait function.

> That said, your "should all be done in the background" comment sounds weird in
> my ears. Sure you can start a new thread and have libcurl do all the work
> there and thus it works "in the background", but I don't see how libcurl must
> be modified to deal with that.

You could do that, but in that case SMP wouldn't be very easy to do.
If hiper is meant to abstract away all the stuff needed for high
performance http, it should also be in charge of threading efficiently
as needed.

> The current multi interface is not blocking. The only blocking in a typical
> app should be done by the event system call that waits for action on one or
> more sockets. And that is made by the app, outside of libcurl's control.

That's where the confusion is :) My proposal moves the blocking into
libcurl where it could be efficiently managed and out of the users
reach.

> So I guess in the end I don't fully understand your proposal. Can you expand
> it to a fully "working" pseudo code example?

> --
> Commercial curl and libcurl Technical Support: http://haxx.se/curl.html
>

--
Cory Nelson
http://www.int64.org