On Wed, Jul 16, 2003 at 11:44:52AM +0200, Daniel Stenberg wrote:
> On Wed, 16 Jul 2003, jose wrote:
>
> > > I don't think this is a feture we need to add to libcurl, as you can make
> > > this happen perfectly easy all by yourself. Yes, it requires you to parse
> > > and "fix" the URL yourself, but it still is a pretty weird operation...
> >
> > Any app that wants to get performance with Curl needs to bypass the Curl
> > sync DNS queries and currently this has not received much attention.
>
> Well, I don't quite agree with your statements. First, I think you can get
> quite good performance with curl even though it looks up names synchronously,
> and secondly, it has gotten some attention.

The fact that it looks up names synchronously pretty much requires that
you use multiple threads for "good performance". This is actually a little
bit of a pain if you are trying to write a library that will be easily
wrapped into other programming languages. We have had constant problems
with languages such as Perl and PHP for which the default version does
not necessarily contain thread support.

For example, the version of PHP being released with Redhat versions 7-9
is compiled without -lpthread. This makes it impossible to have a PHP
extension that uses threads. (If you don't believe me, I can give a
longer explanation of why this is a problem). To use a PHP extension
with threads inside the PHP module of Apache 1.x currently requires that
you recompile both PHP and Apache.

Anyway, because threads are a pain (and on some systems offer a big
performance hit), we have coincidentally been looking into async
DNS libraries this week.

> > Any ideas on how to do this effectively would be very useful to many users.
> > Given that libdenise (Curl asynch DNS sister library) is not progressing, it
> > would be helpful to show how Curl could be easily interface with glibc
> > asynch DNS.
>
> The best way to do progress on this would be to make libdenise move forward,
> IMHO.
>
> IIRC, the async stuff in the glibc name lookups are: 1) badly documented and
> 2) based on signals. The signal dependency effectively kills the ability to
> use this with threads. Not to mention that glibc isn't very portable outside
> Linux boxes.

Have you looked at "ares"? It's an older library out of MIT that doesn't
seem to have much of a web presence. You can download it from:

ftp://athena-dist.mit.edu/pub/ATHENA/ares/

and it appears to be used in the gnome socket library:

http://gnome-socket.sourceforge.net/

The license is MIT which I believe makes it compatible with Curl.
It seems to support exactly the interface that Curl would like to have.
Cutting and pasting from an example program shows the interface:

  status = ares_init(&channel);
  /* calls the function callback with user data *argv on completion */
  ares_gethostbyname(channel, *argv, AF_INET, callback, *argv);

  while (1) {
      FD_ZERO(&read_fds);
      FD_ZERO(&write_fds);
      nfds = ares_fds(channel, &read_fds, &write_fds);
      if (nfds == 0)
        break;
      /* the NULL here is your timeout, if you wanted something different */
      tvp = ares_timeout(channel, NULL, &tv);
      select(nfds, &read_fds, &write_fds, NULL, tvp);
      ares_process(channel, &read_fds, &write_fds);
  }

and callback is:

static void callback(void *arg, int status, struct hostent *host);

As far as I can tell, there are only two problems with this library.
The first is the lack of IPv6 support. I have no idea how serious
a problem this would be. Is there a new DNS protocol or are there
just minor differences that you are able to deal with?
The second is that it is UNIX based. In the last couple of days,
we ported it to windows and the example programs are working (but
we haven't tested it all that extensively yet).

If Curl is interested in looking into this library, I'd be glad to
make a patch available as well as a VC++ project for the windows
version of the library.

Cheers,
Chris.

--
Christopher R. Palmer                     palmer_at_vivisimo.com
Chief Technology Officer                  www.vivisimo.com
Vivisimo, Inc.                            412-422-2499
-------------------------------------------------------
This SF.net email is sponsored by: VM Ware
With VMware you can run multiple operating systems on a single machine.
WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the
same time. Free trial click here: http://www.vmware.com/wl/offer/345/0