cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: DNS-based cluster awareness for connection pools and pipelines

From: Daniel Stenberg <daniel_at_haxx.se>
Date: Fri, 12 Apr 2013 13:22:40 +0200 (CEST)

On Wed, 10 Apr 2013, David Strauss wrote:

> I see some exciting pipeline-management features landing in new cURL
> releases that balance connections to the same hostname. Is there any
> interest in extending such support to balancing/fail-over between multiple
> A/AAAA records returned for a domain?

I think it depends a little on what exactly that means. I'm generally for
implementing and providing features that someone wants to imlement and that
have a fair chance of getting used by more than just a handful of users.

> Since cURL seems to prefer its own DNS client, it should be possible to
> expose more DNS response data to code managing pipelines and the connection
> pool.

We have several different backends for name resolving primarily because the
default API doesn't allow non-blocking functionality.

Fetching other DNS data than the plain A/AAAA records of today, will of course
require use of a suitable DNS-capable lib.

> This would be useful for services that connect to multi-master and
> replicated systems because it would allow client-based recovery without an
> intervening load-balancer. It may also be worthwhile to support reading
> SRV-based weights to provide hints to clients about which servers to prefer
> for load distribution, bandwidth, or latency reasons.

SRV record support has been discussed before but hasn't ever been implemented
by anyone. I think it would be an intersting addition.

But SRV records will only help us connect to a particular address. Once
connected libcurl prefers keeping connected to the same host and re-using that
for all further requests as far as possible. Your talk of "load balancing"
make me suspect that you may have other ideas than that, or what you would
load balance between exactly?

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Received on 2013-04-12