Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Better resolver callbacks.
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: John Wiseman via curl-library <curl-library_at_cool.haxx.se>
Date: Fri, 23 Oct 2020 13:24:18 +0000
> On Thu, 15 Oct 2020, John Wiseman via curl-library wrote:
>
> > I'm trying to avoid flooding the dns resolver when using multi interface. I
> > know it's on the TODO list, but I think it may be doable from user side with
> > callbacks.
>
> What exactly do you want to limit? The number of names to resolve within a
> certain period a time? Is this a real problem for you or are you just taking
> precautions?
It's the number of max concurrent dns queries. Without such limit, my crawler will easyily blow up the dns server of my home network with almost 1 thousand threads doing dns query on just several names. I've set connection limits on multi handle, but seems it won't limit these dns queries.
> > For example, a callback to inform the start of lookup which passes in the
> > name to be resolved. User can return some value in this callback to inform
> > curl to pause on the beginning of lookup. Another callback to inform the end
> > of lookup which passes in the name being resolved and the result.
>
> It could proably get done that way, but I think that if we would work on
> adding such an elaborate menanism, then couldn't we just as well add the
> entire "reduce name resolver speed" thing in libcurl?
Of course, that will be the idea solution.
> Or even: if you would be willing to add code in your application to limit the
> name resolve frequency, couldn't you just as well then just hang on a little
> before you add transfers using a new host name so that you can keep below the
> threshold?
Yes, it's possible, but suboptimal. Things like dns TTL and proxy also make it hard to do right. Without some support from libcurl, it will add lots of complexity to my code and still not fail-safe.
regards, John Wiseman
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2020-10-23
Date: Fri, 23 Oct 2020 13:24:18 +0000
> On Thu, 15 Oct 2020, John Wiseman via curl-library wrote:
>
> > I'm trying to avoid flooding the dns resolver when using multi interface. I
> > know it's on the TODO list, but I think it may be doable from user side with
> > callbacks.
>
> What exactly do you want to limit? The number of names to resolve within a
> certain period a time? Is this a real problem for you or are you just taking
> precautions?
It's the number of max concurrent dns queries. Without such limit, my crawler will easyily blow up the dns server of my home network with almost 1 thousand threads doing dns query on just several names. I've set connection limits on multi handle, but seems it won't limit these dns queries.
> > For example, a callback to inform the start of lookup which passes in the
> > name to be resolved. User can return some value in this callback to inform
> > curl to pause on the beginning of lookup. Another callback to inform the end
> > of lookup which passes in the name being resolved and the result.
>
> It could proably get done that way, but I think that if we would work on
> adding such an elaborate menanism, then couldn't we just as well add the
> entire "reduce name resolver speed" thing in libcurl?
Of course, that will be the idea solution.
> Or even: if you would be willing to add code in your application to limit the
> name resolve frequency, couldn't you just as well then just hang on a little
> before you add transfers using a new host name so that you can keep below the
> threshold?
Yes, it's possible, but suboptimal. Things like dns TTL and proxy also make it hard to do right. Without some support from libcurl, it will add lots of complexity to my code and still not fail-safe.
regards, John Wiseman
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2020-10-23