On Mon, 22 Oct 2012 22:07:16 +0200 Dan Fandrich <dan_at_coneharvesters.com> wrote:
> On Mon, Oct 22, 2012 at 02:24:01AM +0200, Thorben Thuermer wrote:
> > on http://curl.haxx.se/libcurl/features.html we find:
> > "libcurl is designed and implemented entirely thread safe."
> > "libcurl uses certain system calls to obtain information. Some of the most
> > crucial ones are the name resoluition calls (the gethostby* family)."
> >
> > i just debugged a problem in a multithreaded application
> > ( https://github.com/volkszaehler/vzlogger , thread in german:
> > http://volkszaehler.org/pipermail/volkszaehler-users/2012-October/000619.html )
> > that appers to be caused by libcurl's (ab)use of alarm() to timeout
> > gethostbyname() not being thread-safe.
> > (as described here: https://bugzilla.redhat.com/show_bug.cgi?id=539809 ).
> >
> > there appears to be a race-condition when two threads run requests
> > simultaneously, which reproducibly leads to a crash with
> > the logging function being called with an invalid buffer/file,
> > without any actual dns-timeout having occured.
> > (the application only accesses localhost, which is defined in /etc/hosts,
> > curiously the problem persists if a numeric IP is used.)
>
> If the problem exists even with a numeric IP, then it's unlikely to be a
> DNS issue since numeric addresses bypass the external resolver. Are you
> making SSL connections by chance? There is more you may need to do in that
> case; http://curl.haxx.se/libcurl/c/libcurl-tutorial.html#Multi-threading

no, plain http.

> > the problem disappeared when i compiled libcurl without HAVE_ALARM.
>
> CURLOPT_NOSIGNAL ought to do the same at run-time,

i found that option in the meantime, yes.

> but it's odd that
> this occurs with numeric addresses (assuming SSL is not involved).

i just checked by placing a breakpoint in alarm(),
alarm() is called even when connecting to numeric IP addresses,
when gethostbyname itself is apparently not called.
(that was my point actually, the alarm() stuff is non-threadsafe,
 regardless of gethostbyname - thus the default resolver is not threadsafe,
 regardless of the OS-provided gethostbyname, unlike what the features page
 states.
 but as you point out below, one should be setting URLOPT_NOSIGNAL anyway.)

> > i'm not sure if this is a libcurl bug or just a documentation issue,
> > but it has cost us some time to track down...
>
> This is from curl_easy_setopt(3)
> http://curl.haxx.se/libcurl/c/curl_easy_setopt.html
>
> CURLOPT_NOSIGNAL
>
> Pass a long. If it is 1, libcurl will not use any functions that install
> signal handlers or any functions that cause signals to be sent to the
> process. This option is mainly here to allow multi-threaded unix
> applications to still set/use all timeout options etc, without risking
> getting signals. (Added in 7.10)
>
> And this is from libcurl-tutorial(3)
> http://curl.haxx.se/libcurl/c/libcurl-tutorial.html#Multi-threading
>
> When using multiple threads you should set the CURLOPT_NOSIGNAL option
> to 1 for all handles.

ok, so it is documented, just not where i was looking
(i was looking for resolver-related issues after seeing the backtrace).
i did not write the application itself, i'll blame the original developer
for not reading the provided documentation on threaded usage. ;)

> libcurl now gives a number of choices of DNS resolver back-end at
> compile time, some of which are immune to this problem.

yes, i found those.
but it's rather unfortunate that this is compile-time so most users would
have to build libcurl from source to take advantage of this.

> >>> Dan

thanks for the heads up,

- T.
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2012-10-22