On 7/31/2015 12:27 AM, Eric Ridge wrote:
>> On Jul 29, 2015, at 5:35 PM, Eric Ridge <eebbrr_at_gmail.com> wrote:
>>
>> It's probably worth mentioning that I'm also using curl_global_init_mem() to set a custom allocator (this is inside Postgres, so I trust their alloc functions) and that the process is not compiled for thread safety.
> I've spent some time digging deep into this, and, unsurprisingly, I'm pretty sure I've been doing it wrong.
>
> What it looks like is that certain versions of libcurl included with certain Linux distros (and OS X) are compiled with AsyncDNS, which best I can tell causes libcurl to spin off a new thread for every new CURL object.
>
> When that thread decides it needs to allocate some memory using the custom allocator functions (as set by curl_global_init_mem()), the process segfaults.
>
> The reason for this is that, in my case, Postgres' memory allocator is, by design, not thread safe and if it's trying to allocate memory at the same time the AsyncDNS thread is trying to allocate memory, bad things happen.
>
> Now that I know it's possible for libcurl to create threads, it's stunningly obvious why this happens, and I've since worked around it by *not* calling curl_global_init_mem() and instead manually dealing with libcurl working outside of Postgres' transaction-based MemoryContexts, but I'd like to suggest that the curl_global_init_mem() documentation at least make a mention that a particular gotcha is concurrency.

Thanks for your report. I updated the documentation in
curl_global_init_mem.3 and libcurl-thread.3.

https://github.com/bagder/curl/commit/0da1f5d
https://github.com/bagder/curl/commit/23b84e4

Perhaps elsewhere another warning is due for when the threaded resolver
is used and the OS memory functions are not thread safe, if that
configuration can happen? I will leave that for someone more experienced
with the codebase.

> Maybe I just haven't found it in the docs, but it's non-obvious that libcurl is starting threads under some conditions (that cannot be controlled at runtime: http://curl.haxx.se/mail/curlphp-2011-11/0001.html). The "Multi-threading issues" section of the tutorial (http://curl.haxx.se/libcurl/c/libcurl-tutorial.html) links to http://curl.haxx.se/libcurl/c/libcurl-thread.html which 404's. :(
>
> Anyways, thanks for your time and for such a great product. libcurl really is great and powerful!

Thanks for pointing that out and your positive comments. It will be
fixed soon but for now you can view it raw at
https://github.com/bagder/curl/blob/master/docs/libcurl/libcurl-thread.3

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2015-07-31