> On Jul 29, 2015, at 5:35 PM, Eric Ridge <eebbrr_at_gmail.com> wrote:
>
> It's probably worth mentioning that I'm also using curl_global_init_mem() to set a custom allocator (this is inside Postgres, so I trust their alloc functions) and that the process is not compiled for thread safety.

I've spent some time digging deep into this, and, unsurprisingly, I'm pretty sure I've been doing it wrong.

What it looks like is that certain versions of libcurl included with certain Linux distros (and OS X) are compiled with AsyncDNS, which best I can tell causes libcurl to spin off a new thread for every new CURL object.

When that thread decides it needs to allocate some memory using the custom allocator functions (as set by curl_global_init_mem()), the process segfaults.

The reason for this is that, in my case, Postgres' memory allocator is, by design, not thread safe and if it's trying to allocate memory at the same time the AsyncDNS thread is trying to allocate memory, bad things happen.

Now that I know it's possible for libcurl to create threads, it's stunningly obvious why this happens, and I've since worked around it by *not* calling curl_global_init_mem() and instead manually dealing with libcurl working outside of Postgres' transaction-based MemoryContexts, but I'd like to suggest that the curl_global_init_mem() documentation at least make a mention that a particular gotcha is concurrency.

Maybe I just haven't found it in the docs, but it's non-obvious that libcurl is starting threads under some conditions (that cannot be controlled at runtime: http://curl.haxx.se/mail/curlphp-2011-11/0001.html). The "Multi-threading issues" section of the tutorial (http://curl.haxx.se/libcurl/c/libcurl-tutorial.html) links to http://curl.haxx.se/libcurl/c/libcurl-thread.html which 404's. :(

Anyways, thanks for your time and for such a great product. libcurl really is great and powerful!

eric

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2015-07-31