Buy commercial curl support. We
help you work out your issues, debug your libcurl applications, use the API,
port to new platforms, add new features and more. With a team lead by the
curl founder Daniel himself.
Re: Debugging segfaults during pycurl tests
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Scott Talbert via curl-library <curl-library_at_lists.haxx.se>
Date: Sat, 7 Dec 2024 12:06:26 -0500 (EST)
On Mon, 18 Nov 2024, Dan Fandrich via curl-library wrote:
>> Starting around curl 8.10.0, we started to see intermittent segfaults while running pycurl's unit
>> tests. The stacktraces generally all look like this:
>>
>> Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `python3 -m pytest -v'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 __pthread_kill_implementation (threadid=<optimized out>,
>> signo=signo_at_entry=11, no_tid=no_tid_at_entry=0) at ./nptl/pthread_kill.c:44
>>
>> warning: 44 ./nptl/pthread_kill.c: No such file or directory
>> [Current thread is 1 (Thread 0x7fdf4910e080 (LWP 3532))]
>> (gdb) bt
>> #0 __pthread_kill_implementation (threadid=<optimized out>,
>> signo=signo_at_entry=11, no_tid=no_tid_at_entry=0) at ./nptl/pthread_kill.c:44
>> #1 0x00007fdf491a2ebf in __pthread_kill_internal (threadid=<optimized out>,
>> signo=11) at ./nptl/pthread_kill.c:78
>> #2 0x00007fdf4914ec82 in __GI_raise (sig=11) at ../sysdeps/posix/raise.c:26
>> #3 <signal handler called>
>> #4 Curl_llist_insert_next (list=list_at_entry=0xd1d4180, e=0x0,
>> p=p_at_entry=0xd1b09a0, ne=ne_at_entry=0xd1b09c0)
>> at ./debian/build/lib/llist.c:102
>
> Crashes here are almost always the result of using libcurl in a multithreading
> manner without taking the precautions listed in
> https://curl.se/libcurl/c/threadsafe.html The biggest one that's still left in
> recent libcurl versions is that handles may not ever be accessed in more than
> one thread at the same time, but another is to set CURLOPT_NOSIGNAL to 1.
>
> I don't know what your test code looks like or how you're distributing tests
> across threads, but those would be the first two to look at. For example, if the
> test class creates a single handle that each test method then uses, running
> those test methods in parallel could easily result in this kind of chaos.
Thanks for the hints!
It took a long time to track down, but it seems that the crashes were
ultimately caused by this simple typo (that existed for over 10 years
before it became a problem!):
https://github.com/pycurl/pycurl/commit/2f1dcfd6dcd429dc95f8f6122c243c91c710dfff
Regards,
Scott
Date: Sat, 7 Dec 2024 12:06:26 -0500 (EST)
On Mon, 18 Nov 2024, Dan Fandrich via curl-library wrote:
>> Starting around curl 8.10.0, we started to see intermittent segfaults while running pycurl's unit
>> tests. The stacktraces generally all look like this:
>>
>> Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `python3 -m pytest -v'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 __pthread_kill_implementation (threadid=<optimized out>,
>> signo=signo_at_entry=11, no_tid=no_tid_at_entry=0) at ./nptl/pthread_kill.c:44
>>
>> warning: 44 ./nptl/pthread_kill.c: No such file or directory
>> [Current thread is 1 (Thread 0x7fdf4910e080 (LWP 3532))]
>> (gdb) bt
>> #0 __pthread_kill_implementation (threadid=<optimized out>,
>> signo=signo_at_entry=11, no_tid=no_tid_at_entry=0) at ./nptl/pthread_kill.c:44
>> #1 0x00007fdf491a2ebf in __pthread_kill_internal (threadid=<optimized out>,
>> signo=11) at ./nptl/pthread_kill.c:78
>> #2 0x00007fdf4914ec82 in __GI_raise (sig=11) at ../sysdeps/posix/raise.c:26
>> #3 <signal handler called>
>> #4 Curl_llist_insert_next (list=list_at_entry=0xd1d4180, e=0x0,
>> p=p_at_entry=0xd1b09a0, ne=ne_at_entry=0xd1b09c0)
>> at ./debian/build/lib/llist.c:102
>
> Crashes here are almost always the result of using libcurl in a multithreading
> manner without taking the precautions listed in
> https://curl.se/libcurl/c/threadsafe.html The biggest one that's still left in
> recent libcurl versions is that handles may not ever be accessed in more than
> one thread at the same time, but another is to set CURLOPT_NOSIGNAL to 1.
>
> I don't know what your test code looks like or how you're distributing tests
> across threads, but those would be the first two to look at. For example, if the
> test class creates a single handle that each test method then uses, running
> those test methods in parallel could easily result in this kind of chaos.
Thanks for the hints!
It took a long time to track down, but it seems that the crashes were
ultimately caused by this simple typo (that existed for over 10 years
before it became a problem!):
https://github.com/pycurl/pycurl/commit/2f1dcfd6dcd429dc95f8f6122c243c91c710dfff
Regards,
Scott
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2024-12-07