> Are you using the original versions of libcurl and nss from Fedora? If yes,
> please provide their NVR. Or have you compiled libcurl and/or nss on your own?

We are building our own curl, but not our own NSS.

nss-3.12.4-3.fc11.i586

Our curl is built with the patch I sent for the double close bug. That is the only modification.

curl 7.19.6 (i686-pc-linux-gnu) libcurl/7.19.6 NSS/3.12.4.5 zlib/1.2.3 libidn/1.9 libssh2/1.0
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
Features: GSS-Negotiate IDN IPv6 Largefile SSL libz

> It was reported as a random error in a complex multi-threaded application.

The application I am seeing it in is also heavily multithreaded and happens when we have a lot of connections. Please let me know what I can do to help....

>
I was able to reproduce it under valgrind and there were a couple of
invalid writes and an invalid free that happened before the core dump.
these may help.

==2499== Thread 6:
==2499== Invalid write of size 1
==2499== at 0x402A658: memset (mc_replace_strmem.c:493)
==2499== by 0x6E0F36B: nsslibc_memset (string3.h:85)
==2499== by 0x6E0E776: nss_ZFreeIf (arena.c:974)
==2499== by 0x6DFF733: pem_PopulateModulusExponent (prsa.c:214)
==2499== by 0x6DFEB86: pem_FetchPrivKeyAttribute (pobject.c:318)
==2499== by 0x6DFEF5F: pem_FetchAttribute (pobject.c:541)
==2499== by 0x6DFF199: pem_mdCryptoOperationRSA_GetFinalLength (prsa.c:352)
==2499== by 0x6E0DA5C: nssCKFWCryptoOperation_GetFinalLength (crypto.c:178)
==2499== by 0x6E032D9: nssCKFWSession_UpdateFinal (session.c:2219)
==2499== by 0x6E086DB: NSSCKFWC_Sign (wrap.c:3819)
==2499== by 0x6DFAE92: pemC_Sign (nssck.api:1141)
==2499== by 0x4878D4D: PK11_Sign (pk11obj.c:768)
==2499== Address 0x657e33c is 60 bytes inside a block of size 72 free'd
==2499== at 0x4027BCA: free (vg_replace_malloc.c:323)
==2499== by 0x497BD96: PR_Free (in /lib/libnspr4.so)
==2499== by 0x6E0E77E: nss_ZFreeIf (arena.c:975)
==2499== by 0x6DFF733: pem_PopulateModulusExponent (prsa.c:214)
==2499== by 0x6DFEB86: pem_FetchPrivKeyAttribute (pobject.c:318)
==2499== by 0x6DFEF5F: pem_FetchAttribute (pobject.c:541)
==2499== by 0x6DFF199: pem_mdCryptoOperationRSA_GetFinalLength (prsa.c:352)
==2499== by 0x6E0DA5C: nssCKFWCryptoOperation_GetFinalLength (crypto.c:178)
==2499== by 0x6E032D9: nssCKFWSession_UpdateFinal (session.c:2219)
==2499== by 0x6E086DB: NSSCKFWC_Sign (wrap.c:3819)
==2499== by 0x6DFAE92: pemC_Sign (nssck.api:1141)
==2499== by 0x4878D4D: PK11_Sign (pk11obj.c:768)
==2499==
==2499== Invalid write of size 1
==2499== at 0x402A65B: memset (mc_replace_strmem.c:493)
==2499== by 0x6E0F36B: nsslibc_memset (string3.h:85)
==2499== by 0x6E0E776: nss_ZFreeIf (arena.c:974)
==2499== by 0x6DFF733: pem_PopulateModulusExponent (prsa.c:214)
==2499== by 0x6DFEB86: pem_FetchPrivKeyAttribute (pobject.c:318)
==2499== by 0x6DFEF5F: pem_FetchAttribute (pobject.c:541)
==2499== by 0x6DFF199: pem_mdCryptoOperationRSA_GetFinalLength (prsa.c:352)
==2499== by 0x6E0DA5C: nssCKFWCryptoOperation_GetFinalLength (crypto.c:178)
==2499== by 0x6E032D9: nssCKFWSession_UpdateFinal (session.c:2219)
==2499== by 0x6E086DB: NSSCKFWC_Sign (wrap.c:3819)
==2499== by 0x6DFAE92: pemC_Sign (nssck.api:1141)
==2499== by 0x4878D4D: PK11_Sign (pk11obj.c:768)
==2499== Address 0x657e33d is 61 bytes inside a block of size 72 free'd
==2499== at 0x4027BCA: free (vg_replace_malloc.c:323)
==2499== by 0x497BD96: PR_Free (in /lib/libnspr4.so)
==2499== by 0x6E0E77E: nss_ZFreeIf (arena.c:975)
==2499== by 0x6DFF733: pem_PopulateModulusExponent (prsa.c:214)
==2499== by 0x6DFEB86: pem_FetchPrivKeyAttribute (pobject.c:318)
==2499== by 0x6DFEF5F: pem_FetchAttribute (pobject.c:541)
==2499== by 0x6DFF199: pem_mdCryptoOperationRSA_GetFinalLength (prsa.c:352)
==2499== by 0x6E0DA5C: nssCKFWCryptoOperation_GetFinalLength (crypto.c:178)
==2499== by 0x6E032D9: nssCKFWSession_UpdateFinal (session.c:2219)
==2499== by 0x6E086DB: NSSCKFWC_Sign (wrap.c:3819)
==2499== by 0x6DFAE92: pemC_Sign (nssck.api:1141)
==2499== by 0x4878D4D: PK11_Sign (pk11obj.c:768)
==2499==
==2499== Invalid write of size 1
==2499== at 0x402A65F: memset (mc_replace_strmem.c:493)
==2499== by 0x6E0F36B: nsslibc_memset (string3.h:85)
==2499== by 0x6E0E776: nss_ZFreeIf (arena.c:974)
==2499== by 0x6DFF733: pem_PopulateModulusExponent (prsa.c:214)
==2499== by 0x6DFEB86: pem_FetchPrivKeyAttribute (pobject.c:318)
==2499== by 0x6DFEF5F: pem_FetchAttribute (pobject.c:541)
==2499== by 0x6DFF199: pem_mdCryptoOperationRSA_GetFinalLength (prsa.c:352)
==2499== by 0x6E0DA5C: nssCKFWCryptoOperation_GetFinalLength (crypto.c:178)
==2499== by 0x6E032D9: nssCKFWSession_UpdateFinal (session.c:2219)
==2499== by 0x6E086DB: NSSCKFWC_Sign (wrap.c:3819)
==2499== by 0x6DFAE92: pemC_Sign (nssck.api:1141)
==2499== by 0x4878D4D: PK11_Sign (pk11obj.c:768)
==2499== Address 0x657e33e is 62 bytes inside a block of size 72 free'd
==2499== at 0x4027BCA: free (vg_replace_malloc.c:323)
==2499== by 0x497BD96: PR_Free (in /lib/libnspr4.so)
==2499== by 0x6E0E77E: nss_ZFreeIf (arena.c:975)
==2499== by 0x6DFF733: pem_PopulateModulusExponent (prsa.c:214)
==2499== by 0x6DFEB86: pem_FetchPrivKeyAttribute (pobject.c:318)
==2499== by 0x6DFEF5F: pem_FetchAttribute (pobject.c:541)
==2499== by 0x6DFF199: pem_mdCryptoOperationRSA_GetFinalLength (prsa.c:352)
==2499== by 0x6E0DA5C: nssCKFWCryptoOperation_GetFinalLength (crypto.c:178)
==2499== by 0x6E032D9: nssCKFWSession_UpdateFinal (session.c:2219)
==2499== by 0x6E086DB: NSSCKFWC_Sign (wrap.c:3819)
==2499== by 0x6DFAE92: pemC_Sign (nssck.api:1141)
==2499== by 0x4878D4D: PK11_Sign (pk11obj.c:768)
==2499==
==2499== Invalid write of size 1
==2499== at 0x402A663: memset (mc_replace_strmem.c:493)
==2499== by 0x6E0F36B: nsslibc_memset (string3.h:85)
==2499== by 0x6E0E776: nss_ZFreeIf (arena.c:974)
==2499== by 0x6DFF733: pem_PopulateModulusExponent (prsa.c:214)
==2499== by 0x6DFEB86: pem_FetchPrivKeyAttribute (pobject.c:318)
==2499== by 0x6DFEF5F: pem_FetchAttribute (pobject.c:541)
==2499== by 0x6DFF199: pem_mdCryptoOperationRSA_GetFinalLength (prsa.c:352)
==2499== by 0x6E0DA5C: nssCKFWCryptoOperation_GetFinalLength (crypto.c:178)
==2499== by 0x6E032D9: nssCKFWSession_UpdateFinal (session.c:2219)
==2499== by 0x6E086DB: NSSCKFWC_Sign (wrap.c:3819)
==2499== by 0x6DFAE92: pemC_Sign (nssck.api:1141)
==2499== by 0x4878D4D: PK11_Sign (pk11obj.c:768)
==2499== Address 0x657e33f is 63 bytes inside a block of size 72 free'd
==2499== at 0x4027BCA: free (vg_replace_malloc.c:323)
==2499== by 0x497BD96: PR_Free (in /lib/libnspr4.so)
==2499== by 0x6E0E77E: nss_ZFreeIf (arena.c:975)
==2499== by 0x6DFF733: pem_PopulateModulusExponent (prsa.c:214)
==2499== by 0x6DFEB86: pem_FetchPrivKeyAttribute (pobject.c:318)
==2499== by 0x6DFEF5F: pem_FetchAttribute (pobject.c:541)
==2499== by 0x6DFF199: pem_mdCryptoOperationRSA_GetFinalLength (prsa.c:352)
==2499== by 0x6E0DA5C: nssCKFWCryptoOperation_GetFinalLength (crypto.c:178)
==2499== by 0x6E032D9: nssCKFWSession_UpdateFinal (session.c:2219)
==2499== by 0x6E086DB: NSSCKFWC_Sign (wrap.c:3819)
==2499== by 0x6DFAE92: pemC_Sign (nssck.api:1141)
==2499== by 0x4878D4D: PK11_Sign (pk11obj.c:768)
==2499==
==2499== Invalid write of size 1
==2499== at 0x402A66C: memset (mc_replace_strmem.c:493)
==2499== by 0x6E0F36B: nsslibc_memset (string3.h:85)
==2499== by 0x6E0E776: nss_ZFreeIf (arena.c:974)
==2499== by 0x6DFF733: pem_PopulateModulusExponent (prsa.c:214)
==2499== by 0x6DFEB86: pem_FetchPrivKeyAttribute (pobject.c:318)
==2499== by 0x6DFEF5F: pem_FetchAttribute (pobject.c:541)
==2499== by 0x6DFF199: pem_mdCryptoOperationRSA_GetFinalLength (prsa.c:352)
==2499== by 0x6E0DA5C: nssCKFWCryptoOperation_GetFinalLength (crypto.c:178)
==2499== by 0x6E032D9: nssCKFWSession_UpdateFinal (session.c:2219)
==2499== by 0x6E086DB: NSSCKFWC_Sign (wrap.c:3819)
==2499== by 0x6DFAE92: pemC_Sign (nssck.api:1141)
==2499== by 0x4878D4D: PK11_Sign (pk11obj.c:768)
==2499== Address 0x657e340 is 64 bytes inside a block of size 72 free'd
==2499== at 0x4027BCA: free (vg_replace_malloc.c:323)
==2499== by 0x497BD96: PR_Free (in /lib/libnspr4.so)
==2499== by 0x6E0E77E: nss_ZFreeIf (arena.c:975)
==2499== by 0x6DFF733: pem_PopulateModulusExponent (prsa.c:214)
==2499== by 0x6DFEB86: pem_FetchPrivKeyAttribute (pobject.c:318)
==2499== by 0x6DFEF5F: pem_FetchAttribute (pobject.c:541)
==2499== by 0x6DFF199: pem_mdCryptoOperationRSA_GetFinalLength (prsa.c:352)
==2499== by 0x6E0DA5C: nssCKFWCryptoOperation_GetFinalLength (crypto.c:178)
==2499== by 0x6E032D9: nssCKFWSession_UpdateFinal (session.c:2219)
==2499== by 0x6E086DB: NSSCKFWC_Sign (wrap.c:3819)
==2499== by 0x6DFAE92: pemC_Sign (nssck.api:1141)
==2499== by 0x4878D4D: PK11_Sign (pk11obj.c:768)
==2499==
==2499== Invalid free() / delete / delete[]
==2499== at 0x4027BCA: free (vg_replace_malloc.c:323)
==2499== by 0x497BD96: PR_Free (in /lib/libnspr4.so)
==2499== by 0x6E0E77E: nss_ZFreeIf (arena.c:975)
==2499== by 0x6DFF733: pem_PopulateModulusExponent (prsa.c:214)
==2499== by 0x6DFEB86: pem_FetchPrivKeyAttribute (pobject.c:318)
==2499== by 0x6DFEF5F: pem_FetchAttribute (pobject.c:541)
==2499== by 0x6DFF199: pem_mdCryptoOperationRSA_GetFinalLength (prsa.c:352)
==2499== by 0x6E0DA5C: nssCKFWCryptoOperation_GetFinalLength (crypto.c:178)
==2499== by 0x6E032D9: nssCKFWSession_UpdateFinal (session.c:2219)
==2499== by 0x6E086DB: NSSCKFWC_Sign (wrap.c:3819)
==2499== by 0x6DFAE92: pemC_Sign (nssck.api:1141)
==2499== by 0x4878D4D: PK11_Sign (pk11obj.c:768)
==2499== Address 0x657e300 is 0 bytes inside a block of size 72 free'd
==2499== at 0x4027BCA: free (vg_replace_malloc.c:323)
==2499== by 0x497BD96: PR_Free (in /lib/libnspr4.so)
==2499== by 0x6E0E77E: nss_ZFreeIf (arena.c:975)
==2499== by 0x6DFF733: pem_PopulateModulusExponent (prsa.c:214)
==2499== by 0x6DFEB86: pem_FetchPrivKeyAttribute (pobject.c:318)
==2499== by 0x6DFEF5F: pem_FetchAttribute (pobject.c:541)
==2499== by 0x6DFF199: pem_mdCryptoOperationRSA_GetFinalLength (prsa.c:352)
==2499== by 0x6E0DA5C: nssCKFWCryptoOperation_GetFinalLength (crypto.c:178)
==2499== by 0x6E032D9: nssCKFWSession_UpdateFinal (session.c:2219)
==2499== by 0x6E086DB: NSSCKFWC_Sign (wrap.c:3819)
==2499== by 0x6DFAE92: pemC_Sign (nssck.api:1141)
==2499== by 0x4878D4D: PK11_Sign (pk11obj.c:768)
==2499==
==2499== Thread 26:
==2499== Invalid read of size 4
==2499== at 0x47F7BA1: ssl_DefRecv (ssldef.c:91)
==2499== by 0x47F28C5: ssl3_GatherCompleteHandshake (ssl3gthr.c:90)
==2499== by 0x47F54BA: ssl_GatherRecord1stHandshake (sslcon.c:1258)
==2499== by 0x47FB6F4: ssl_Do1stHandshake (sslsecur.c:151)
==2499== by 0x47FCDA6: SSL_ForceHandshake (sslsecur.c:407)
==2499== by 0x47FCE76: SSL_ForceHandshakeWithTimeout (sslsecur.c:428)
==2499== by 0x437ABC7: Curl_nss_connect (nss.c:1214)
==2499== by 0x4371181: Curl_ssl_connect (sslgen.c:185)
==2499== by 0x43504F8: Curl_http_connect (http.c:1792)
==2499== by 0x43579AD: Curl_protocol_connect (url.c:3056)
==2499== by 0x435CC7E: Curl_connect (url.c:4690)
==2499== by 0x4365A03: Curl_perform (transfer.c:2481)
==2499== Address 0x8 is not stack'd, malloc'd or (recently) free'd
==2499==
==2499== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==2499== Access not within mapped region at address 0x8
==2499== at 0x47F7BA1: ssl_DefRecv (ssldef.c:91)
==2499== by 0x47F28C5: ssl3_GatherCompleteHandshake (ssl3gthr.c:90)
==2499== by 0x47F54BA: ssl_GatherRecord1stHandshake (sslcon.c:1258)
==2499== by 0x47FB6F4: ssl_Do1stHandshake (sslsecur.c:151)
==2499== by 0x47FCDA6: SSL_ForceHandshake (sslsecur.c:407)
==2499== by 0x47FCE76: SSL_ForceHandshakeWithTimeout (sslsecur.c:428)
==2499== by 0x437ABC7: Curl_nss_connect (nss.c:1214)
==2499== by 0x4371181: Curl_ssl_connect (sslgen.c:185)
==2499== by 0x43504F8: Curl_http_connect (http.c:1792)
==2499== by 0x43579AD: Curl_protocol_connect (url.c:3056)
==2499== by 0x435CC7E: Curl_connect (url.c:4690)
==2499== by 0x4365A03: Curl_perform (transfer.c:2481)

________________________________
From: Kamil Dudka <kdudka_at_redhat.com>
To: Kevin Baughman <curb_pks_at_yahoo.com>
Cc: curl-library_at_cool.haxx.se
Sent: Tue, November 10, 2009 4:49:51 AM
Subject: Re: Seg fault in curl_perform stack trace

Hi Kevin,

On Mon November 9 2009 19:52:23 Kevin Baughman wrote:
> >>I am seeing a seg fault that seems to happen when errors
>
> occur. It is pretty reproducible so I got a valgrind trace of the
> issue occuring.

thanks for the report and the traces! I've seen exactly that backtrace in our
bugzilla:

https://bugzilla.redhat.com/show_bug.cgi?id=504257#c7

It was reported as a random error in a complex multi-threaded application. The
reporter hadn't had enough time to narrow it down to a minimal example. We
were not able to reproduce it, thus had to close it unresolved.

But you are saying it's "pretty reproducible". Could you please give us some
steps to reproduce it?

> >>curl 7.19.6 (i686-pc-linux-gnu) libcurl/7.19.6 NSS/3.12.4.5 zlib/1.2.3
> >> libidn/1.9 libssh2/1.0 Protocols: tftp ftp telnet dict ldap ldaps http
> >> file https ftps scp sftp Features: GSS-Negotiate IDN IPv6 Largefile SSL
> >> libz

Are you using the original versions of libcurl and nss from Fedora? If yes,
please provide their NVR. Or have you compiled libcurl and/or nss on your own?

Thanks in advance for your answers!

Kamil

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2009-11-10