Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak in CURLOPT_XOAUTH2_BEARER #8841

Closed
Tachi107 opened this issue May 13, 2022 · 7 comments
Closed

Memory leak in CURLOPT_XOAUTH2_BEARER #8841

Tachi107 opened this issue May 13, 2022 · 7 comments
Assignees

Comments

@Tachi107
Copy link
Contributor

I did this

Given the following code:

#include <curl/curl.h>

int main(void) {
  curl_global_init(CURL_GLOBAL_ALL);

  CURL* curl = curl_easy_init();

  curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_BEARER);
  curl_easy_setopt(curl, CURLOPT_XOAUTH2_BEARER, "c4e448d652a961fda0ab64f882c8c161d5985f805d45d80c9ddca108f8e2fde3");
  curl_easy_setopt(curl, CURLOPT_HTTPGET, 1L);
  curl_easy_setopt(curl, CURLOPT_URL, "https://andrea.pappacoda.it");

  for (int i = 0; i < 5; i++) {
    curl_easy_perform(curl);
  }

  curl_easy_cleanup(curl);

  curl_global_cleanup();
}

AddressSanitizer reports a memory leak:

$ cc -g -fsanitize=address main.c $(pkg-config --cflags --libs libcurl) -o asan && ./asan
=================================================================
==41730==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 260 byte(s) in 4 object(s) allocated from:
    #0 0x7f52f54d97a7 in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cpp:454
    #1 0x7f52f54423cd  (/lib/x86_64-linux-gnu/libcurl.so.4+0x673cd)

SUMMARY: AddressSanitizer: 260 byte(s) leaked in 4 allocation(s).

and valgrind does too:

$ cc -g main.c $(pkg-config --cflags --libs libcurl) -o valgrind && valgrind --leak-check=full ./valgrind
==41878== 
==41878== HEAP SUMMARY:
==41878==     in use at exit: 3,710 bytes in 12 blocks
==41878==   total heap usage: 32,937 allocs, 32,925 frees, 3,397,085 bytes allocated
==41878== 
==41878== 260 bytes in 4 blocks are definitely lost in loss record 5 of 8
==41878==    at 0x483F7B5: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==41878==    by 0x499331A: strdup (strdup.c:42)
==41878==    by 0x48CB3CD: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.8.0)
==41878==    by 0x48AB9B7: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.8.0)
==41878==    by 0x48AC81D: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.8.0)
==41878==    by 0x4884AE2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.8.0)
==41878==    by 0x1092FB: main (main.c:15)
==41878== 
==41878== LEAK SUMMARY:
==41878==    definitely lost: 260 bytes in 4 blocks
==41878==    indirectly lost: 0 bytes in 0 blocks
==41878==      possibly lost: 0 bytes in 0 blocks
==41878==    still reachable: 3,450 bytes in 8 blocks
==41878==         suppressed: 0 bytes in 0 blocks
==41878== Reachable blocks (those to which a pointer was found) are not shown.
==41878== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==41878== 
==41878== For lists of detected and suppressed errors, rerun with: -s
==41878== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

The size of the leak depends on the value of CURLOPT_XOAUTH2_BEARER; in this example it is 260 because the token is 65 characters long (including '\0') times 4.

I expected the following

Repeatedly performing HTTP requests with the same handle when a bearer token is set with the CURLOPT_XOAUTH2_BEARER option should not leak the token on each request.

curl/libcurl version

curl 7.83.0 (x86_64-pc-linux-gnu) libcurl/7.83.0 OpenSSL/1.1.1o zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.10.0 nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.11
Release-Date: 2022-04-27
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

operating system

Linux debian 5.17.0-1-amd64 #1 SMP PREEMPT Debian 5.17.3-1 (2022-04-18) x86_64 GNU/Linux

@bagder bagder self-assigned this May 13, 2022
@bagder
Copy link
Member

bagder commented May 13, 2022

I can reproduce with a new test case. PR pending.

bagder added a commit that referenced this issue May 13, 2022
Make use of conn_free() better and avoid duplicate code.

Reported-by: Andrea Pappacoda
Fixes #8841
Tachi107 added a commit to Tachi107/cloudflare-ddns that referenced this issue May 13, 2022
Also make sure that the supplied API token is 40 characters long.

For further information about the leak, see
curl/curl#8841
@bagder bagder closed this as completed in 06d1210 May 14, 2022
@rlorigro
Copy link

rlorigro commented Sep 25, 2023

Hi @bagder,

I am using AddressSanitizer on htslib, which also (as I understand it) relies on libcurl for fetching portions of remote GCS files with OAuth tokens, and I see a large leak from inside libcurl:

=================================================================
==28900==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 13996032 byte(s) in 2 object(s) allocated from:
    #0 0x7f5ceeab4c38 in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:164
    #1 0x7f5cef430401  (/lib/x86_64-linux-gnu/libcurl.so.4+0x1e401)

SUMMARY: AddressSanitizer: 13996032 byte(s) leaked in 2 allocation(s).

I am on ubuntu 22.04. Do you know if this is simply a version issue? Or somehow curl is misused here?

@dfandrich
Copy link
Contributor

dfandrich commented Sep 25, 2023 via email

@rlorigro
Copy link

Thanks for the quick response. Here is a slightly more verbose output from valgrind:

==33413== 
==33413== HEAP SUMMARY:
==33413==     in use at exit: 5,611,520 bytes in 1 blocks
==33413==   total heap usage: 451,403 allocs, 451,402 frees, 69,411,210 bytes allocated
==33413== 
==33413== 5,611,520 bytes in 1 blocks are possibly lost in loss record 1 of 1
==33413==    at 0x484DCD3: realloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==33413==    by 0x48C0401: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0)
==33413==    by 0x48EC5B7: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0)
==33413==    by 0x4906B00: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0)
==33413==    by 0x48E8193: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0)
==33413==    by 0x48EB3BD: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0)
==33413==    by 0x1AB591: wait_perform (hfile_libcurl.c:733)
==33413==    by 0x1ACCC7: restart_from_position (hfile_libcurl.c:1059)
==33413==    by 0x1AE063: libcurl_seek (hfile_libcurl.c:977)
==33413==    by 0x19F70D: hseek (hfile.c:469)
==33413==    by 0x18E630: bgzf_check_EOF_common (bgzf.c:1536)
==33413==    by 0x1900C7: bgzf_check_EOF (bgzf.c:2149)
==33413== 
==33413== LEAK SUMMARY:
==33413==    definitely lost: 0 bytes in 0 blocks
==33413==    indirectly lost: 0 bytes in 0 blocks
==33413==      possibly lost: 5,611,520 bytes in 1 blocks
==33413==    still reachable: 0 bytes in 0 blocks
==33413==         suppressed: 0 bytes in 0 blocks
==33413== 
==33413== For lists of detected and suppressed errors, rerun with: -s
==33413== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

Let me know if you need something more/different.

I have a test executable here which I am using to get the above output:
https://github.com/rlorigro/sv_merge/blob/main/src/test/test_hts_gcp.cpp

The project makes use of CMake to download and install the htslib dependency automatically, so with some basic libs, it should be possible to use the conventional build steps to get this running:

cd sv_merge
mkdir build
cd build
cmake ..
make -j [n_threads]

But I understand if this is too much to expect. I can try to take it up with the htslib authors if you think the issue is more likely on their end (or perhaps on my end).

Thanks

@bagder
Copy link
Member

bagder commented Sep 25, 2023

==33413== 5,611,520 bytes in 1 blocks are possibly lost in loss record 1 of 1
==33413== at 0x484DCD3: realloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)

5.6 megabytes in a single realloc done by libcurl? That looks highly unlikely.

My guess is on the write callback installed by the application.

@jmarshall
Copy link
Contributor

FYI @rlorigro's problem turns out to have been #10971.

@bagder
Copy link
Member

bagder commented Sep 28, 2023

aaaaah, thanks for the update

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

5 participants