It has generated tons of files in my machine， approximately 6GB as my service call this function every 15 seconds

libcurl does internet transfers for you. You decide what to download and how/if to store it. I can't see how curl makes any such decision for you.

libcurl does internet transfers for you. You decide what to download and how/if to store it. I can't see how curl makes any such decision for you.

I just test the same code in Adnimistrator process and Service process. It seems that in adnimistrator process, it will not write the file. In service process, it will write the file into C:\ProgramData\Microsoft\Crypto\RSA\S-1-5-18

I have searched on the internet that the folder C:\ProgramData\Microsoft\Crypto\RSA\S-1-5-18 is most likely relative with tls certificate authentication (use rsa key). I imagine that libcurl use windows crypto api and in system process, the api behaves different with in admin process

Does your libcurl build use Schannel? Is that the reason? So that file is created no matter what you do in your transfer?

It seems that in administrator process, it will not write the file. In service process, it will write the file

What is the difference between those processes?

According to links like this, this what some windows crypto API does.

Sounds like a Windows bug to me.

Does your libcurl build use Schannel? Is that the reason? So that file is created no matter what you do in your transfer?

It seems that in administrator process, it will not write the file. In service process, it will write the file

What is the difference between those processes?

I use this command line to build libcurl. Maybe it uses crypto api as default?
nmake /f Makefile.vc mode=dllVC=15 MACHINE=x64 DEBUG=no

The difference can be quite a lot and I don't know how it affects the crypto api.

Sounds like a Windows bug to me.

Well, I don't know if windows crypto provides apis like CryptoCreateKeyFile(it's a imaginary name) or CryptoDeleteKeyFile or something. Maybe the defualt implementation of libcurl forget to call CryptoDeleteKeyFile or something to delete the file? For now, I will test with Schannel instead.

Saving a file on disk behind our back is insane, even more so if it is crypto related. I cannot see how that could be a mistake in libcurl code. But I know very little about Schannel/windows crypto.

Well, if you guys find the exact reason causing the problem, please comment in this issue. It will help a lot.

Can anyone else running on Windows confirm/reproduce this?

I don't think we can do much more than to document this issue, and maybe recommend not using schannel if this is a problem for the application.

Try use Process Monitor to find out which process created this file.
Maybe some IDS/IPS software tried to examine the https server and saved the file.

This Windows documentation page on Key Storage and Retrieval seems to include details on how these files are made.

Mentioned by @sleevi on twitter

holy cow - Windows Crypto API
@ShadowZzj
https://social.msdn.microsoft.com/Forums/en-US/c75d4013-f732-4721-9c8e-fbb35e03b1c5/crypto-library-api?forum=windowssecurity

just a recommendation - switch to WSL2 and run your code inside a Container.
Seems like you do a lot of curling - thats why your C:\ProgramData\Microsoft\Crypto\ folder is stuffed.
Every new SSL/https connection will generate new data under the folder....

or you clean the folder in your code after x requests.

and yeah - windows problem....

@ShadowZzj any chance you can try out #9363 and see if it fixes the problem for you?

I apply the commit: schannel: when importing PFX, disable key persistence. curl_easy_perform returns CURLE_SSL_CONNECT_ERROR and the error message is schannel: next InitializeSecurityContext failed: SEC_E_INTERNAL_ERROR (0x80090304).

If I don't apply the commit, the code works. The reason I imagine is that some steps in tls handshake needs to read the key, if the key is not written to the local file, these steps fail?

Thanks for testing this! I validated it with curl as a stand-alone tool, and will try on Monday to reproduce this issue with libcurl.

What specific version of Windows are you on?

@bagder since you’re so close to the next release of curl, it may be a good idea to revert this commit in case it causes a broad issue on Windows until we can get to the root cause.

@ShadowZzj that’s a fair observation! It’s supposed to keep the key resident in memory, but if we reference it later by its ID instead of its crypto handle we might be running into some trouble.

@DHowett Thanks for the quick reply. The windows version is Windows 10 Professional 21H1 (19043.1889).

Reopened due to the issues

thanks @DHowett, it is now reverted in aec8d30 while researching. I say we aim for landing any fix post this release.

I'm having trouble reproducing this with the snippet that you included. On Windows 10 (17763, not 19043) and Windows 11, a simple client certificate exchange with client.badssl.com (using their provided client certificate) using this code reports success.

#include <curl/curl.h>
int main(int argc, char *argv[])
{
    CURL *curl = curl_easy_init();
    CURLcode res;
    int result = 0;
    char errBuf[CURL_ERROR_SIZE]{0};
    char *version = curl_version();

    struct curl_slist *http_headers = NULL;
    char* ret                       = NULL;
    http_headers = curl_slist_append(http_headers, "Accept: application/json");
    http_headers = curl_slist_append(http_headers, "Content-Type: application/json");
    http_headers = curl_slist_append(http_headers, "charsets: utf-8");
    curl_easy_setopt(curl, CURLOPT_HTTPHEADER, http_headers);

    curl_easy_setopt(curl, CURLOPT_URL, "https://client.badssl.com");
    curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
    curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errBuf);

    curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 60);
    curl_easy_setopt(curl, CURLOPT_TIMEOUT, 60);
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, true);
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, true);
    curl_easy_setopt(curl, CURLOPT_SSL_OPTIONS, CURLSSLOPT_AUTO_CLIENT_CERT);
    curl_easy_setopt(curl, CURLOPT_SSLCERTTYPE, "P12");
    curl_easy_setopt(curl, CURLOPT_SSLCERT, "badssl.com-client.p12");
    curl_easy_setopt(curl, CURLOPT_KEYPASSWD, "badssl.com");
    res = curl_easy_perform(curl);
    curl_slist_free_all(http_headers);
    curl_easy_cleanup(curl);

    fprintf(stderr, "RES: %d\n", res);
    fprintf(stderr, "RET: %s\n", ret);
    fprintf(stderr, "ERR: %s\n", errBuf);
    return result;	
}

Is there anything about your specific use case that I may be missing?

I'll try a build of 19041/19043 in a little bit.

I use the code you provide and It shows the same error. Besides, I can't run the libcurl in master. It has another error. I use the libcurl tag 7_74_0(commit hash e052859) which I have been using 1 year ago. And I only change the code in schannel.c to the code below(

The command I use to compile libcurl is "nmake /f Makefile.vc mode=dll VC=15 MACHINE=x64 DEBUG=yes RTLIBCFG=static"

int main(int argc, char* argv[])
{
    CURL* curl = curl_easy_init();
    CURLcode res;
    int result = 0;
    char errBuf[CURL_ERROR_SIZE]{ 0 };
    char* version = curl_version();

    struct curl_slist* http_headers = NULL;
    char* ret = NULL;
    std::string retString = "";
    http_headers = curl_slist_append(http_headers, "Accept: application/json");
    http_headers = curl_slist_append(http_headers, "Content-Type: application/json");
    http_headers = curl_slist_append(http_headers, "charsets: utf-8");
    curl_easy_setopt(curl, CURLOPT_HTTPHEADER, http_headers);

    curl_easy_setopt(curl, CURLOPT_URL, "https://testurl.com");
    curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
    curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errBuf);
    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "{\"test\":\"123\" }");
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteData);
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, &retString);
    curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 60);
    curl_easy_setopt(curl, CURLOPT_TIMEOUT, 60);
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, true);
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, true);
    curl_easy_setopt(curl, CURLOPT_SSLCERTTYPE, "P12");
    curl_easy_setopt(curl, CURLOPT_SSLCERT, R"(C:\personal.p12)");
    curl_easy_setopt(curl, CURLOPT_KEYPASSWD, "da773a19e2f24037");
    res = curl_easy_perform(curl);
    curl_slist_free_all(http_headers);
    curl_easy_cleanup(curl);

    fprintf(stderr, "RES: %d\n", res);
    fprintf(stderr, "RET: %s\n", ret);
    fprintf(stderr, "ERR: %s\n", errBuf);
    return result;
}

Thanks for the additional info! I think I've figured this out.

I couldn't reproduce your issue because I had checked out the wrong branch. 😁

In addition to the original fix, we need to keep the certificate store handle open until we're done with the certificates inside it.

Can you test this patch against the version of curl you're using?

I have tested the patch. It works! No more files in C:\ProgramData\Microsoft\Crypto\RSA\S-1-5-18. And the curl_easy_perform returns good result. It seems the problem has been solved. I think then you should check if there is memory leak or something. Maybe some execution path will not close the cert store eventually.

ps: pr is required as soon. Thanks for your help!