Sv: Re: Curl callback question, porting from OpenSSL 1.x to 3.x and from 32bit plain to 64bit UTF16

From: Anders Gustafsson via curl-library <curl-library_at_lists.haxx.se>
Date: Thu, 21 Aug 2025 09:47:35 +0300

OK. Point taken 😀

Yes. I do make assumptions based on how this app is used and yes it is C++/MFC on windows going from an old
version with no specified coding to UTF.

The certificate data in this case is "regular" ie a PEM payload that is not UTF encoded, just:

-----BEGIN CERTIFICATE-----
xxxxxxx
-----END CERTIFICATE-----

Ie one byte per character. Filenames OTOH might be a different kettle of fish. I probably should include a
testcase where the path has non-ascii characters.

And yes, what I wrote then was completely clear to me, but I realise when rereading that it was probably not,
so, let me elaborate:

The original code on Windows 32-bit no unicode used the CURLOPT_SSL_CTX_FUNCTION

I replaced that with the function in the documentation, but that apparently did not add the key to the
context, just the certificate so I needed to add code to handle the certificate

The string passed to CURLOPT_SSL_CTX_DATA is a CStringA so it is OK to convert to char, especially assuming
that the data is just PEM

The example code in https://curl.se/libcurl/c/CURLOPT_SSL_CTX_DATA.html adds the certificate to a store in
the context, what my code does is to take the certificate and key, again as CStringA then BIO_new_mem_buf()
the do PEM_Read_bio_x590 then SSL_CTX_use_certificate, then do the same for the key, SSL_CTX_use_PrivateKey.
Ie it sets the certificate and key directly without going through a store. I am not sure why my way works, but
not the store way. I guess the "store" concept was added to OpenSSL along the line and I guess I should read
up on that.

Yes. String handling in Windows is weird. I sort of understand the performance aspect of using UTF-16, but it
makes other things harder. I guess another option would be to use std::string all the way.


> (OTOH one could just assume that
> all [relevant] ACP encodings are ASCII subset, thus
> simply NOT do transcoding since it then ought to be
> ASCII-compliant content already anyway).

Which is pretty much what I do. All certificate data is assumed to be PEM and coded as ASCII with one byte
per character.

Thanks for your reply and your pointers.

-- 
Med vänlig hälsning
Anders Gustafsson, ingenjör
anders.gustafsson_at_pedago.fi  |  Support +358 18 12060  |  Direkt +358 9 315 45 121  |  Mobil +358 40506 7099
Pedago interaktiv ab, Nygatan 7 B , AX-22100 MARIEHAMN, ÅLAND, FINLAND
>>> Andreas Mohr <andi_at_lisas.de> 2025-08-20 17:44 >>>
Hi,
disclaimer:
quite experienced in certain areas, yet not too much in others (CURL).
TL;DR:
thus discussing potential string encoding issues only.
On Wed, Aug 20, 2025 at 03:58:14PM +0300, Anders Gustafsson via curl-library wrote:
> So, yes this is windows 😀 libcurl/8.15.0-DEV OpenSSL/3.5.2 zlib/1.3.1
> 
> I had some issues and I just want to check whether I am going about this the right way. The function calls
an
> API where the client certificate is used to authenticate the caller so in the original version I used the
> sslctx_function(). To complicate matters does my app support PEM certificates and keys in two different
ways:
> 1. As files (Say on a removable secure media) and 2. As strings in the database for ease of use.
> 
> The first way (filenames) worked right away, ie:
> 
> 			m_Certificate.Trim();
> 			if (m_Certificate.IsEmpty())
> 				curl_easy_setopt(curl, CURLOPT_SSLCERT, m_CertificateFile.GetString());
Bleeep - ATLMFC CStringT::GetString() encountered.
This might thus be
dirty "encoding shortcutting" here
(simply invoking .GetString() to
"quickly" "get at" some "char"-compatibly-typed - hah! - input, rather than
doing active transcoding to
the actually *correct* encoding spec of
some char-typed handling).
Thus, consulting this one:
> Where m_Certificate and m_Key and regular (char) strings with the PEM coded data.
What would "regular" mean?
Considering that
CString errfilename;
with
CT2A(errfilename),
one would think that this is
a CString[T] with UNICODE config setting (put differently, CStringW)
environment, however
since you said "regular (char) strings", I am assuming that
you are on !UNICODE config (i.e., CStringA).
https://manpages.ubuntu.com/manpages/kinetic/man3/CURLOPT_SSLCERT.3.html 
Yeah nice - that page does not specify at all which
encoding char *cert (a filesystem item argument!! - which could be
containing all the smileys available in this universe, with
only a bit of luck...) is expected to have.
Thus on Windows one would tend to
assume ACP crap - which of course would mean that it is
Unicode-compliance-broken (since: neither UTF16 nor UTF8 nor UTF7 nor
UTF-EBCDIC or whatever ;-)).
> 	errno_t fileerr = fopen_s(&errfile, CT2A(errfilename), "w+, ccs=UTF-8");
Unicode-compliance-broken filesystem item handling!
CT2A(errfilename) will be
wide-typed to CP_ACP transition (in UNICODE config setting), and
"nothing" *) (in non-UNICODE).
*) BTW *HORRIBLE* atlconv.h comment "// Code page doesn't matter" atlconv.h transcoding protocol breakage!!!
Yeah, as if that would be the case for
e.g. CP_ACP to CP_UTF8 transcoding, which *is* a valid transcoding use case...)
(think of
CT2CA(..., CP_UTF8)
protocol behaviour **DIFFERENCE**)
Thus, your errfilename possibly is ACP (CP_ACP, GetACP()) content,
which would be
"compatible" since
fopen_s() API is equally ACP-specced on Windows (yuck).
...but: then it would be
Unicode-compliance-broken (due to
being ACP crap, rather than
e.g. UTF8 as usually on Linux).
Since fopen_s() should have an overload for wide-typed input I'd think,
the way to go would at least be
CT2W(errfilename) - thereby
properly preserving Unicode-compliant (since wide-typed!) encoding (when
on UNICODE config setting - and ACP crap on !UNICODE).
Or, better do utf8everywhere.org (i.e., std::string[-means-utf8] -
to have ensured that
**every** string traffic anywhere is Unicode-compliant), and thus do
std::string errfilename = "myUtf8InputStuffStringFromSomewhere"; // e.g. std::filesystem **) API
fopen_s(... CA2W(errfilename, CP_UTF8) ...);
**) rather *horribly* Unicode-compliance-broken (on Windows!) - I digress...
"<filesystem>: prevent filesystem::path dangerous conversions to/from default code page encoding"
  https://github.com/microsoft/STL/issues/909 
> In the second scenario, PEM in database, I had some problems and I just wanted to check that the code I
came
> up with is sane. Ie the authentication will not happen unless I have both certificate and key, so:
> 
> 			if (!m_Certificate.IsEmpty())
> 			{
> 				curl_easy_setopt(curl, CURLOPT_SSL_CTX_FUNCTION, sslctx_function);
> 				curl_easy_setopt(curl, CURLOPT_SSL_CTX_DATA, m_Certificate.GetString());
WARNING CORRUPTION: CURLOPT_SSL_CTX_DATA has a void*-typed ptr arg, thus
both .GetString() CString[T] is accepted, *silently*).
IOW, once on UNICODE config setting it would be
*broken*.
So, questionable encoding stuff again.
According to
https://curl.se/libcurl/c/CURLOPT_SSL_CTX_DATA.html 
  "char *mypem = /* CA cert in PEM format"
it seems to *appear* that
"PEM format" means some plain ASCII-only payload.
Now to be maximally precise one could do
const UINT nCP_PEM = 20127 /*CP_ASCII*/ /* these clearly are [to be!] all ASCII-only, right!!!? */;
std::string strCertificate = CW2A(CA2W(m_Certificate), nCP_PEM);
(this transition expects that m_Certificate has system ACP content, of course)
(OTOH one could just assume that
all [relevant] ACP encodings are ASCII subset, thus
simply NOT do transcoding since it then ought to be
ASCII-compliant content already anyway).
> Then, below, which seems to work OK. I first used the example here:
> https://curl.se/libcurl/c/CURLOPT_SSL_CTX_FUNCTION.html 
> but that one did not fix my key for me. Yes, this code still leaves allocated memory in case of errors.
"fix my key" - that wording might hint at
encoding issues.
But perhaps we're talking about
a plain CURL certificate config issue only after all.
I could not precisely identify (thus: discuss) particular ***) issues in
your handling, but I'd hope that
this will give you some ideas (*if* it is an encoding issue).
***) well, except for the broken fopen_s() filesystem item handling...
Greetings
Andreas Mohr
-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2025-08-21