Re: Help using libcurl with HTTP proxy on Android device

From: Henrik Holst via curl-library <curl-library_at_lists.haxx.se>
Date: Wed, 12 Apr 2023 03:16:37 +0200

the main issue for libcurl is that openssl (of which boringssl is a fork)
only allows CA certificates in PEM format to be loaded, other types of
certificates can be in PEM or DER but CA must be in PEM. So it's really
strange (and dumb to be frank) of Google to force the CA certs to be
converted to DER when imported in Android, what makes this even more
strange is that AFAIK Chrome uses openssl on Android as well so they must
have code on Chrome to convert from DER to PEM in memory and then inserting
the cert to openssl.

For the callback I mentioned before it is this one:
https://curl.se/libcurl/c/CURLOPT_SSL_CTX_FUNCTION.html the example there
looks to manually load a cert so something similar (but using the
convertion routines of openssl) should be possible to use. I have never
used any of this though so unfortunately have no experience to share here.

/HH

Den ons 12 apr. 2023 kl 03:03 skrev David Castillo <casvel.d_at_gmail.com>:

> > There used to be at least two locations used on Android for
> > certificates. Maybe OpenSSL is only using one of them?
>
> Yes, user-installed certificates are stored in the
> "/data/misc/user/0/cacerts-added" directory while system certificates are
> stored in "/system/etc/security/cacerts" directory. That's why when I
> detect there's a proxy, I update the "CURLOPT_CAPATH" option to the
> user-installed CA cert directory. But the certificates in the
> user-installed directory are installed as DER format and libcurl fails to
> open them with this error: `BoringSSL: error:0900006e:PEM
> routines:OPENSSL_internal:NO_START_LINE`
>
> > well if the plan is to always use this particular proxy then you can
> always bundle the root cert and point libcurl to it as a local file.
>
> Unfortunately the plan is to not always use this particular proxy. The
> idea is to allow users to set any proxy they want, for example they can set
> Charles, Fiddler, Proxyman or any other proxy.
>
> > AFAIK the ssl callback functions of libcurl could be used to load the
> cert in DER format and supply it via memory buffer to the ssl library but
> that requires knowledge of the exact ssl library used and not all of them
> support the ssl callback functions AFAIK
>
> Do you know where I can look for documentation or anything to see how to
> do this?
>
> Although I have the feeling it shouldn't be that complicated to support
> SSL proxy on Android with libcurl. I have the feeling that I might be doing
> something wrong or missing something. I found that this option exists
> CURLOPT_PROXY_CAPATH (https://curl.se/libcurl/c/CURLOPT_PROXY_CAPATH.html)
> and I tried to set it, but got again this error: `SSL certificate problem:
> self signed certificate in certificate chain`
>
> Then I saw that this proxy CA path option is only used for HTTPS proxies
> and I found this CURLOPT_PROXYTYPE option (
> https://curl.se/libcurl/c/CURLOPT_PROXYTYPE.html) and saw that HTTP proxy
> is set by default, so I tried to set it to CURLPROXY_HTTPS and now I got
> this error: `BoringSSL: error:100000f7:SSL
> routines:OPENSSL_internal:WRONG_VERSION_NUMBER`
>
> Could it be that my version of libcurl doesn't support HTTPS proxies? This
> is the configuration I have when building libcurl
>
> Host setup: aarch64-unknown-linux-android
>
> Install prefix: /build/curl/arm64-v8a
>
> Compiler: /
> ndk/22.0.7026061/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android23-clang
>
> CFLAGS: -Qunused-arguments -Wno-pointer-bool-conversion -O2
>
> CPPFLAGS: -isystem /build/zlib/arm64-v8a/include -isystem
> /build/boringssl/arm64-v8a/include
>
> LDFLAGS: -/build/boringssl/arm64-v8a/lib
> /build/zlib/arm64-v8a/lib -L/build/zlib/arm64-v8a/lib
> -L/build/boringssl/arm64-v8a/lib
>
> LIBS: -lssl -lcrypto -lz -lssl -lcrypto -lc++ -lz
>
>
> curl version: 7.87.1-DEV
>
> SSL: enabled (BoringSSL)
>
> SSH: no (--with-{libssh,libssh2})
>
> zlib: enabled
>
> brotli: no (--with-brotli)
>
> zstd: no (--with-zstd)
>
> GSS-API: no (--with-gssapi)
>
> GSASL: no (libgsasl not found)
>
> TLS-SRP: no (--enable-tls-srp)
>
> resolver: POSIX threaded
>
> IPv6: enabled
>
> Unix sockets: enabled
>
> IDN: no (--with-{libidn2,winidn})
>
> Build libcurl: Shared=yes, Static=yes
>
> Built-in manual: no (--enable-manual)
>
> --libcurl option: enabled (--disable-libcurl-option)
>
> Verbose errors: no
>
> Code coverage: disabled
>
> SSPI: no (--enable-sspi)
>
> ca cert bundle: no
>
> ca cert path: /system/etc/security/cacerts (warning: certs not
> found)
>
> ca fallback: no
>
> LDAP: no (--enable-ldap / --with-ldap-lib /
> --with-lber-lib)
>
> LDAPS: no (--enable-ldaps)
>
> RTSP: enabled
>
> RTMP: no (--with-librtmp)
>
> PSL: no (libpsl not found)
>
> Alt-svc: enabled (--disable-alt-svc)
>
> Headers API: enabled (--disable-headers-api)
>
> HSTS: enabled (--disable-hsts)
>
> HTTP1: enabled (internal)
>
> HTTP2: no (--with-nghttp2, --with-hyper)
>
> HTTP3: no (--with-ngtcp2, --with-quiche --with-msh3)
>
> ECH: no (--enable-ech)
>
> WebSockets: no (--enable-websockets)
>
> Protocols: DICT FILE FTP FTPS GOPHER GOPHERS HTTP HTTPS IMAP
> IMAPS MQTT POP3 POP3S RTSP SMB SMBS SMTP SMTPS TELNET TFTP
>
> Features: AsynchDNS HSTS HTTPS-proxy IPv6 Largefile NTLM
> NTLM_WB SSL UnixSockets alt-svc libz threadsafe
>
> On Tue, Apr 11, 2023 at 2:37 PM Henrik Holst <henrik.holst_at_millistream.com>
> wrote:
>
>> well if the plan is to always use this particular proxy then you can
>> always bundle the root cert and point libcurl to it as a local file. AFAIK
>> the ssl callback functions of libcurl could be used to load the cert in DER
>> format and supply it via memory buffer to the ssl library but that requires
>> knowledge of the exact ssl library used and not all of them support the ssl
>> callback functions AFAIK.
>>
>> /HH
>>
>> Den tis 11 apr. 2023 kl 23:09 skrev David Castillo <casvel.d_at_gmail.com>:
>>
>>> > But did you install it as a new root certificate or as a client
>>> certificate on the android device?
>>>
>>> As far as I understand, I installed it as a new root certificate, but
>>> I'm not sure. I went to "Settings -> Passwords & security -> Privacy ->
>>> Encryption and Credentials -> Install a certificate -> CA certificate".
>>> Then if I go to "Trusted Credentials" on the device, I can see the Charles
>>> certificate in the "User" tab
>>>
>>> > Can you access other https locations? If so then the other root ca:s
>>> works on the device for some reason
>>>
>>> Yeah, I can access HTTPS locations without the proxy. I think the system
>>> certificates that are in the "/system/etc/security/cacerts" directory
>>> don't have any problem because those are PEM format (I checked that by
>>> grabbing one of those certificates and doing "openssl x509 -in
>>> <certificate> -inform PEM -text -noout" on my computer).
>>>
>>> The problem seems to be that when Android installs the new certificate,
>>> it converts it to DER format for some reason, which libcurl can't handle. I
>>> want to know if there's a way to handle this using libcurl. Maybe there's a
>>> way to convert the certificate before libcurl tries to verify it?
>>>
>>

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2023-04-12