All 3 native methods called from getsubject returns NULL when CA Cert is invalid. Therefore the proper error code is CURLE_SSL_CACERT.
- SecCertificateCopySubjectSummary: https://developer.apple.com/documentation/security/1396041-seccertificatecopysubjectsummary?language=objc
- SecCertificateCopyLongDescription: https://developer.apple.com/documentation/security/1396088-seccertificatecopylongdescriptio?language=objc
- SecCertificateCopyCommonName: https://developer.apple.com/documentation/security/1394814-seccertificatecopycommonname?language=objc

… certificate store

…ficate

The reason we want to do this are two fold.
1. The methods SecTrustSetAnchorCertificates, SecTrustSetAnchorCertificatesOnly, and SecTrustEvaluate return error codes from the Security Framework rather than Secure Transport according the Apple's documentation https://developer.apple.com/documentation/security/1396098-sectrustsetanchorcertificates?language=objc. The previous implementation maps return codes from Secure Transport which is wrong.
2. We would like to follow a similar approach as pkp_pin_peer_pubkey for consistency. In that method, error codes returned from intermediate calls are grouped under a single CURLE_SSL_PINNEDPUBKEYNOTMATCH error code.

Since the CopyCertSubject function is called during both client and server cert verification, we will need to translate the error code returned during client cert verification to maintain consistency with other tls backends.

Both openssl and schannel tls backends return this error code when they encounter error setting the cipher suite. We should do the same for darwinssl for consistency. Note that this error code is not used to represent failures during the TLS negotiation process with the server. It is used purely used on the client side.

Since invalid path to the client certificate is treated as an error for the certificate itself on other tls backends, we should do the same for schannel.

Although this restriction does not seem to exist for other tls backends, we should be consistent with the error code returned when handling the CA bundle imported from the local certificate store.

The schannel version of the above method does not return SEC_E_WRONG_PRINCIPAL error code according to their documentation https://msdn.microsoft.com/en-us/library/windows/desktop/aa374716(v=vs.85).aspx. I decided to have the full list of possible error codes for easier reference.

…tion error

The IntitializeSecurityContext method returns SEC_E_WRONG_PRINCIPAL on failure to verify the host name of server's certificate. We should return the same error code as other tls backends, i.e. CURLE_SSL_CACERT. In addition, memory error should be mapped to CURLE_OUT_OF_MEMORY. I also decided to use the full list of return codes here for clarity and future reference.

…host

In openssl and darwinssl implementations of *_connect_step3 methods, CURLE_PEER_FAILED_VERIFICATION is returned when non-memory related error is encountered during any intermediate steps of the host name verification procedure. We should do the same here for consistency.

…suer

Failure to extract the issuer name from the server certificate should return a more specific error code like on other TLS backends.

The previous implementation is incorrect on two levels.
1. The return type of the function is CURLcode so we should adhere to that instead of using int literals.
2. 0 is actually mapped to CURLE_OK which means we are wrongly treating a memory error as no error.

SSLHandle may return a wide variety of Secure Transport error codes. We are grouping those related to server certificate validation under CURLE_SSL_CACERT. Note that most of these additions should have no effect as they occur way less often than errSSLXCertChainInvalid. In addition, OCSP response error is also mapped to the appropriate code that's consistent with openssl.

I decided to add the full list of Secure Transport error codes here for clarity and future reference. Currently, they are all mapped to CURLE_SSL_CONNECT_ERROR like in the previous implementation. If desired, we can have more specific codes for each error category, such as TLS negotiation, encryption error, etc. With the complete list of return codes, we can also understand future bug reports better since they contain the detailed error message.

The translation of error codes is now done inline using a switch case.

…ertificate

CURLE_PEER_FAILED_VERIFICATION makes more sense because Curl_parseX509 does not allocate memory internally as its first argument is a pointer to the certificate structure. The same error code is also returned by Curl_verifyhost when its call to Curl_parseX509 fails so the change makes error handling more consistent.

…issues

As discussed in the PR, CURLE_PEER_FAILED_VERIFICATION is a more general error code for server certificate issues than CURLE_SSL_CACERT. The latter should be restricted to only those problems with certificate authority's cert.