Following curl://up 2019 I've finally started the task of making the NTLM code TLS backend agnostic. As this is a fairly chunky piece of work I have split it up into the following sub tasks:

• Move the TLS specific MD4 code out of curl_ntlm_core.c
• Move the TLS specific MD5 code out of curl_ntlm_core.c
• Move the TLS specific DES implementation out of curl_ntlm_core.c
• Add our own DES implementation

As you will appreciate, this will probably (more than likely) be spanned across several releases - especially as it has taken me 4 years and several discussion with folk on the curl://team to actually start this :-P

In summary this patch set centralises the MD4 code for the TLS libraries into md4.c but it also:

• Adds support for MD4, by calling our local implementation, when OpenSSL does not support it
• Adds support for MD4 in the NTLM code when no crypto libraries are used (No effect yet, given that we still rely on these libraries for MD5 and DES)
• Allows the NTLM type-3 response message to always include the extended (and more secure) NT response as opposed to just the LM response.
• Simplifies the MD4 code in curl_ntlm_core.cpp as it replaces the complex pre-processor block with one line of code.
• Maintains similar code between md4.c and md5.c.

The downsides:

Two TLS backends (SecureTransport and mbed TLS) support a single line function call when creating the MD4 hash. As the Curl_md4it() function implements an OpenSSL style API (calling multiple functions) we have to store the data to be encrypted and as such store that in a temporary buffer. The buffer is malloc'ed and as such will be slower for these two backend libraries.

Concerns:

md4.c and md5.c are a little different, from my point of view, in two areas:

• The SecureTransport pre-processor directives are different. md4.c (now) uses the USE_SECTRANSP directive, as that is what was used in the NTLM code which has now moved to md4.c. md5.c, on the other hand, uses a few Mac and IOS specific directives. @nickzman
• The MD5 code, when using OpenSSL, has an Amiga OS specific pre-processor directive present (USE_AMISSL) from AmiSSL support #3677. Do we need one in the MD4 code as well? The original NTLM code didn't have this so I'm not sure. @chris-y

Notes:

Whilst I have compiled this on Windows using both OpenSSL and Schannel I cannot compile it for the other TLS, non OpenSSL style, backends (GNU TLS, mbed, SecureTransport, NSS, OS/400). As such I am relying on the automated build system and tests, as well as AppVeyor and Travis CI to let me know if there are any problems. I will of course update this patch set if and when they fail.