On Tue, 2016-09-27 at 19:54 +0100, David Woodhouse wrote:
>
> I was trying to be a little more helpful and use pkcs11-spy which
> actually lets you watch all the calls into a PKCS#11 provider module
> (there are software ones like SoftHSM which you can use for testing/
> learning). But right now I can't make curl work at all with the engine;
> something odd is going on. So I'll leave you reading the above, while I
> try to fix that up :)

OK, from our point of view, the above clusterf*ck is addressed by 
https://github.com/curl/curl/pull/1042

I believe I technically have the permissions to push that myself now,
but I'll let it sit for review for at least a little while before I do
that; even though I don't really expect anyone else to get involved in
ENGINE esoterica.

So I can give you that recipe, to help understand what's going on.

You're going to be providing a PKCS#11 module — a shared library with
functions like C_Encrypt() and C_Sign() that are going to translate
into those IOCTLs to talk to the actual hardware. And then there's a
bit more boilerplate and housekeeping, where it'll call into your
module to discover what capabilities it has, what keys and other
objects it contains (yours will probably only report a single key),
etc.

For now, let's use SoftHSM as the provider module. That's a purely
software token, which provides the same PKCS#11 API you're going to be
providing from your own module.

Again I'm assuming Fedora 24, because this stuff mostly Just Works
there. You'll want the latest curl with that fix above, built against
OpenSSL though. (Actually you can also build against GnuTLS. It's only
NSS, which unfortunately curl is built against in Fedora, which is
broken.)

First, let's configure SoftHSM's storage and import your key (which I
shall assume exists in key.pem and cert.pem)...

 $ mkdir /tmp/softhsm
 $ echo "directories.tokendir = /tmp/softhsm" > softhsm2.conf
 $ export SOFTHSM2_CONF=`pwd`/softhsm2.conf
 $ softhsm2-util --init-token --slot 0 --label test --pin 1234 --so-pin 12345678
The token has been initialized.
 $ softhsm2-util --import key.pem --slot 0 --label mykey --id 01 --pin 1234
The key pair has been imported.

So now you have a PKCS#11 token with a single key in it, which is
basically what you need to provide for your own hardware. The point of
this email is to let you see how to use it, and how to peek at the
calls that get made into the PKCS#11 provider module.

Firstly, let's take a look at the key in your token:

 $ p11tool --list-privkeys 'pkcs11:token=test;pin-value=1234' --login
Object 0:
        URL: pkcs11:model=SoftHSM%20v2;manufacturer=SoftHSM%20project;serial=e68be48e79fcab65;token=test;id=%01;object=mykey;type=private
        Type: Private key
        Label: mykey
        Flags: CKA_PRIVATE; CKA_SENSITIVE;
        ID: 01

Now, there is magic happening behind the scenes here... how did p11tool
*know* that it should load the SoftHSM provider module? On Linux
systems, this is the responsibility of p11-kit. The softhsm package
installs a file in /usr/share/p11-kit/modules/softhsm.module which says
that the libsofthsm2.so library should be loaded and used as a PKCS#11
provider. 

Edit that softhsm.module file (as root) and add the line:
 log-calls: yes

Now run the above 'p11tool --list-privkeys...' command again. You'll
see a lot more output this time, as it shows the inputs and outputs of
every function call into the SoftHSM PKCS#11 module — starting with the
C_Initialize() call, then C_GetInfo() to find out about the library
module, then C_GetSlotList() to find the "slots" therein, and then
iterating over the slots with C_GetTokenInfo() to find out which slots
have a token in...

I already gave you a reference to the PKCS#11 spec. Now you have a
worked example of how a PKCS#11 module gets used in practice, to help
you understand and implement it.

The above was just listing the key, but you can actually use it with
(your own non-NSS build of) curl as follows...

$ src/curl --engine pkcs11 -E cert.pem --key 'pkcs11:token=test;object=mykey;pin-value=1234' --key-type ENG https://localhost:4443/ -k
C_Initialize
  IN: pInitArgs = NULL
C_Initialize = CKR_OK
C_GetSlotList
  IN: tokenPresent = CK_FALSE
  IN: pulCount = 0x7FFC829AE6F0 = 2
 OUT: pSlotList = (2) NO-VALUES
C_GetSlotList = CKR_OK
C_GetSlotList
  IN: tokenPresent = CK_FALSE
  IN: pulCount = 0x7FFC829AE6F0 = 2
 OUT: pSlotList = (2) [ SL0, SL1 ]
C_GetSlotList = CKR_OK
C_GetSlotInfo
  IN: slotID = SL0
 OUT: pInfo = {
        slotDescription: "SoftHSM slot 0"
        manufacturerID: "SoftHSM project"
        flags: 1 = CKF_TOKEN_PRESENT
        hardwareVersion: 2.1
        firmwareVersion: 2.1
      }
C_GetSlotInfo = CKR_OK
C_GetTokenInfo
  IN: slotID = SL0
 OUT: pInfo = {
        label: "test"
        manufacturerID: "SoftHSM project"
        model: "SoftHSM v2"
        serialNumber: "e68be48e79fcab65"
        flags: 1069 = CKF_RNG | CKF_LOGIN_REQUIRED | CKF_USER_PIN_INITIALIZED | CKF_RESTORE_KEY_NOT_NEEDED | CKF_TOKEN_INITIALIZED
        ulMaxSessionCount: CK_UNAVAILABLE_INFORMATION
        ulSessionCount: 18446744073709551615
        ulMaxRwSessionCount: CK_UNAVAILABLE_INFORMATION
        ulRwSessionCount: 18446744073709551615
        ulMaxPinLen: 255
        ulMinPinLen: 4
        ulTotalPublicMemory: CK_UNAVAILABLE_INFORMATION
        ulFreePublicMemory: CK_UNAVAILABLE_INFORMATION
        ulTotalPrivateMemory: CK_UNAVAILABLE_INFORMATION
        ulFreePrivateMemory: CK_UNAVAILABLE_INFORMATION
        ulFreePrivateMemory: CK_UNAVAILABLE_INFORMATION
        hardwareVersion: 2.1
        firmwareVersion: 2.1
        utcTime:
      }
C_GetTokenInfo = CKR_OK
C_GetSlotInfo
  IN: slotID = SL1
 OUT: pInfo = {
        slotDescription: "SoftHSM slot 1"
        manufacturerID: "SoftHSM project"
        flags: 1 = CKF_TOKEN_PRESENT
        hardwareVersion: 2.1
        firmwareVersion: 2.1
      }
C_GetSlotInfo = CKR_OK
C_GetTokenInfo
  IN: slotID = SL1
 OUT: pInfo = {
        label: ""
        manufacturerID: "SoftHSM project"
        model: "SoftHSM v2"
        serialNumber: ""
        flags: 12582949 = CKF_RNG | CKF_LOGIN_REQUIRED | CKF_RESTORE_KEY_NOT_NEEDED | CKF_SO_PIN_LOCKED | CKF_SO_PIN_TO_BE_CHANGED
        ulMaxSessionCount: CK_UNAVAILABLE_INFORMATION
        ulSessionCount: 18446744073709551615
        ulMaxRwSessionCount: CK_UNAVAILABLE_INFORMATION
        ulRwSessionCount: 18446744073709551615
        ulMaxPinLen: 255
        ulMinPinLen: 4
        ulTotalPublicMemory: CK_UNAVAILABLE_INFORMATION
        ulFreePublicMemory: CK_UNAVAILABLE_INFORMATION
        ulTotalPrivateMemory: CK_UNAVAILABLE_INFORMATION
        ulFreePrivateMemory: CK_UNAVAILABLE_INFORMATION
        ulFreePrivateMemory: CK_UNAVAILABLE_INFORMATION
        hardwareVersion: 2.1
        firmwareVersion: 2.1
        utcTime:
      }
C_GetTokenInfo = CKR_OK
C_OpenSession
  IN: slotID = SL0
  IN: flags = 4 = CKF_SERIAL_SESSION
  IN: pApplication = NULL
  IN: Notify = NULL
 OUT: phSession = 0x014E07E8 = S1
C_OpenSession = CKR_OK
C_FindObjectsInit
  IN: hSession = S1
  IN: pTemplate = (1) [ CKA_CLASS ]
C_FindObjectsInit = CKR_OK
C_FindObjects
  IN: hSession = S1
  IN: max_object_count = 1
 OUT: object = (0) [ ]
C_FindObjects = CKR_OK
C_FindObjectsFinal
  IN: hSession = S1
C_FindObjectsFinal = CKR_OK
C_Login
  IN: hSession = S1
  IN: userType = CKU_USER
  IN: pPin = (4) "1234"
C_Login = CKR_OK
C_FindObjectsInit
  IN: hSession = S1
  IN: pTemplate = (1) [ CKA_CLASS ]
C_FindObjectsInit = CKR_OK
C_FindObjects
  IN: hSession = S1
  IN: max_object_count = 1
 OUT: object = (1) [ H2 ]
C_FindObjects = CKR_OK
C_GetAttributeValue
  IN: hSession = S1
  IN: hObject = H2
  IN: pTemplate = (1) [ CKA_KEY_TYPE ]
 OUT: pTemplate = (1) [ { CKA_KEY_TYPE = CKK_RSA } ]
C_GetAttributeValue = CKR_OK
C_GetAttributeValue
  IN: hSession = S1
  IN: hObject = H2
  IN: pTemplate = (1) [ CKA_LABEL ]
 OUT: pTemplate = (1) [ { CKA_LABEL = (5) NULL } ]
C_GetAttributeValue = CKR_OK
C_GetAttributeValue
  IN: hSession = S1
  IN: hObject = H2
  IN: pTemplate = (1) [ CKA_LABEL ]
 OUT: pTemplate = (1) [ { CKA_LABEL = (5) "mykey" } ]
C_GetAttributeValue = CKR_OK
C_GetAttributeValue
  IN: hSession = S1
  IN: hObject = H2
  IN: pTemplate = (1) [ CKA_ID ]
 OUT: pTemplate = (1) [ { CKA_ID = (1) NULL } ]
C_GetAttributeValue = CKR_OK
C_GetAttributeValue
  IN: hSession = S1
  IN: hObject = H2
  IN: pTemplate = (1) [ CKA_ID ]
 OUT: pTemplate = (1) [ { CKA_ID = (1) "\x01" } ]
C_GetAttributeValue = CKR_OK
C_GetAttributeValue
  IN: hSession = S1
  IN: hObject = H2
  IN: pTemplate = (1) [ CKA_ID ]
 OUT: pTemplate = (1) [ { CKA_ID = (1) "\x01" } ]
C_GetAttributeValue = CKR_OK
C_FindObjects
  IN: hSession = S1
  IN: max_object_count = 1
 OUT: object = (0) [ ]
C_FindObjects = CKR_OK
C_FindObjectsFinal
  IN: hSession = S1
C_FindObjectsFinal = CKR_OK
C_GetAttributeValue
  IN: hSession = S1
  IN: hObject = H2
  IN: pTemplate = (1) [ CKA_MODULUS ]
 OUT: pTemplate = (1) [ { CKA_MODULUS = (256) NOT-PRINTED } ]
C_GetAttributeValue = CKR_OK
C_GetAttributeValue
  IN: hSession = S1
  IN: hObject = H2
  IN: pTemplate = (1) [ CKA_MODULUS ]
 OUT: pTemplate = (1) [ { CKA_MODULUS = (256) NOT-PRINTED } ]
C_GetAttributeValue = CKR_OK
C_GetAttributeValue
  IN: hSession = S1
  IN: hObject = H2
  IN: pTemplate = (1) [ CKA_PUBLIC_EXPONENT ]
 OUT: pTemplate = (1) [ { CKA_PUBLIC_EXPONENT = (3) NOT-PRINTED } ]
C_GetAttributeValue = CKR_OK
C_GetAttributeValue
  IN: hSession = S1
  IN: hObject = H2
  IN: pTemplate = (1) [ CKA_PUBLIC_EXPONENT ]
 OUT: pTemplate = (1) [ { CKA_PUBLIC_EXPONENT = (3) NOT-PRINTED } ]
C_GetAttributeValue = CKR_OK
C_GetAttributeValue
  IN: hSession = S1
  IN: hObject = H2
  IN: pTemplate = (1) [ CKA_ALWAYS_AUTHENTICATE ]
 OUT: pTemplate = (1) [ { CKA_ALWAYS_AUTHENTICATE = (1) "\x00" } ]
C_GetAttributeValue = CKR_OK
C_SignInit
  IN: hSession = S1
  IN: pMechanism = {
        mechanism: CKM_RSA_PKCS
        pParameter: (0) NULL
      }
  IN: hKey = H2
C_SignInit = CKR_OK
C_Sign
  IN: hSession = S1
  IN: pData = (83) "0Q0\r\x06\t`\x86H\x01e\x03\x04\x02\x03\x05\x00\x04@\xD3B\xF3I\xAB\xC9\xEB\xEC^\xD5\xF2\x8B\xF4o>r\xFE7 n 1\x8E;\xF7^&\x..."
 OUT: pSignature = (256) "\xA7\xCB\xFE/\xCE\x12z\x14N\x9D\xE2\x8B\x92\xA7\xE9\x1B~\xAA\x8E\r\xFC\x1B\xDD\x15\x9B\x8C\xE0C;\xA73F\\xF0LD\xF2\x96\x..."
C_Sign = CKR_OK

-- 
dwmw2

-------------------------------------------------------------------
List admin: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html