Add PKCS#11 provider support for ECDH and XDH key exchange

[olszomal]

Pull Request Type

• Bug fix
• New feature
• Code style / formatting / renaming
• Refactoring (no functional or API changes)
• Build / CI related changes
• Documentation
• Other (please describe):

Related Issue

Issue number: N/A

Current Behavior

The PKCS#11 provider does not implement OSSL_OP_KEYEXCH.

New Behavior

This change adds PKCS#11 provider support for key exchange operations.

The provider now implements OSSL_OP_KEYEXCH and routes EC, X25519, and X448 derive operations through PKCS#11. For EC keys, ECDH shared-secret derivation is performed with CKM_ECDH1_DERIVE or CKM_ECDH1_COFACTOR_DERIVE, including cofactor mode handling and peer public key import.

The change also adds X25519 and X448 key generation support using CKM_EC_MONTGOMERY_KEY_PAIR_GEN, detects XDH key types from CKA_EC_PARAMS, handles raw public key export/import for X25519 and X448, and wires XDH derive plumbing through the provider path.

Scope of Changes

• Add OSSL_OP_KEYEXCH dispatch and provider registration.
• Add PKCS11_evp_pkey_derive() as the common front-end derive helper.
• Add ECDH derive support through PKCS#11.
• Add X25519/X448 key generation through EC_MONTGOMERY.
• Add X25519/X448 key type detection and raw public key handling.
• Add legacy EVP_PKEY_METHOD derive hooks for X25519/X448 on OpenSSL versions before 4.0.
• Add X25519/X448 key generation examples.
• Add SoftHSM-based ECDH derive tests for both engine and provider paths.

Testing

• Existing tests
• New tests added
• Manual testing -> performed with Thales TCT Luna T-5000 Network HSM and YubiKey 5C

New SoftHSM tests generate an EC key pair with derive usage on the token and verify that:
ECDH(token_private, software_public) == ECDH(software_private, token_public)

Additional Notes

X25519/X448 derive support is wired through the provider path, but the actual hardware derive path remains untested for now because no available token/module advertises EC_MONTGOMERY derive support.

License Declaration

• I hereby agree to license my contribution under the project's license.

[mtrojnar]

[Medium] Provider ECDH derive fails when the peer key is also loaded from pkcs11prov

src/provider_helpers.c:1939

p11_keyexch_ctx_get_peer_pub() only reads keydata->pubkey, which is populated for imported cross-provider peers. Token-loaded provider keys store public bytes in pubdata / params, so using a PKCS#11 public key as -peerkey fails even though the public key was loaded successfully.

Impact: provider ECDH works with a software peer key, but fails for token-private + token-public derive flows.

Suggested fix: make p11_keyexch_ctx_get_peer_pub() fall back to pubdata.ec.pub / pubdata.raw.pub, or populate pubkey when creating P11_KEYDATA from an EVP_PKEY.

---

[Medium] X25519/X448 legacy wrappers can break non-PKCS#11 software derives

src/p11_eddsa.c:346

pkcs11_xdh_pmeth_derive() returns 0 when the key has no libp11 ex_data, but callers only delegate to OpenSSL when ret < 0 (src/p11_eddsa.c:406). After the global X25519/X448 method is registered, unrelated software XDH derives can fail instead of falling back to OpenSSL.

Suggested fix: return -1 for “foreign key” cases, or explicitly fall back whenever no PKCS#11 ex_data is present.

---

[Medium] X25519/X448 public-key equality is broken when GROUP_NAME is present

src/provider_helpers.c:869

The PR adds OSSL_PKEY_PARAM_GROUP_NAME values "X25519" / "X448", but p11_public_equal() treats any non-EdDSA group as classical EC and calls ec_point_equal_by_value() on raw XDH bytes. Identical X25519/X448 public keys with a group name therefore compare unequal.

Suggested fix: special-case "x25519" and "x448" alongside Ed25519/Ed448 and compare PUB_KEY as raw octets.

---

[Low] XDH keygen examples read past argv when PIN is omitted

examples/x25519keygen.c:92
examples/x448keygen.c:92

Both examples check argc < 5 but later use argv[5] for the PIN. Running the examples without a PIN can read past the argument array.

Suggested fix: require six arguments, e.g. argc < 6 or argc != 6.

[olszomal]

Thanks, fixed all issues:

• provider ECDH now accepts peer public keys loaded from pkcs11prov;
• legacy X25519/X448 derives fall back for non-PKCS#11 keys;
• X25519/X448 public keys are compared as raw octet strings.

I also fixed dangling PKCS11_KEY references in EVP_PKEY ex data.
The ECDH tests were extended to cover token-private plus token-public peer derives.

[mtrojnar]

Review findings

The previously reported provider-peer, software fallback, and XDH equality issues are fixed. I found the following remaining issues:

[Medium] Oversized output buffers cause ECDH derivation to fail

src/provider.c:1974

outlen is the caller’s buffer capacity, but it is passed as the requested PKCS#11 CKA_VALUE_LEN. A caller may legally provide a buffer larger than the required secret size. For P-256, passing 64 bytes
instead of the required 32 makes SoftHSM reject the derivation.

Set *secretlen = need before calling PKCS11_evp_pkey_derive(); use outlen only to verify capacity.

I reproduced this by allocating 32 extra bytes after the size query: provider-ec-derive.softhsm then fails with CKR_ATTRIBUTE_TYPE_INVALID.

[Medium] EVP_PKEY_derive_init_ex() parameters are silently ignored

src/provider_helpers.c:1871

p11_keyexch_ctx_init() discards params and resets cofactor_mode to -1. Consequently, parameters supplied through EVP_PKEY_derive_init_ex(), including OSSL_EXCHANGE_PARAM_EC_ECDH_COFACTOR_MODE, are not
applied.

Please process the initialization parameters using the same validation as keyexch_set_ctx_params().

[Medium] Derivation does not handle CKA_ALWAYS_AUTHENTICATE

src/p11_pkey.c:624

pkcs11_derive_with_mechanism() calls C_DeriveKey without performing context-specific authentication. Private keys with CKA_ALWAYS_AUTHENTICATE=CK_TRUE will therefore fail key exchange even though signing and
decryption support these keys.

Please call pkcs11_authenticate(key, session) before C_DeriveKey, as the other private-key operation helpers do.

[Medium] XDH certificate fallback rejects X25519/X448 certificates

src/p11_eddsa.c:614

The X25519/X448 loaders use pkcs11_get_raw_public_key(), which falls back to extracting the public key from a matching certificate. However, the certificate extractor accepts only Ed25519 and Ed448.

A token containing an XDH private key and certificate but no separate public-key object cannot load the private key. Please also accept EVP_PKEY_X25519 and EVP_PKEY_X448, or make this a generic ECX
certificate extractor.

[Medium] Legacy XDH methods are not cleaned up correctly

src/p11_eddsa.c:523
src/p11_eddsa.c:926

pkcs11_xdh_key_method_free() is never called from libp11_global_free(), leaving globally registered X25519/X448 methods after the last context is released. Additionally, the X448 loader registers
pkcs11_x25519_method_free() with atexit() instead of pkcs11_x448_method_free().

Please invoke pkcs11_xdh_key_method_free() alongside pkcs11_ed_key_method_free() and correct the X448 callback.

[Low] XDH security strength is reported as zero

src/provider_helpers.c:693

p11_keydata_get_security_bits() handles Ed25519 and Ed448 but omits X25519 and X448, causing EVP_PKEY_get_security_bits() to return zero. Add X25519 → 128 and X448 → 224.

[Low] XDH examples still read past argv when PIN is omitted

examples/x25519keygen.c:92
examples/x448keygen.c:92

Both examples check argc < 5 but later access argv[5]. Please require at least six arguments.

Tests

• ./bootstrap && ./configure --enable-strict && make -j2 — pass
• make -C tests check TESTS='ec-derive.softhsm provider-ec-derive.softhsm' — pass
• Provider ECDH with an oversized output buffer — fail with CKR_ATTRIBUTE_TYPE_INVALID, confirming the first finding
• XDH hardware derivation remains untested because the available module does not support EC Montgomery derivation

[olszomal]

I have addressed all reported findings:

• use the required secret size instead of the caller’s buffer capacity,
• process EVP_PKEY_derive_init_ex() parameters,
• handle CKA_ALWAYS_AUTHENTICATE during derivation,
• support X25519/X448 certificate fallback,
• fix legacy XDH method cleanup,
• report XDH security strength correctly,
• and fix the argument count checks in the XDH examples.