Restoring Kubernetes’ ability to sign certificates

Note

Requires at least version 6.0

Since Tarook migrated to Hashicorp Vault for managing PKI and therefore clears the Kubernetes cluster root CA key from the control plane, Tarook clusters are not able to respond to certificate signing requests (CSRs) anymore since access to PKI keys ceased. [1]

While in the long term we want to integrate Vault via a Kubernetes custom signer [2] which would redirect CSRs in Kubernetes directly to Vault, for now we provide a workaround fix to restore the CSR functionality. The fix must be explicitly turned on by setting yk8s.kubernetes.controller_manager.enable_signing_requests to true.

Enabling the fix

Attention

This weakens the security of your deployment!

Enable signing requests in the config

Set yk8s.kubernetes.controller_manager.enable_signing_requests = true in the config
Configure k8s_control_plane Vault policy

Executing ‘tools/vault/init.sh’ will take care of that. This assumes you already have a Vault instance running and reachable from your Tarook shell environment.

Note

Requires a Vault root token
```
./managed-k8s/tools/vault/init.sh
```
The Kubernetes cluster root CA key is made available through the k8s-pki/cluster-root-ca secret in the Vault kv2 store of the cluster. The script adds a Vault policy rule (among other things) that grants read-only access to it for the control plane nodes role.

Create the Kubernetes cluster root CA and backup its key

Executing ‘tools/vault/mkcluster-root.sh’ will take care of that.

Note

Requires a Vault root token
```
./managed-k8s/tools/vault/mkcluster-root.sh
```
The script creates the Kubernetes cluster root CA and backs up its private key to k8s-pki/cluster-root-ca in the Vault kv2 store of the cluster [3].
Build the cluster using the apply-all action

This will copy and configure the Kubernetes cluster root CA key from Vault’s kv2 store on all control plane nodes.

Recreate the Kubernetes cluster root CA and backup its key

You must perform a complete root CA rotation, see Root Certificate Authority Rotation

Executing the preparation phase of ‘tools/vault/rotate-root-ca-root.sh’ will backup the Kubernetes cluster root CA key to k8s-pki/cluster-root-ca in the Vault kv2 store of the cluster [3]. when the CA is recreated.
Tip

SHORTCUT

If you have access to the current CA key you may skip the root CA rotation and instead manually upload it to Vault’s kv2 store:
```
clustername="$(yq --raw-output .vault_cluster_name inventory/yaook-k8s/group_vars/all/vault-backend.yaml)"
vault kv put -mount="yaook/${clustername}/kv" k8s-pki/cluster-root-ca private_key=@current-ca.key
```
Tip

SHORTCUT

If you previously had the fix enabled and disabled it again and since then did not change the Kubernetes cluster root CA you may try to undelete the CA key in Vault’s kv2 store:
```
clustername="$(yq --raw-output .vault_cluster_name inventory/yaook-k8s/group_vars/all/vault-backend.yaml)"
export VAULT_TOKEN=$root_token
ca_key_version="$(vault kv get -format=json -mount=yaook/${clustername}/kv k8s-pki/cluster-root-ca | jq .data.metadata.version)"
vault kv undelete -versions="${ca_key_version}" -mount=yaook/${clustername}/kv k8s-pki/cluster-root-ca
```
Run at least the k8s-master tag of the apply-k8s-core action
```
AFLAGS="--tags k8s-master" ./managed-k8s/actions/apply-k8s-core.sh
```
This will copy and configure the Kubernetes cluster root CA key from Vault’s kv2 store on all control plane nodes.

Optional: Check that certificate signing is functional again now

See the Kubernetes documentation on how to do that.

Disabling the fix

Set yk8s.kubernetes.controller_manager.enable_signing_requests = true in the config

Nothing to care about, go ahead as normal

Run at least the k8s-master tag of the apply-k8s-core action

Note

Requires a Vault root token

AFLAGS="--tags k8s-master" ./managed-k8s/actions/apply-k8s-core.sh

This will delete and deconfigure the Kubernetes cluster root CA key on all control plane nodes AND delete the backup in Vault’s kv2 store (this requires a Vault root token).

Note

The key backup is not destroyed [4], you may still undelete it again later:

Note

Requires a Vault root token

clustername="$(yq --raw-output .vault_cluster_name inventory/yaook-k8s/group_vars/all/vault-backend.yaml)"
ca_key_version="$(vault kv get -format=json -mount=yaook/${clustername}/kv k8s-pki/cluster-root-ca | jq .data.metadata.version)"
vault kv undelete -versions="${ca_key_version}" -mount=yaook/${clustername}/kv k8s-pki/cluster-root-ca

If you wish to completely remove the key backup from Vault, run:

Note

Requires a Vault root token

clustername="$(yq --raw-output .vault_cluster_name inventory/yaook-k8s/group_vars/all/vault-backend.yaml)"
vault kv destroy -mount=yaook/${clustername}/kv \
    -versions="0,$(
        vault kv metadata get -format=json -mount=yaook/${clustername}/kv k8s-pki/cluster-root-ca \
        | jq '.data.versions | keys_unsorted[] |  tonumber' | tr '\n' ','
    )" \
    k8s-pki/cluster-root-ca \
&& vault kv metadata delete -mount=yaook/${clustername}/kv k8s-pki/cluster-root-ca