Using JWT authentication in XCP to XCP Edge connections

TSB’s global control plane (XCP) distributes configuration from the management plane to control plane clusters. These connections use mutual TLS (mTLS) by default.

This document will describe how to configure XCP to use JWT authentication instead of mTLS. This feature requires TSB 1.4 or newer.

Overview

The management plane controls XCP, which in turn is composed of XCP Central and XCP Edge. XCP Central is deployed in the management plane, XCP Edge is deployed in each of the onboarded clusters where user applications run. XCP Central communicates with each of the XCP Edges. It is very crucial for TSB that each of the XCP components can talk to each other in a secure way, and hence these connections are authenticated.

In a default TSB configuration, control plane clusters receive a TLS key and certificate from the management plane which is used by XCP Edge to authenticate with the XCP Central.

The default setup is straightforward, but all of the authentication uses the same root of trust (root CA) as the other control planes. This may pose a problem if the management of your clusters are shared among several organizations, for example when the Management Plane and Control Plane clusters are owned by different teams who do not necessarily want to share to root CA keys.

Using JWT authentication as an alternate form of authentication would solve this problem, and allow finer grain of control of the authentication.

By not needing to access the same root CA with other control planes, you will be able to limit the number of components that need to generate certificates from the CA and thus increase the overall security of your system.

Migrating TSB to use JWT Authentication

Migrating a TSB installation from mTLS to JWT authentication is a 4 step process:

1. Generate a new XCP central certificate that includes its address as well as the SPIFFE ID used for mTLS authentication.
2. Enable JWT authentication is the management plane custom resource.
3. Migrate each control plane cluster to use JWT authentication.
4. Disable mTLS authentication in the management plane custom resource.

Generate a new XCP central certificate

To allow control plane clusters to connect via regular (non-mutual) TLS, the XCP central certificate must include its address in its subject alternate names (SANs). This will either be a DNS name or an IP address.

You can check what address is being used by running the following command in a control plane cluster:

kubectl get -A controlplane -o jsonpath='{.items[0].spec.managementPlane.host}

For example, if you are using cert-manager to create the certificate, edit the Certificate resource and add in either the DNS name or IP address that your control plane clusters use to connect to the management plane. Then delete the generated certificate secret to make cert-manager generate a new one.

If you are using an IP address, edit the field spec.ipAddresses:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mpc-certs
  namespace: tsb
spec:
  secretName: xcp-central-cert
  ipAddresses:
  - a.b.c.d  ## <--- HERE
  issuerRef:
    name: xcp-identity-issuer
    kind: Issuer
  duration: 30000h0m0s
  uris:
  - spiffe://xcp.tetrate.io/central
  usages:
  - server auth
  - client auth

Or, if you are using domain names, edit the field spec.dnsNames

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mpc-certs
  namespace: tsb
spec:
  dnsNames:
  - example-tsb.tetrate.io ## <-- HERE
  secretName: xcp-central-cert
  issuerRef:
    name: xcp-identity-issuer
    kind: Issuer
  duration: 30000h0m0s
  uris:
  - spiffe://xcp.tetrate.io/central
  usages:
  - server auth
  - client auth

Enable JWT authentication in the management plane

Edit the management plane custom resource and enable JWT authentication, e.g.

$ kubectl edit -n tsb managementplane/tsbmgmtplane

And change the value of centralAuthMode.jwt to true:

...
spec:
  components:
    ...
    xcp:
      centralAuthModes:
        jwt: true  ## <-- HERE
        mutualTls: true  # allows current control plane clusters to connect
...

After applying the changes to the Management Plane CR, the TSB operator will configure XCP Central to also expect connections using JWT authentication.

Migrating control plane clusters to use JWT authentication

These steps must be completed for each of your control plane clusters.

Generating cluster secrets

You will need to update/create secrets for each cluster that are onboarded to TSB.

Instructions on generating cluster secrets differ depending on whether your XCP central certificate is signed by a public root CA (e.g. Digicert or Let's Encrypt) or you are using a private PKI. Either way you will be using tctl to generate the control plane secrets for the cluster.

If you are using a Public CA, use the following command to create and deploy the necessary secrets. This will create a secret for XCP edge that contains a JWT used to authenticate with XCP central.

tctl install manifest control-plane-secrets | kubectl apply -f -

After performing this step, proceed to updating the cluster configuration to use JWT authentication further down in this section.

If you are using a private PKI, you will need to create two secrets. One contains the JWT used to authenticate with XCP central. The other secret is the root CA used by XCP edge to verify XCP central's TLS certificate.

Create a file ca.crt containing your root CA certificate encoded in PEM format. If you're using cert-manager this can be extracted from the management plane cluster via:

kubectl get secret -n tsb xcp-central-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt

Then generate and apply the secrets to your control plane cluster:

tctl install manifest control-plane-secrets \
  --xcp-central-ca-bundle=”$(cat ca.crt) \
  | kubectl apply -f -

Once the secrets are created, update cluster to use JWT authentication. Edit the control plane custom resource using kubectl edit:

$ kubectl edit -n istio-system controlplane/controlplane

And edit the configuration to enable JWT authentication:

...
spec:
  components:
    ...
    xcp:
      centralAuthMode: JWT ## <-- HERE
...

After applying the changes to the Control Plane CR, the TSB operator will configure the XCP edge to connect using this JWT.

Finally, validate that the control plane cluster is still connected to your management plane by checking the cluster's last sync time in the clusters UI.

The XCP edge certificate created for using mTLS is no longer needed, and can be deleted via:

kubectl delete secret -n istio-system secret/xcp-edge-cert

If you are using cert-manager to create this secret, the Certificate resource in the management plane cluster should also be deleted.

You will have to repeat this step for other clusters, if you have any.

Disable mTLS authentication in the management plane

Once all control plane clusters have been migrated to use JWT authentication, edit the management plane resource and disable mTLS completely. Execute the following command to start editing:

$ kubectl edit -n tsb managementplane/tsbmgmtplane

Then edit the configuration:

...
spec:
  components:
    ...
    xcp:
      centralAuthModes:
        jwt: true
        mutualTls: false  ## <--- HERE. Set to false, or alternatively remove the mutualTls entry
...

At this point TSB should be configured to communicate internally using JWTs, and mTLS should be disabled.