1.0 Upgrade Notes

TSB 1.0 introduces a new cross cluster control plane agent called XCP, deprecating tsbd. Because of this, there are some considerations that need to be taken into account on upgrade.

In order to upgrade, follow the instructions in sequence:

1. Tier1 routing based on network
2. Management Plane Upgrade
3. Control Plane Upgrade

Multi Phase Upgrade

If you are planning for a multi-phase upgrade, read the note about OAP

Tier1 routing based on network

Up to TSB 0.9.x, you could define routing rules in Tier1Gateway objects that would forward traffic to all the clusters exposing a given host that have a given network tag. For instance:

  externalServers:
  - name: bookinfo
    hostname: bookinfo.1.2.3.4.nip.io
    port: 443
    tls:
      mode: SIMPLE
      secretName: tier1-cert
    clusters:
    - network: tier2

This is no longer supported with the introduction of XCP. You will need to switch to label based routing. The first thing you need to do is make sure your clusters have labels that you can use for routing.

apiVersion: api.tsb.tetrate.io/v2
kind: Cluster
metadata:
  name: my-tier2-cluster
  tenant: tetrate
  labels:
    network: tier2
spec:
  network: tier2
  tier1Cluster: false
  tokenTtl: 31536000s

In the Cluster above, a label called network has been added with the same value as the network tag of the cluster. Once your Cluster objects have been updated with the right labels, you can change the Tier1Gateway configuration to route based on this label instead of the network attribute. The example shown above becomes:

  externalServers:
  - name: bookinfo
    hostname: bookinfo.1.2.3.4.nip.io
    port: 443
    tls:
      mode: SIMPLE
      secretName: tier1-cert
    clusters:
    - label:
        network: tier2

service disruption

Depending on the cloud provider or infrastructure you use, this action can cause a brief service disruption (~5s).

Management Plane Upgrade

Another requirement of XCP is the use of TLS certificates for communication between the central component of XCP in the management plane, and their edge counterparts in the control planes.

One easy way to achieve this is to deploy cert-manager in your cluster and leverage tctl to create the needed cert-manager object that will generate such certificates.

tctl install manifest management-plane-secrets \
  --elastic-password ${ELASTIC_PASSWORD} \
  --elastic-username ${ELASTIC_USERNAME} \
  --elastic-ca-certificate "${ELASTIC_CA}" \
  --postgres-password ${POSTGRES_PASSWORD} \
  --postgres-username ${POSTGRES_USERNAME} \
  --tsb-admin-password admin \
  --tsb-server-certificate "$(cat ~/src/tetrate/test/certs/dummy.cert.pem)" \
  --tsb-server-key "$(cat ~/src/tetrate/test/certs/dummy.key.pem)" -y \
  --xcp-certs | kubectl apply -f -

The --xcp-certs flag creates some Issuer and Certificate objects that populate the needed secrets in the management plane namespace.

If you are not using cert-manager, you can create the certificates using your existing PKI and store the TLS certificate in the Kubernetes secret called xcp-central-cert in the management namespace. Check the set up instructions for further information.

Once the central XCP is up and running, you will need to create XCP’s GlobalSetting object to specify network reachability, similar to what you did in TSB UI or API up to 0.9.x.

apiVersion: xcp.tetrate.io/v2
kind: GlobalSetting
metadata:
  namespace: tsb
  name: xcp-global-settings
spec:
  networkSettings:
    networkReachability:
      tier1: tier2

The example above configures XCP so that clusters with network: tier1 will set up multi cluster routing with clusters having network: tier2. You will need to set this up mimicking your current setup in TSB API.

Control Plane Upgrade

After upgrading the management plane to TSB 1.0 (not before!), you can start upgrading existing control planes. Before starting the upgrade, you need to create the certificate that the edge XCP will use to authenticate to the central XCP.

For that, if you use the cert-manager approach, you can also leverage tctl to get the needed Kubernetes secret YAML. Having your kubectl context pointing to the management plane cluster:

tctl install cluster-certs --cluster <cluster-name> \
    > /tmp/edge-cert.yaml

Where <cluster-name> is the TSB cluster name of the control plane you are about to upgrade. You can apply the resulting YAML in the control plane cluster. If you are not using cert-manager, check the set up instructions for the control plane for the requirements of the certificate that needs to be present in the control plane namespace.

At this point it is safe to upgrade the TSB control plane:

tctl install manifest cluster-operators \
    --registry <registry-location>

The command above will output the needed Kubernetes manifests which you can dump to a file for review or modification (adding tolerations, etc.). Once the YAML is OK you can proceed and apply it to the cluster. The new pod will upgrade the control plane components after start up. You will see tsbd deployments disappearing, and the new XCP components showing up. Wait for all components to be ready.

Once XCP edge is up, you will need to remove the old tsbd configurations, if any. You can use the command below to that end:

while read p; do $p; done < <(kubectl get gw,vs,dr,se --all-namespaces -o=jsonpath='{range .items[?(@.metadata.annotations.tsb\.tetrate\.io/contentHash!="")]}kubectl delete -n {.metadata.namespace} {.kind}/{.metadata.name}{"\n"}{end}')

service disruption

Depending on the cloud provider or infrastructure you use, this could cause a brief service disruption (~1min).

NOTE about OAP

If you are not planning for a synchronized upgrade of all the control planes at the same time, you will need to run a CronJob in the TSB management plane namespace (by default tsb) that will create some additional indices that are deprecated in 1.0, but still needed for the 0.9.x OAP in the control planes to function. Once all the control planes have been upgraded to 1.0 you can remove the oap-init CronJob from the management plane namespace.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  labels:
    app: oap-init
    platform.tsb.tetrate.io/application: oap-init
    platform.tsb.tetrate.io/component: oap
    platform.tsb.tetrate.io/plane: management
  name: oap-init
  namespace: tsb
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        metadata:
          creationTimestamp: null
          labels:
            platform.tsb.tetrate.io/application: oap-init
            platform.tsb.tetrate.io/component: oap
            platform.tsb.tetrate.io/plane: management
        spec:
          containers:
          - env:
            - name: JAVA_OPTS
              value: -Xmx4096M -Dmode=init
            # This denotes the target Elasticsearch nodes. Should match
            # `spec.telemetryStore.elastic.host`:`spec.telemetryStore.elastic.port`
            - name: SW_STORAGE_ES_CLUSTER_NODES
              value: <host>:<port>
            # This denotes the protocol to use when connecting to Elasticsearch nodes. Should match
            # `spec.telemetryStore.elastic.protocol`
            - name: SW_STORAGE_ES_HTTP_PROTOCOL
              value: <http|https>
            # Credentials will be the same as used by OAP
            - name: SW_ES_USER
              valueFrom:
                secretKeyRef:
                  key: username
                  name: elastic-credentials
                  optional: true
            - name: SW_ES_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: elastic-credentials
                  optional: true
            - name: SW_SW_STORAGE_ES_SSL_JKS_PATH
              value: /etc/skywalking/truststore.jks
            - name: SW_SW_STORAGE_ES_SSL_JKS_PASS
              value: tetrate
            # Replace with your Docker registry URL. Image and tag should be kept as is.
            image: <docker-registry>/spm-central:11b382507838a7051bad260c9188c7fed2ab83f9-es7
            imagePullPolicy: IfNotPresent
            name: oap-init
            volumeMounts:
            - mountPath: /skywalking/ext-config/log4j2.xml
              name: config
              subPath: log4j2.xml
            - mountPath: /etc/skywalking
              name: keystore-volume
          initContainers:
          - args:
            - -c
            - csplit -z -f crt- $CA_BUNDLE '/-----BEGIN CERTIFICATE-----/' '{*}' && for
              file in crt-*; do keytool -import -noprompt -keystore $TRUSTSTORE_JKS -file
              $file -storepass $PASSWORD -alias service-$file; done
            command:
            - /bin/bash
            env:
            - name: CA_BUNDLE
              value: /var/run/secrets/elasticsearch.io/ca.crt
            - name: TRUSTSTORE_JKS
              value: /var/run/secrets/java.io/keystores/truststore.jks
            - name: PASSWORD
              value: tetrate
            # Replace with your Docker registry URL. Image and tag should be kept as is.
            image: <docker-registry>/sso71-openshift:1.1-16
            imagePullPolicy: IfNotPresent
            name: pem-to-truststore
            volumeMounts:
            - mountPath: /var/run/secrets/elasticsearch.io
              name: es-certs
            - mountPath: /var/run/secrets/java.io/keystores
              name: keystore-volume
          restartPolicy: OnFailure
          serviceAccount: tsb-oap
          serviceAccountName: tsb-oap
          volumes:
          - configMap:
              defaultMode: 420
              items:
              - key: log4j2.xml
                path: log4j2.xml
              name: oap-config
            name: config
          - name: es-certs
            secret:
              defaultMode: 420
              optional: true
              secretName: es-certs
          - emptyDir: {}
            name: keystore-volume
  schedule: 0 0 * * *
  successfulJobsHistoryLimit: 3
  suspend: false

note

This is a generic CronJob with the necessary images and configuration for OAP to create the necessary indices. If you need to add tolerations, affinity, or other Kubernetes requirements specific to your environment or distribution, you will need to add them to the CronJob depicted above.