Skip to main content
Version: 1.4.x

TSB Upgrade

This page will walk you through how to upgrade TSB using the tctl CLI, rendering the Kubernetes manifests for the different operators and applying them to the clusters to upgrade using kubectl.

Before you start, make sure that you have:

✓ Checked the new version’s requirements
✓ Use organization root instead of tenant root if the cluster was TSB <1.2.

The upgrade procedure between operator-based releases is fairly simple. Once the operator pods are updated with the new release images, the newly spun up operator pods will upgrade all the necessary components to the new version for you.

Create Backups​

In order to make sure you can restore everything when something goes wrong, please create backups for the Management Plane and each of your clusters' local Control Planes.

Backup the Management Plane​

Backup the tctl binary​

Since each new tctl binary potentially comes with new operators and configurations to deploy and configure TSB, you should backup the current tctl binary you are using. Please do this before syncing the new images.

Copy the tctl binary with version suffix (e.g. -1.3.0) to quickly restore the older one if needed.

cp ~/.tctl/bin/tctl ~/.tctl/bin/tctl-{version}

If you have misplaced your binary, you may be able to find the right version from this URL. However, it is strongly recommended that you backup your current copy to be sure.

Backup the Management Plane Custom Resource​

Create a backup of the Management Plane CR by executing the following command:

kubectl get managementplane -n tsb -o yaml > mp-backup.yaml

Backup the PostgreSQL database​

Create a backup of your PostgreSQL database.

The exact procedure for connecting to the database may differ depending on your environment, please refer to the documentation for your environment.

Backup the Control Plane Custom Resource​

Create a backup of all ControlPlane CRs by executing the following command on each of your onboarded clusters:

kubectl get controlplane -n istio-system -o yaml > cp-backup.yaml

Upgrade Procedure​

Download tctl and Sync Images​

Now that you have taken backups, download the new version's tctl binary, then obtain the new TSB container images.

Details on how to do this is described in the Requirements and Download page

Upgrading to TSB 1.4.x from < 1.2.0​

If you are upgrading to TSB 1.4.x from an older release, you may need to be prepared for a short downtime in order to upgrade the Elasticsearch indices that are used by SkyWalking.

As a general rule, if you are upgrading from a version of TSB lower than 1.2.0 (i.e. 1.1.x, 1.0.x, 0.9.x) then you will need to perform the following procedures. Another way to check if you need to do this is to examine the indices stored in the Elasticsearch. If the list of indices resembles that of the list in this file, you will need to perform the upgrade procedures.

First, scale down the oap deployment in the management namespace:

kubectl -n <management-plane-namespace> \
  scale deployment oap --replicas=0

Then scale down the oap-deployment deployment. This must be done in all onboarded clusters:

kubectl -n <control-plane-namespace> \
  scale deployment oap-deployment --replaices=0

Once all of the deployments have been scaled back, run the following shell script against your Elasticsearch installation. It is assumed that you have all of the necessary information to access Elasticsearch, such as elasticsearch-host, elasticsearch-port, etc.

for tmpl in $(curl -u "<elasticsearch-user>:<elasticsearch-password>" http://<elasticsearch-host>:<elasticsearch-port>/_cat/templates | \
  egrep "alarm_record|browser_|events|log|meter-|metrics-|endpoint_|envoy_|http_access_log|profile_|service_|register_lock|instance_traffic|segment|network_address|top_nj" | \
  awk '{print $1}'); do curl -u "<elasticsearch-user>:<elasticsearch-password>" http://<elasticsearch-host>:<elasticsearch-port>/_template/$tmpl -XDELETE ;
done

for idx in $(curl -u "<elasticsearch-user>:<elasticsearch-password>" http://<elasticsearch-host>:<elasticsearch-port>/_cat/indices | \
  egrep "alarm_record|browser_|events|log|meter-|metrics-|endpoint_|envoy_|http_access_log|profile_|service_|register_lock|instance_traffic|segment|network_address|top_n" | \
  awk '{print $3}'); do curl -u "<elasticsearch-user>:<elasticsearch-password>" http://<elasticsearch-host>:<elasticsearch-port>/$idx -XDELETE ;
done

The above command assumes that your Elasticsearch instance accepts plain HTTP connections with basic authentication. Modify the commands as necessary. For example, if you forcing HTTPS connections, you will need to change the URL scheme to https.

Once the above script runs successfully and all indices are deleted, you can proceed with upgrading the Management Plane.

Remember to make sure that the oap deployment is properly running again once you have upgraded the Management Plane. The Management Plane should have created the index templates and indices required by the new system.

Upgrading to TSB 1.4.x from 1.2.x​

If you are upgrading to TSB 1.4.x from 1.2.x you will need to manually scale down tsb-data-reconcile deployment manually before the management plane upgrade.

You can scale down the deployment running:

kubectl scale deployment tsb-data-reconcile --replicas 0 -n tsb

Once upgraded to 1.4.x you can remove the deployment:

kubectl delete deployment tsb-data-reconcile -n tsb

Create the Management Plane Operator​

Create the base manifest which will allow you to update the management plane operator from your private Docker registry:

tctl install manifest management-plane-operator \
    --registry <your-docker-registry> \
    > managementplaneoperator.yaml
Management namespace name

Starting with TSB 0.9.0 the default Management Plane namespace name is tsb as opposed to tcc used in older versions. If you installed TSB using an earlier version than 0.9.0, your Management Plane probably lives in the tcc namespace. You will need to add a --management-namespace tcc flag to reflect this.

Customization

The managementplaneoperator.yaml file created by the install command can now be used as a base template for your Management Plane upgrade. If your existing TSB configuration contains specific adjustments on top of the standard configuration, you should copy them over to the new template.

Now, add the manifest to source control or apply it directly to the management plane cluster by using the kubectl client:

kubectl apply -f managementplaneoperator.yaml

After applying the manifest, you will see the new operator running in the tsb namespace:

kubectl get pod -n tsb
NAME                                            READY   STATUS    RESTARTS   AGE
tsb-operator-management-plane-d4c86f5c8-b2zb5   1/1     Running   0          8s

For more information on the manifest and how to configure it, please review the ManagementPlane resource reference

Create the Control and Data Plane operators​

To deploy the new Control and Data Plane operators in your application clusters, you must run tctl install manifest cluster-operators to retrieve the Control Plane and Data Plane operator manifests for the new version.

tctl install manifest cluster-operators \
    --registry <your-docker-registry> \
    > clusteroperators.yaml
Customization

The clusteroperators.yaml file can now be used for your cluster upgrade. If your existing control and Data Planes have specific adjustments on top of the standard configuration, you should copy them over to the template.

Istio Operator​

If you are upgrading from TSB 1.2.x or older, you must manually remove the old Istio Operator first.

First, make sure that there are no instances of the old controller running by issuing the following command:

kubectl -n istio-system scale --replicas=0 deploy/istio-operator

Then, patch the Istio Operator so that finalizers are disabled:

kubectl -n istio-system patch iop tsb-istiocontrolplane --patch '{"metadata":{"finalizers":[null]}}' --type merge

Then, scale down tsb-operator-control-plane deployment to 0:

kubectl -n istio-system scale --replicas=0 deploy/tsb-operator-control-plane

Finally, delete the old Istio Operator (tsb-istiocontrolplane)

kubectl -n istio-system delete iop tsb-istiocontrolplane

Applying the Manifest​

Now, add the manifest to source control or apply it directly to the appropriate clusters by using the kubectl client:

kubectl apply -f clusteroperators.yaml

For more information on each of these manifests and how to configure them, please check out the following guides:

Upgrading from TSB < 1.0​

Since TSB > 1.0, the XCP component is enabled by default. XCP requires extra setup such that the Management Plane (XCP Central) can communicate with Control Planes (XCP Edge).

Please refer to Certificate Requirements for details.

Rollback​

In case something goes wrong and you want to rollback TSB to the previous version, you will need to rollback both the Management Plane and the Control Planes.

Rollback the Control Plane​

Scale down istio-operator and tsb-operator​

kubectl scale deployment \
   -l "platform.tsb.tetrate.io/component in (tsb-operator,istio)" \
   -n istio-system \
   --replicas=0

Delete the IstioOperator Resource​

kubectl delete istiooperator -n istio-system --all

Scale down istio-operator and tsb-operator for the Data Plane operator​

kubectl scale deployment \
   -l "platform.tsb.tetrate.io/component in (tsb-operator,istio)" \
   -n istio-gateway \
   --replicas=0

Delete the IstioOperator Resource for the Data Plane​

kubectl delete istiooperator -n istio-gateway --all

Create the Cluster Operators, and rollback the Control Plane CR​

Using the tctl binary from the previous version, follow the instructions to create the cluster operators.

Then apply the the backup of the Control Plane CR:

kubectl apply -f cp-backup.yaml

Rollback the Management Plane​

Scale Down Pods in Management Plane​

Scale down all of the pods in the Management Plane so that the it is inactive.

kubectl scale deployment tsb iam tsb-data-reconcile -n tsb --replicas=0

Restore PostgreSQL​

Restore your PostgreSQL database from your backup. The exact procedure for connecting to the database may differ depending on your environment, please refer to the documentation for your environment.

Restore tctl and create the Management Plane operator​

Restore tctl from the backup copy that you made, or download the binary for the specific version you would like to use.

mv ~/.tctl/bin/tctl-{version} ~/.tctl/bin/tctl

Follow the instructions for upgrading to create the Management Plane operator. Then apply the backup of the Management Plane CR:

kubectl apply -f mp-backup.yaml

Scale back the deployments​

Finally, scale back the deployments.

kubectl scale deployment tsb iam tsb-data-reconcile -n tsb --replicas 1