Skip to main content
Version: 1.5.x

Control Plane

ControlPlane resource exposes a set of configurations necessary to automatically install the Service Bridge control plane on a cluster. The installation API is an override API so any unset fields that aren't required will use sensible defaults.

Prior to creating the ControlPlane resource, a cluster needs to be created in the management plane. Control plane install scripts would create the following secrets in the Kubernetes namespace the control plane is deployed into. Make sure they exist:

  • oap-token
  • zipkin-token
  • otel-token

If your Elasticsearch backend requires authentication, ensure you create the following secret:

  • elastic-credentials

A minimal resource must have the container registry hub, telemetryStore, and managementPlane fields set.

apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
hub: docker.io/tetrate
telemetryStore:
elastic:
host: elastic
port: 5678
managementPlane:
host: tsb.tetrate.io
port: 8443
clusterName: cluster

To configure infrastructure specific settings such as resource limits in Kubernetes, set the relevant field in a component. Remember that the installation API is an override API so if these fields are unset the operator will use sensible defaults. Only a subset of Kubernetes configuration is available and only for individual components.

apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
hub: docker.io/tetrate
imagePullSecrets:
- name: my-registry-creds
telemetryStore:
elastic:
host: elastic
port: 5678
managementPlane:
host: tsb.tetrate.io
port: 8443
clusterName: cluster
components:
collector:
kubeSpec:
resources:
limits:
memory: 750Mi
requests:
memory: 500Mi

ControlPlaneComponentSet

The set of components that make up the control plane. Use this to override application settings or Kubernetes settings for each individual component.

FieldDescriptionValidation Rule

collector

tetrateio.api.install.controlplane.v1alpha1.OpenTelemetryCollector

oap

tetrateio.api.install.controlplane.v1alpha1.Oap

zipkin

tetrateio.api.install.controlplane.v1alpha1.Zipkin

xcp

tetrateio.api.install.controlplane.v1alpha1.XCP

istio

tetrateio.api.install.controlplane.v1alpha1.Istio

rateLimitServer

tetrateio.api.install.controlplane.v1alpha1.RateLimitServer

hpaAdapter

tetrateio.api.install.controlplane.v1alpha1.HpaAdapter

onboarding

tetrateio.api.install.controlplane.v1alpha1.Onboarding
Workload Onboarding.

satellite

tetrateio.api.install.controlplane.v1alpha1.Satellite
Satellite provide load balancing capabilities for data content before the data from Envoy reaches the SPM in Control Plane. When envoy points the address to Satellite, it can load balance the traffic to the SPM service.

ngac

tetrateio.api.install.controlplane.v1alpha1.NGAC

gitops

tetrateio.api.install.controlplane.v1alpha1.GitOps
Configuration for the integration of the Control Plane with Continuous Deployment pipelines.

internalCertProvider

tetrateio.api.install.common.InternalCertProvider
Configure the Kubernetes CSR certificate provider for TSB internal purposes like Webhook TLS certificates. This configuration is required for kubernetes version 1.22 and above.

ControlPlaneSpec

ControlPlaneSpec defines the desired installed state of control plane components. Specifying a minimal ControlPlaneSpec with hub, clusterName, and managementPlane set will create an installation with sensible defaults.

FieldDescriptionValidation Rule

hub

string
REQUIRED
TSB container hub path e.g. docker.io/tetrate.

string = {
  min_len: 1
}

imagePullSecrets

List of tetrateio.api.install.kubernetes.LocalObjectReference
Pull secrets can be specified globally for all components, or defined into the kubeSpec.serviceAccount of every component if needed. In case both are defined, the most specific one (the one defined at the component) level is used.

List of references to secrets in the same namespace to use for pulling any images in pods that reference this ServiceAccount. ImagePullSecrets are distinct from Secrets because Secrets can be mounted in the pod, but ImagePullSecrets are only accessed by the kubelet. More info: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#service_account-v1-core

components

tetrateio.api.install.controlplane.v1alpha1.ControlPlaneComponentSet
The set of components that make up the control plane. Use this to override settings for individual components. These components assume the following secrets are present: oap-token, zipkin-token and otel-token.

managementPlane

tetrateio.api.install.controlplane.v1alpha1.ManagementPlaneSettings
REQUIRED
Configure the management plane to retrieve configuration from.

message = {
  required: true
}

meshExpansion

tetrateio.api.install.controlplane.v1alpha1.MeshExpansionSettings
Configure mesh expansion to connect workloads external to Kubernetes to the mesh.

telemetryStore

tetrateio.api.install.controlplane.v1alpha1.ControlPlaneSpec.TelemetryStore
REQUIRED
Configure the store that TSB will use to persist application telemetry data.

message = {
  required: true
}

tier1Cluster

bool
DEPRECATED**: This should not be set through Control plane API Instead use TSB Cluster API. Indicates that this cluster is used for tier1 gateways. Tier one clusters can only contain tier 1 gateways. Non-tier1 clusters contain tier2 gateways but not tier 1.

TelemetryStore

Configure the store that TSB will use to persist application telemetry data. Select one of the TelemetryStore settings to see complete examples.

FieldDescriptionValidation Rule

elastic

tetrateio.api.install.controlplane.v1alpha1.ElasticSearchSettings

ElasticSearchSettings

Configure an Elasticsearch connection.

apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
telemetryStore:
elastic:
host: elastic
port: 5678
protocol: https
selfSigned: true
version: 7
FieldDescriptionValidation Rule

host

string
REQUIRED
Elasticsearch host address (can be hostname or IP address).

string = {
  address: true
}

port

int32
REQUIRED
Port Elasticsearch is listening on.

int32 = {
  lte: 65535
  gte: 1
}

protocol

tetrateio.api.install.controlplane.v1alpha1.ElasticSearchSettings.Protocol
Protocol to communicate with Elasticsearch, defaults to https.

selfSigned

bool
Use Self-Signed certificates. The Self-signed CA bundle and key must be in a secret called es-certs.

version

int32
Major version of the Elasticsearch cluster. Currently supported Elasticsearch major versions are 6 and 7

int32 = {
  lte: 7
  gte: 6
}

Protocol

The list of supported protocols to communicate with Elasticsearch.

NameNumberDescription

https

0

http

1

GitOps

The GitOps component configures the features that allow integrating the control plane cluster with Continuous Deployment pipelines.

FieldDescriptionValidation Rule

enabled

bool
The GitOps component is in beta and disabled by default.

reconcileInterval

google.protobuf.Duration
Interval at which the reconcile process will run. The reconcile process will read all TSB CRs that exist in the cluster and reapply them to the management plane, to make sure the cluster CRs remain as the source of truth. Format: 1h/1m/1s/1ms. A value of 0 disables the reconcile loop. Default: 10m.

batchWindow

google.protobuf.Duration
When configured, all admission requests will be paused for the configured duration. Once the window interval is closed, all paused admission requests will be sent together to the Management Plane as a single request. Batching of requests is disabled by default and should be enabled only if there is high concurrency and ordering of resources could be an issue. By configuring a batch window the concurrency and ordering issues may be mitigated, although it will introduce a constant latency to all requests of the configured time window. When enabled, it is recommended to use a small value, for example 1 second.

HpaAdapter

Kubernetes settings for the OAP (SkyWalking) HPA adapter component.

FieldDescriptionValidation Rule

kubeSpec

tetrateio.api.install.kubernetes.KubernetesComponentSpec
Configure Kubernetes specific settings

Istio

Mesh and Kubernetes settings for Istio.

FieldDescriptionValidation Rule

kubeSpec

tetrateio.api.install.kubernetes.KubernetesIstioComponentSpec
Configure Kubernetes specific settings.

traceSamplingRate

double
The percentage of traces Envoy will sample.

defaultWorkloadCertTTL

google.protobuf.Duration
The default TTL of issued workload certificates. This sets both the default client-side CSR TTL and the default server-side issued certificate TTL.

maxWorkloadCertTTL

google.protobuf.Duration
The maximum TTL that can be set in issued workload certificates.

trustDomain

string
The trust domain corresponds to the trust root of a system. Refer to SPIFFE-ID. If omitted, TSB will configure the trust domain as CLUSTER_NAME.tsb.local, where CLUSTER_NAME is the name of the cluster object in TSB for this control plane.

ManagementPlaneSettings

Configure the management plane connection.

apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
managementPlane:
host: tsb.tetrate.io
port: 8443
selfSigned: true
clusterName: cluster
FieldDescriptionValidation Rule

host

string
REQUIRED
Management plane host address (can be hostname or IPv4/IPv6 address).

port

int32
REQUIRED
Port management plane is listening on.

int32 = {
  lte: 65535
  gte: 1
}

selfSigned

bool
Management plane uses a self signed or private TLS certificate. If true, the CA bundle used to verify the MP's TLS certificate must be in a secret mp-certs under the key ca.crt.

clusterName

string
REQUIRED
The unique identifier for this cluster that was created in the management plane.

string = {
  min_len: 1
}

MeshExpansionSettings

Configure mesh expansion to connect workloads external to Kubernetes to the mesh.

To enable mesh expansion set it to an empty object:

apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
meshExpansion: {}

If external workloads are unable to communicate with the default mesh expansion gateway via external IPs or hostnames, then you must specify the gateway that enables them to do so. This custom gateway must be configured to forward this communcation to the VM gateway service:

apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
meshExpansion:
customGateway:
host: customgateway.tetrate.io
port: 15443

To automate onboarding of workloads from auto-scaling groups of VMs, you need to enable the Workload Onboarding Plane.

Workload Onboarding Agent, a component that you install next to the workload, will connect to the Workload Onboarding Plane to authenticate itself, ask permission to join the mesh, register the workload into the mesh and retrieve boot configuration required to start Istio Sidecar.

All communication between the Workload Onboarding Agent and the Workload Onboarding Plane must occur over TLS.

Therefore, to enable Workload Onboarding Plane you must provide a TLS certificate for the endpoint that exposes Workload Onboarding API to Workload Onboarding Agents.

Make sure that TLS certificate is signed by the certificate authority known to Workload Onboarding Agents.

apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
meshExpansion:
onboarding:
endpoint:
hosts:
- onboarding.example.org
secretName: onboarding-tls-cert
tokenIssuer:
jwt:
expiration: 1h
localRepository: {}

To onboard workloads from custom on-premise environments, you can leverage support for OIDC ID Tokens.

If workloads in your custom environment can authenticate themselves by means of an OIDC ID Token, you can define a list of JWT issuers permitted by the Workload Onboarding Plane.

For example,

apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
meshExpansion:
onboarding:
endpoint:
hosts:
- onboarding.example.org
secretName: onboarding-tls-cert
localRepository: {}
workloads:
authentication:
jwt:
issuers:
- issuer: "https://mycompany.corp"
jwksUri: "https://mycompany.corp/jwks.json"
shortName: "mycorp"
tokenFields:
attributes:
jsonPath: .custom_attributes

To ensure there will be no traffic loss when an onboarded workload gets shutdown, you can configure the time period to delay the shutdown for after deregistering the workload from the mesh, which will give enough time to reconfigure all affected mesh nodes to not load balance requests to the deregistered workload before it becomes unavailable.

For example,

apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
meshExpansion:
onboarding:
endpoint:
hosts:
- onboarding.example.org
secretName: onboarding-tls-cert
localRepository: {}
workloads:
deregistration:
propagationDelay: 15s
FieldDescriptionValidation Rule

customGateway

tetrateio.api.install.controlplane.v1alpha1.MeshExpansionSettings.Gateway
A custom mesh expansion gateway. This is required when the workload can't access the default gateway directly via the external IP or hostname.

onboarding

tetrateio.api.install.controlplane.v1alpha1.MeshExpansionSettings.OnboardingPlane
Configuration of the Workload Onboarding Plane.

Gateway

A custom mesh expansion gateway. This is required when the workload can't access the default gateway directly via the external IP or hostname.

FieldDescriptionValidation Rule

host

string
REQUIRED
Mesh expansion gateway host address (can be hostname or IP address).

string = {
  address: true
}

port

int32
REQUIRED
Port mesh expansion gateway is listening on.

int32 = {
  lte: 65535
  gte: 1
}

OnboardingPlane

Configuration of the Workload Onboarding Plane.

FieldDescriptionValidation Rule

uid

string
Unique identifier of this particular installation of the Workload Onboarding Plane.

Is used in the workload authentication flow to prevent replay attacks that abuse compromised workload credentials intended for a different installation of the Workload Onboarding Plane.

Defaults to an auto-generated UUID.

string = {
  min_len: 1
}

endpoint

tetrateio.api.install.controlplane.v1alpha1.MeshExpansionSettings.OnboardingPlane.Endpoint
REQUIRED
Configuration of the endpoint exposing Workload Onboarding API to Workload Onboarding Agents.

message = {
  required: true
}

tokenIssuer

tetrateio.api.install.controlplane.v1alpha1.MeshExpansionSettings.OnboardingPlane.TokenIssuer
Configuration of the built-in Workload Onboarding Token Issuer.

message = {
  required: true
}

localRepository

tetrateio.api.install.controlplane.v1alpha1.MeshExpansionSettings.OnboardingPlane.LocalRepository
Configuration of the local repository with DEB and RPM packages of the Workload Onboarding Agent and Istio Sidecar.

Local repository is disabled by default. To enable it, set this field to an empty value, i.e. localRepository: {}.

workloads

tetrateio.api.onboarding.config.install.v1alpha1.WorkloadConfiguration
Configuration of the workload handling.

Endpoint

Configuration of the endpoint exposing Workload Onboarding API to Workload Onboarding Agents.

FieldDescriptionValidation Rule

hosts

List of string
REQUIRED
List of hosts included in the TLS certificate.

repeated = {
  min_items: 1
  items: {string:{address:true}}
}

secretName

string
REQUIRED
Name of the secret that holds TLS certificate chain and private key.

string = {
  min_len: 1
}

LocalRepository

Configuration of the local repository with DEB and RPM packages of the Workload Onboarding Agent and Istio Sidecar.

NameNumberDescription

TokenIssuer

Configuration of the built-in Workload Onboarding Token Issuer.

FieldDescriptionValidation Rule

jwt

tetrateio.api.install.controlplane.v1alpha1.MeshExpansionSettings.OnboardingPlane.TokenIssuer.JwtTokenIssuer
Configuration of the built-in JWT Token Issuer.

JwtTokenIssuer

Configuration of the built-in JWT Token Issuer.

FieldDescriptionValidation Rule

expiration

google.protobuf.Duration
Expiration is the duration issued tokens are valid for. Defaults to 1h.

duration = {
  required: true
  gt: {nanos:0}
}

NGAC

Kubernetes settings for the NGAC component.

FieldDescriptionValidation Rule

enabled

bool
NGAC is an experimental component. If enabled is false, this component will not be installed.

kubeSpec

tetrateio.api.install.kubernetes.KubernetesComponentSpec
Configure Kubernetes specific settings

log_levels

map<string, string>
The log level configuration by scopes. Supported log level: "none", "error", "info", "debug".

Oap

Kubernetes settings for the OAP (SkyWalking) component.

FieldDescriptionValidation Rule

streamingLogEnabled

bool
Feature flag to determine whether on-demand streaming logs should be enabled.

onDemandEnvoyMetricsEnabled

bool
Feature flag to determine whether on-demand envoy metrics should be enabled. If enabled, the envoy proxy will provide a set of metrics that can be queried using the metrics service. OAP will provide a query API that can be used to collect envoy proxy metrics for specific pods. This is only for temporary and real-time queries that can be used, for example, for application troubleshooting use cases. These metrics are not persisted.

kubeSpec

tetrateio.api.install.kubernetes.KubernetesComponentSpec
Configure Kubernetes specific settings

Onboarding

Settings for the Workload Onboarding component.

FieldDescriptionValidation Rule

operator

tetrateio.api.install.controlplane.v1alpha1.OnboardingOperator
Configure Workload Onboarding Operator component.

repository

tetrateio.api.install.controlplane.v1alpha1.OnboardingRepository
Configure Workload Onboarding Repository component.

OnboardingOperator

Kubernetes settings for the Workload Onboarding Operator component.

FieldDescriptionValidation Rule

kubeSpec

tetrateio.api.install.kubernetes.KubernetesComponentSpec
Configure Kubernetes specific settings.

OnboardingRepository

Kubernetes settings for the Workload Onboarding Repository component.

FieldDescriptionValidation Rule

kubeSpec

tetrateio.api.install.kubernetes.KubernetesComponentSpec
Configure Kubernetes specific settings.

OpenTelemetryCollector

Kubernetes settings for the OpenTelemetryCollector component.

FieldDescriptionValidation Rule

kubeSpec

tetrateio.api.install.kubernetes.KubernetesComponentSpec
Configure Kubernetes specific settings

RateLimitServer

Configuration settings for the RateLimit Server

FieldDescriptionValidation Rule

backend

tetrateio.api.install.controlplane.v1alpha1.RateLimitServer.Backend
REQUIRED
Configure Database backend settings. This field must be configured by the user.

message = {
  required: true
}

domain

string
The domain field allows ratelimits to be namespaced to a certain domain. To support common ratelimits across multiple clusters set this string to a common value, across them. This assumes that the same backend (uri) is being used. By default the domain is set to the name of the control plane cluster.

kubeSpec

tetrateio.api.install.kubernetes.KubernetesComponentSpec
Configure Kubernetes specific settings.

Backend

External Backend Database types. This points to the backend used by the ratelimit server as a key/value store.

FieldDescriptionValidation Rule

redis

tetrateio.api.install.controlplane.v1alpha1.RateLimitServer.Backend.RedisSettings
Settings for redis database backend.

RedisSettings

Configuration for the External Redis Backend Database

FieldDescriptionValidation Rule

uri

string
REQUIRED
The Redis Database URI. The value of the URI decides the scope for ratelimiting across multiple clusters.

string = {
  min_bytes: 1
}

Satellite

Kubernetes settings for the Satellite (SkyWalking-Satellite) component.

FieldDescriptionValidation Rule

enabled

bool
Satellite is an optional component. If enabled is false, this component will not be installed.

kubeSpec

tetrateio.api.install.kubernetes.KubernetesComponentSpec
Configure Kubernetes specific settings

XCP

Kubernetes settings for the XCP component.

FieldDescriptionValidation Rule

centralAuthMode

tetrateio.api.install.controlplane.v1alpha1.XCP.CentralAuthMode
Authentication mode for connections from XCP Edges to XCP Central. If not set will default to mutual TLS.

configProtection

tetrateio.api.install.common.ConfigProtection
ConfigProtection contains settings for enabling/disabling config protection over XCP created resources. Config protections are disabled by default.

kubeSpec

tetrateio.api.install.kubernetes.KubernetesComponentSpec
Configure Kubernetes specific settings

revision

string
Configures the istio revision tag. If configured, istio upgrade will not be in-place upgrade. A new istio control plane with the configured revision will be deployed. Selectively sidecars and gateways could be moved to newer control plane. Note that it is not the istio version. Istio version is fixed for a particular tsb version and that is not a configurable setting. Revision should be configured to human readable value for example tsb-1-5. For further reference, https://istio.io/latest/blog/2020/multiple-control-planes/#configuring

CentralAuthMode

Authentication mode for connections from XCP Edges to XCP Central

NameNumberDescription

UNKNOWN

0

Default when unset, do not use

MUTUAL_TLS

1

GRPC stream is encrypted with mutual TLS

JWT

2

XCP Edges present a JWT bearer token in the GRPC headers

Zipkin

Kubernetes settings for the Zipkin component.

FieldDescriptionValidation Rule

kubeSpec

tetrateio.api.install.kubernetes.KubernetesComponentSpec
Configure Kubernetes specific settings