Resource Consumption and Capacity Planning
This document describes a conservative guideline for capacity planning of Tetrate Service Bridge (TSB) in Management and Control planes.
These parameters apply to production installations: TSB will run with minimal resources if you are using a demo-like environment.
disclaimer
The resource provisioning guidelines described in this document are very conservative.
Also please be aware that the resource provisioning described in this document are applicable to vertical resource scaling. Multiple replicas of the same TSB components do not share the load with each other, and therefore you cannot expect the combined resources from multiple components to have the same effect. Replicas of TSB components should only be used for high availability purposes only.
Recommended baseline production installation resource requirements
For a baseline installation of TSB with 1 registered cluster and 1 deployed service within that cluster, the following resources are recommended.
To reiterate, the amount of memory described below are very conservative. Also, the actual performance given by the number of vCPUs tend to fluctuate depending on your underlying infrastructure. You are advised to verify the results in your environment.
| Component | vCPU # | Memory MiB |
|---|---|---|
| TSB server (Management Plane) 1 | 2 | 512 |
| XCP Central Components 2 | 2 | 128 |
| XCP Edge | 1 | 128 |
| Front Envoy | 1 | 50 |
| IAM | 1 | 128 |
| TSB UI | 1 | 256 |
| OAP | 4 | 5192 |
| OTEL-collector | 2 | 1024 |
| Zipkin | 2 | 2048 |
2 Including the Kubernetes operator.
Recommended scaling resource parameters
The TSB stack is mostly CPU-bound. Additional clusters registered with TSB via XCP increase the CPU utilization by ~4%.
The effect of additional registered clusters or additional deployed workload services on memory utilisation is almost negligible. Likewise, the effect of additional clusters or workloads on resource consumption of the majority of TSB components is mostly negligible, with the notable exceptions of TSB, XCP Central component, TSB UI and IAM.
note
Components that are part of the visibility stack (e.g. OTel/Zipkin, etc.) have their resource utilisation driven by requests, thus the resource scaling should follow the user request rate statistics. As a general rule of thumb, more than 1 vCPU is preferred. It is also important to notice that the visibility stack performance is largely bound by Elasticsearch performance.
Thus, we recommend vertically scaling the components by 1 vCPU for a number of deployed workflows:
Management Plane
Besides OAP, All components don't require any resource adjustment. Those components are architectured and tested to support very large clusters.
OAP in Management plane requires extra CPU and Memory ~ 100 millicores of CPU and 1024 MiB of RAM per every 1000 services. E.g. 4000 services aggregated in TSB Management Plane from all TSB clusters would require approximately 400 millicores of CPU and 4096 MiB of RAM in total.
Control Plane Resource Requirements
Following table shows typical peak resource utilization for TSB control plane with the following assumptions:
- 50 services with sidecars
- Traffic on entire cluster is 500 repository
- Zipkin sampling rate is 1% of the traffic
- Metric is captured for every request at every workload.
Note that average CPU utilization would be a fraction of the typical peak value.
| Component | Typical Peak CPU (m) | Typical Peak Memory (Mi) |
|---|---|---|
| Istiod | 300m | 250Mi |
| OAP | 2500m | 2500Mi |
| Zipkin | 200m | 1000Mi |
| XCP Edge | 100m | 100Mi |
| Istio Operator - Control Plane | 50m | 100Mi |
| Istio Operator - Data Plane | 150m | 100Mi |
| TSB Control Plane Operator | 100m | 100Mi |
| TSB Data Plane Operator | 150m | 100Mi |
| OTEL Collector | 50m | 100Mi |
TSB/Istio Operator resource usage per Ingress Gateway
The following table shows the resources used by TSB Operator and Istio Operator per Ingress Gateways
note
Keep in mind that these are estimated numbers depending on your application deployed, this can vary, but you can have a general idea of the consumption with these values
| Ingress Gateways | TSB Operator CPU(m) | TSB Operator Mem(Mi) | Istio Operator CPU(m) | Istio Operator Mem(Mi) |
|---|---|---|---|---|
| 0 | 100m | 50Mi | 10m | 45Mi |
| 50 | 2600m | 125Mi | 1100m | 120Mi |
| 100 | 3500m | 200Mi | 1300m | 175Mi |
| 150 | 3800m | 250Mi | 1400m | 200Mi |
| 200 | 4000m | 325Mi | 1400m | 250Mi |
| 250 | 4700m | 325Mi | 1750m | 300Mi |
| 300 | 5000m | 475Mi | 1750m | 400Mi |
Component resource utilization
The following tables will show how the different components of TSB scale with 4000 services and peaking with 60 rpm, this is divided by information from the Management Plane, and the Control Plane.
Management Plane
| Services | Gateways | Traffic(rpm) | Central CPU(m) | Central Mem(Mi) | MPC CPU(m) | MPC Mem(Mi) | OAP CPU(m) | OAP Mem(Mi) | Otel CPU(m) | Otel Mem(Mi) | TSB CPU(m) | TSB Mem(Mi) | Zipkin CPU(m) | Zipkin Mem(Mi) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0 rpm | 3m | 39Mi | 5m | 30Mi | 37m | 408Mi | 22m | 108Mi | 14m | 57Mi | 2m | 708Mi |
| 420 | 7 | 600 rpm | 4m | 42Mi | 15m | 31Mi | 116m | 736Mi | 24m | 123Mi | 50m | 63Mi | 14m | 835Mi |
| 820 | 9 | 600 rpm | 4m | 54Mi | 24m | 34Mi | 43m | 909Mi | 26m | 127Mi | 85m | 75Mi | 25m | 948Mi |
| 1220 | 11 | 600 rpm | 4m | 59Mi | 32m | 41Mi | 28m | 1141Mi | 27m | 210Mi | 213m | 78Mi | 25m | 954Mi |
| 1620 | 13 | 600 rpm | 5m | 63Mi | 44m | 48Mi | 209m | 1475Mi | 29m | 249Mi | 113m | 86Mi | 25m | 957Mi |
| 2020 | 15 | 600 rpm | 5m | 73Mi | 41m | 51Mi | 51m | 1655Mi | 24m | 319Mi | 211m | 91Mi | 27m | 957Mi |
| 2420 | 17 | 300 rpm | 4m | 84Mi | 72m | 62Mi | 57m | 1910Mi | 29m | 381Mi | 227m | 97Mi | 27m | 755Mi |
| 2820 | 19 | 60 rpm | 5m | 90Mi | 73m | 65Mi | 43m | 2136Mi | 16m | 466Mi | 275m | 104Mi | 27m | 770Mi |
| 3220 | 21 | 60 rpm | 5m | 106Mi | 85m | 78Mi | 89m | 2600Mi | 43m | 574Mi | 382m | 108Mi | 27m | 802Mi |
| 3620 | 23 | 60 rpm | 5m | 123Mi | 94m | 71Mi | 245m | 2772Mi | 37m | 578Mi | 625m | 115Mi | 27m | 825Mi |
| 4020 | 25 | 60 rpm | 5m | 147Mi | 90m | 81Mi | 521m | 3224Mi | 15m | 704Mi | 508m | 122Mi | 27m | 856Mi |
note
IAM will peak at 509m/52Mi, LDAP at 2m/17Mi and XCP Operator at 9m/37Mi
Control Plane
| Services | Gateways | Traffic(rpm) | Edge CPU(m) | Edge Mem(Mi) | Istiod CPU(m) | Istiod Mem(Mi) | OAP CPU(m) | OAP Mem(Mi) | Otel CPU(m) | Otel Mem(Mi) | Zipkin CPU(m) | Zipkin Mem(Mi) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0 rpm | 6m | 49Mi | 9m | 53Mi | 48m | 610Mi | 26m | 80Mi | 25m | 723Mi |
| 400 | 2 | 600 rpm | 350m | 120Mi | 600m | 600Mi | 900m | 1510Mi | 27m | 86Mi | 75m | 931Mi |
| 800 | 4 | 600 rpm | 700m | 230Mi | 2170m | 1140Mi | 1720m | 2310Mi | 32m | 91Mi | 123m | 1030Mi |
| 1200 | 6 | 600 rpm | 1010m | 366Mi | 2680m | 1890Mi | 2630m | 3280Mi | 35m | 101Mi | 139M | 1080Mi |
| 1600 | 8 | 600 rpm | 1600m | 438Mi | 2690m | 2490Mi | 3610m | 4030Mi | 41m | 180Mi | 180m | 1070Mi |
| 2000 | 10 | 600 rpm | 1900m | 514Mi | 3240m | 3820Mi | 4470m | 5890Mi | 43m | 106Mi | 209m | 1080Mi |
| 2400 | 12 | 300 rpm | 682m | 628Mi | 2010m | 4660Mi | 3910m | 5750Mi | 37m | 110Mi | 281m | 1070Mi |
| 4000 | 20 | 600 rpm | 1470m | 1040Mi | 3730m | 9790Mi | 13300m | 35000Mi | 37m | 135Mi | 465m | 1100Mi |
note
Metric Server will peak at 11m/32Mi, Onboarding Operator at 6m/38Mi, and XCP-Operator at 11m/46Mi