Kubernetes

When installing on Kubernetes, these configuration settings can be used to override the default Kubernetes configuration. Kubernetes configuration can be set on each component in the install API using the kubeSpec field.

The API allows for customization of every field in the rendered Kubernetes manifests. The more common configuration fields, such as resources and service type, are supported directly; and can be configured like so:

apiVersion: install.tetrate.io/v1alpha1
kind: ManagementPlane
metadata:
  name: managementplane
spec:
  hub: docker.io/tetrate
  components:
    apiServer:
      kubeSpec:
        service:
          type: LoadBalancer
        deployment:
          resources:
            limits:
              memory: 750Mi
            requests:
              memory: 500Mi

All components have a deployment and service object. Some, such as apiServer, also have a job object associated with them. This can be configured in a similar manner:

apiVersion: install.tetrate.io/v1alpha1
kind: ManagementPlane
metadata:
  name: managementplane
spec:
  hub: docker.io/tetrate
  components:
    apiServer:
      kubeSpec:
        job:
          podAnnotations:
            annotation-key: annotation-value

Not all fields in a Kubernetes manifest can be configured directly. This is to avoid re-implementing the entire Kubernetes API within the install API. Instead, the kubeSpec object provides an overlays mechanism. This field is applied after the operator renders the initial manifests and enables support for customization of any field in a rendered manifest.

Overlays can be applied by selecting the Kubernetes object you wish to overlay and then describe a list of patches you wish to apply. For example, to add a hostPort on port 8443 to the frontEnvoy component, do the following:

apiVersion: install.tetrate.io/v1alpha1
kind: ManagementPlane
metadata:
  name: managementplane
spec:
  hub: docker.io/tetrate
  components:
    frontEnvoy:
      kubeSpec:
        overlays:
        - apiVersion: apps/v1
          kind: Deployment
          name: envoy
          patches:
          - path: spec.template.spec.containers.[name:envoy].ports.[containerPort:8443].hostPort
            value: 8443

The path refers to the location of the field in the Kubernetes object you with to patch. The format is a.[key1:value1].b.[:value2]. Where [key1:value1] is a selector for a key-value pair to identify a list element and [:value] is a value selector to identify a list element in a leaf list. All path intermediate nodes must exist.

Overlays are inspired by and bear a loose resemblence to [kustomize](https://kustomize.io/). We use the library from the Istio Operator. For more examples of how to construct paths take a look at the tests in the upstream.

Affinity

The scheduling constraints for the pod. https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity

Field	Description	Validation Rule
nodeAffinity	tetrateio.api.install.kubernetes.NodeAffinity Group of node affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#nodeaffinity-v1-core	–
podAffinity	tetrateio.api.install.kubernetes.PodAffinity Group of inter-pod affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#podaffinity-v1-core	–
podAntiAffinity	tetrateio.api.install.kubernetes.PodAntiAffinity Group of inter-pod anti-affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#podantiaffinity-v1-core	–

CNI

Configure Istio's CNI plugin For further details see: https://istio.io/docs/setup/additional-setup/cni/

Field	Description	Validation Rule
binaryDirectory	string Directory on the host to install the CNI binary. Must be the same as the environment’s --cni-bin-dir setting (kubelet parameter).	–
configurationDirectory	string Directory on the host to install the CNI config. Must be the same as the environment’s --cni-conf-dir setting (kubelet parameter).	–
chained	bool Whether to deploy the configuration file as a plugin chain or as a standalone file in the configuration directory. Some Kubernetes flavors (e.g. OpenShift) do not support the chain approach.	–
configurationFileName	string Leave unset to auto-find the first file in the cni-conf-dir (as kubelet does). Primarily used for testing install-cni plugin configuration. If set, install-cni will inject the plugin configuration into this file in the cni-conf-dir.	–
clusterRole	string The ClusterRole Istio CNI will bind to in the ControlPlane namespace. This is useful if you use Pod Security Policies and want to allow istio-cni to run as privileged Pods.	–

ClientIPConfig

Field	Description	Validation Rule
timeoutSeconds	int32	–

ConfigMapKeySelector

Field	Description	Validation Rule
localObjectReference	tetrateio.api.install.kubernetes.LocalObjectReference	–
key	string	–
optional	bool	–

ContainerPort

ContainerPort represents a network port in a single container.

Field	Description	Validation Rule
name	string If specified, this must be an IANA_SVC_NAME and unique within the pod. Each named port in a pod must have a unique name. Name for the port that can be referred to by services.	–
hostPort	int32 Number of port to expose on the host. If specified, this must be a valid port number, 0 < x < 65536.	–
containerPort	int32 Number of port to expose on the pod's IP address. This must be a valid port number, 0 < x < 65536.	–
protocol	string Protocol for port. Must be UDP, TCP, or SCTP. Defaults to "TCP".	–
hostIP	string What host IP to bind the external port to.	–

CrossVersionObjectReference

Field	Description	Validation Rule
kind	string	–
name	string	–
apiVersion	string	–

Deployment

The Kubernetes resource configuration for a Deployment

Field	Description	Validation Rule
pod_annotations	map<string, string> Pod annotations are an unstructured key value map stored with the pod. https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
env	List of tetrateio.api.install.kubernetes.EnvVar Environment variables for all containers in the deployment. https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/	–
affinity	tetrateio.api.install.kubernetes.Affinity The scheduling constraints for the pod. https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity	–
replicaCount	uint32 Number of desired pods. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#deploymentspec-v1-apps	–
resources	tetrateio.api.install.kubernetes.Resources Compute Resources required by the primary container in the deployment PodSpec. https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container	–
strategy	tetrateio.api.install.kubernetes.DeploymentStrategy The deployment strategy to use to replace existing pods with new ones. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#deploymentstrategy-v1-apps	–
tolerations	List of k8s.io.api.core.v1.Toleration Tolerations are applied to pods, and allow (but do not require) the pods to schedule onto nodes with matching taints. Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this marks that the node should not accept any pods that do not tolerate the taints. https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/	–
hpaSpec	tetrateio.api.install.kubernetes.HorizontalPodAutoscalerSpec Horizontal Pod Autoscaler automatically scales the number of pods in a deployment based on a specified metric. Kubernetes periodically adjusts the number of replicas in a deployment to match the observed metric to the target specified. The version of Horizontal Pod Autoscaler currently used is [v2beta1](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#horizontalpodautoscaler-v2beta1-autoscaling). https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/	–

DeploymentStrategy

The deployment strategy to use to replace existing pods with new ones. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#deploymentstrategy-v1-apps

Field	Description	Validation Rule
type	string	–
rollingUpdate	tetrateio.api.install.kubernetes.RollingUpdateDeployment	–

EnvVar

Field	Description	Validation Rule
name	string	–
value	string	–
valueFrom	tetrateio.api.install.kubernetes.EnvVarSource	–

EnvVarSource

Field	Description	Validation Rule
fieldRef	tetrateio.api.install.kubernetes.ObjectFieldSelector	–
resourceFieldRef	tetrateio.api.install.kubernetes.ResourceFieldSelector	–
configMapKeyRef	tetrateio.api.install.kubernetes.ConfigMapKeySelector	–
secretKeyRef	tetrateio.api.install.kubernetes.SecretKeySelector	–

ExternalMetricSource

Field	Description	Validation Rule
metricName	string	–
metricSelector	k8s.io.apimachinery.pkg.apis.meta.v1.LabelSelector	–
targetValue	tetrateio.api.install.kubernetes.TypeIntOrString	–
targetAverageValue	tetrateio.api.install.kubernetes.TypeIntOrString	–

HorizontalPodAutoscalerSpec

Horizontal Pod Autoscaler automatically scales the number of pods in a deployment based on a specified metric. Kubernetes periodically adjusts the number of replicas in a deployment to match the observed metric to the target specified. This mirrors the Kubernetes spec except from the top level scaleTargetRef field, which we set for you. The version of Horizontal Pod Autoscaler currently used is [v2beta1](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#horizontalpodautoscaler-v2beta1-autoscaling). https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

Field	Description	Validation Rule
minReplicas	int32 Must be set in order to create the HPA resource in Kubernetes	–
maxReplicas	int32 Must be set in order to create the HPA resource in Kubernetes	–
metrics	List of tetrateio.api.install.kubernetes.MetricSpec	–

Job

The Kubernetes resource configuration for a CronJob or Job

Field	Description	Validation Rule
pod_annotations	map<string, string> Pod annotations are an unstructured key value map stored with the pod. https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
env	List of tetrateio.api.install.kubernetes.EnvVar Environment variables for all containers in the job. https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/	–
affinity	tetrateio.api.install.kubernetes.Affinity The scheduling constraints for the pod. https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity	–
tolerations	List of k8s.io.api.core.v1.Toleration Tolerations are applied to pods, and allow (but do not require) the pods to schedule onto nodes with matching taints. Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this marks that the node should not accept any pods that do not tolerate the taints. https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/	–

KubernetesComponentSpec

KubernetesComponentSpec is a common set of Kubernetes resource configuration for components.

Field	Description	Validation Rule
deployment	tetrateio.api.install.kubernetes.Deployment Settings related to the component deployment	–
service	tetrateio.api.install.kubernetes.Service Settings related to the component service	–
overlays	List of tetrateio.api.install.kubernetes.TypeK8SObjectOverlay Post-render overlays to mutate Kubernetes manifests https://istio.io/latest/docs/reference/config/istio.operator.v1alpha1/#K8sObjectOverlay	–

KubernetesIstioComponentSpec

KubernetesIstioComponentSpec is the common set of Kubernetes resource configuration for Istio. It differs from the standard component specs in that it supports CNI configuration.

Field	Description	Validation Rule
deployment	tetrateio.api.install.kubernetes.Deployment Settings related to the component deployment	–
service	tetrateio.api.install.kubernetes.Service Settings related to the component service	–
CNI	tetrateio.api.install.kubernetes.CNI Configure Istio's CNI plugin For further details see: https://istio.io/docs/setup/additional-setup/cni/	–
overlays	List of tetrateio.api.install.kubernetes.TypeK8SObjectOverlay Post-render overlays to mutate Kubernetes manifests https://istio.io/latest/docs/reference/config/istio.operator.v1alpha1/#K8sObjectOverlay	–

KubernetesJobComponentSpec

KubernetesJobComponentSpec is a common set of Kubernetes resource configuration for components with a job associated with them.

Field	Description	Validation Rule
deployment	tetrateio.api.install.kubernetes.Deployment Settings related to the component deployment	–
service	tetrateio.api.install.kubernetes.Service Settings related to the component service	–
job	tetrateio.api.install.kubernetes.Job Settings related to the component job or cronjob	–
overlays	List of tetrateio.api.install.kubernetes.TypeK8SObjectOverlay Post-render overlays to mutate Kubernetes manifests https://istio.io/latest/docs/reference/config/istio.operator.v1alpha1/#K8sObjectOverlay	–

LocalObjectReference

Field	Description	Validation Rule
name	string	–

MetricSpec

Field	Description	Validation Rule
type	string	–
object	tetrateio.api.install.kubernetes.ObjectMetricSource	–
pods	tetrateio.api.install.kubernetes.PodsMetricSource	–
resource	tetrateio.api.install.kubernetes.ResourceMetricSource	–
external	tetrateio.api.install.kubernetes.ExternalMetricSource	–

NodeAffinity

Group of node affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#nodeaffinity-v1-core

Field	Description	Validation Rule
requiredDuringSchedulingIgnoredDuringExecution	tetrateio.api.install.kubernetes.NodeSelector	–
preferredDuringSchedulingIgnoredDuringExecution	List of tetrateio.api.install.kubernetes.PreferredSchedulingTerm The scheduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#preferredschedulingterm-v1-core	–

NodeSelector

Field	Description	Validation Rule
nodeSelectorTerms	List of tetrateio.api.install.kubernetes.NodeSelectorTerm	–

NodeSelectorRequirement

Field	Description	Validation Rule
key	string	–
operator	string	–
values	List of string	–

NodeSelectorTerm

Field	Description	Validation Rule
matchExpressions	List of tetrateio.api.install.kubernetes.NodeSelectorRequirement	–
matchFields	List of tetrateio.api.install.kubernetes.NodeSelectorRequirement	–

ObjectFieldSelector

Field	Description	Validation Rule
apiVersion	string	–
fieldPath	string	–

ObjectMetricSource

Field	Description	Validation Rule
target	tetrateio.api.install.kubernetes.CrossVersionObjectReference	–
metricName	string	–
targetValue	tetrateio.api.install.kubernetes.TypeIntOrString	–
selector	k8s.io.apimachinery.pkg.apis.meta.v1.LabelSelector	–
averageValue	tetrateio.api.install.kubernetes.TypeIntOrString	–

PodAffinity

Group of inter-pod affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#podaffinity-v1-core

Field	Description	Validation Rule
requiredDuringSchedulingIgnoredDuringExecution	List of tetrateio.api.install.kubernetes.PodAffinityTerm	–
preferredDuringSchedulingIgnoredDuringExecution	List of tetrateio.api.install.kubernetes.WeightedPodAffinityTerm	–

PodAffinityTerm

Field	Description	Validation Rule
labelSelector	k8s.io.apimachinery.pkg.apis.meta.v1.LabelSelector	–
namespaces	List of string	–
topologyKey	string	–

PodAntiAffinity

Group of inter-pod anti-affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#podantiaffinity-v1-core

Field	Description	Validation Rule
requiredDuringSchedulingIgnoredDuringExecution	List of tetrateio.api.install.kubernetes.PodAffinityTerm	–
preferredDuringSchedulingIgnoredDuringExecution	List of tetrateio.api.install.kubernetes.WeightedPodAffinityTerm	–

PodsMetricSource

Field	Description	Validation Rule
metricName	string	–
targetAverageValue	tetrateio.api.install.kubernetes.TypeIntOrString	–
selector	k8s.io.apimachinery.pkg.apis.meta.v1.LabelSelector	–

PreferredSchedulingTerm

Field	Description	Validation Rule
weight	int32	–
preference	tetrateio.api.install.kubernetes.NodeSelectorTerm	–

ResourceFieldSelector

Field	Description	Validation Rule
containerName	string	–
resource	string	–
divisor	tetrateio.api.install.kubernetes.TypeIntOrString	–

ResourceMetricSource

Field	Description	Validation Rule
name	string	–
targetAverageUtilization	tetrateio.api.install.kubernetes.TypeIntOrString	–
targetAverageValue	tetrateio.api.install.kubernetes.TypeIntOrString	–

Resources

Mirrors k8s.io.api.core.v1.ResourceRequirements for unmarshalling.

Field	Description	Validation Rule
limits	map<string, string>
requests	map<string, string>

RollingUpdateDeployment

Mirrors k8s.io.api.apps.v1.RollingUpdateDeployment for unmarshalling.

Field	Description	Validation Rule
maxUnavailable	tetrateio.api.install.kubernetes.TypeIntOrString	–
maxSurge	tetrateio.api.install.kubernetes.TypeIntOrString	–

SecretKeySelector

Field	Description	Validation Rule
localObjectReference	tetrateio.api.install.kubernetes.LocalObjectReference	–
key	string	–
optional	bool	–

Service

The Kubernetes resource configuration for a Service

Field	Description	Validation Rule
annotations	map<string, string> Pod annotations are an unstructured key value map stored with the service. https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
ports	List of tetrateio.api.install.kubernetes.ServicePort List of ports exposed by the component's service. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#serviceport-v1-core	–
type	string Determines how the Service is exposed. Valid options are ExternalName, ClusterIP, NodePort, and LoadBalancer. https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types	–

ServicePort

Field	Description	Validation Rule
name	string	–
protocol	string	–
port	int32	–
targetPort	tetrateio.api.install.kubernetes.TypeIntOrString	–
nodePort	int32	–

SessionAffinityConfig

Field	Description	Validation Rule
clientIP	tetrateio.api.install.kubernetes.ClientIPConfig	–

Toleration

Field	Description	Validation Rule
key	string	–
operator	string	–
value	string	–
effect	string	–
tolerationSeconds	int64	–

TypeIntOrString

GOTYPE: *IntOrString

Name	Number	Description

TypeK8SObjectOverlay

GOTYPE: *K8SObjectOverlayForPB

Name	Number	Description

WeightedPodAffinityTerm

Field	Description	Validation Rule
weight	int32	–
podAffinityTerm	tetrateio.api.install.kubernetes.PodAffinityTerm	–

k8s.io.api.core.v1.Toleration

The pod this Toleration is attached to tolerates any taint that matches the triple <key,value,effect> using the matching operator <operator>.

Field	Description	Validation Rule
key	string Key is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be Exists; this combination means to match all values and all keys. +optional	–
operator	string Operator represents a key's relationship to the value. Valid operators are Exists and Equal. Defaults to Equal. Exists is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category. +optional	–
value	string Value is the taint value the toleration matches to. If the operator is Exists, the value should be empty, otherwise just a regular string. +optional	–
effect	string Effect indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute. +optional	–
tolerationSeconds	int64 TolerationSeconds represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system. +optional	–

k8s.io.apimachinery.pkg.apis.meta.v1.LabelSelector

A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.

Field	Description	Validation Rule
matchLabels	map<string, string> matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
matchExpressions	List of k8s.io.apimachinery.pkg.apis.meta.v1.LabelSelectorRequirement matchExpressions is a list of label selector requirements. The requirements are ANDed. +optional	–

k8s.io.apimachinery.pkg.apis.meta.v1.LabelSelectorRequirement

A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.

Field	Description	Validation Rule
key	string key is the label key that the selector applies to. +patchMergeKey=key +patchStrategy=merge	–
operator	string operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.	–
values	List of string values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. +optional	–