Setting Up Workload Onboarding

This document describes how to setup your environment so that your VMs are ready to be onboarded using the Workload Onboarding Agent.

The setup for Workload Onboarding consists of the following steps:

1. Enable Workload Onboarding
2. Create WorkloadGroup
3. Allow workloads to join WorkloadGroup
4. Create Sidecar configuration
5. Install Workload Onboarding Agent on a VM

Enable Workload Onboarding

To enable Workload Onboarding in a given Kubernetes Cluster, you need to edit TSB ControlPlane resource as follows:

apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
  name: controlplane
  namespace: istio-system
spec:
  ...
  meshExpansion:
    onboarding:                                           # (1) REQUIRED
      endpoint:
        hosts:
        - <onboarding-endpoint-dns-name>                  # (2) REQUIRED
        secretName: <onboarding-endpoint-tls-cert>        # (3) REQUIRED
      tokenIssuer:
        jwt:
          expiration: <onboarding-token-expiration-time>  # (4) OPTIONAL
      localRepository: {}                                 # (5) OPTIONAL

where

1. To enable Workload Onboarding in a given Kubernetes Cluster, you need to edit spec.meshExpansion.onboarding section and provide values for all mandatory fields
2. You must provide DNS name for the Workload Onboarding Endpoint, e.g. onboarding-endpoint.your-company.corp
3. You must provide name of the Kubernetes Secret that holds TLS certificate for the Workload Onboarding Endpoint
4. You can choose custom expiration time for onboarding tokens, defaults to 1 hour
5. You can choose to deploy a local copy of the repository with DEB/RPM packages of the Workload Onboarding Agent and Istio sidecar

Workload Onboarding Endpoint

Workload Onboarding Endpoint is a component that individual Workload Onboarding Agent(s) connect to in order to join the mesh.

In production scenarios, Workload Onboarding Endpoint must be highly available, have a stable address and enforce TLS on connections to it.

For that reason, DNS name and TLS certificate have been made mandatory parameters for enabling Workload Onboarding.

DNS name

You can choose any DNS name for the Workload Onboarding Endpoint.

That name must be associated with the address of Kubernetes Service vmgateway from istio-system namespace.

In production scenarios, you can achieve that by using external-dns.

TLS certificate

To provide a certificate for the Workload Onboarding Endpoint, you need to create a Kubernetes secret of type TLS in the istio-system namespace.

You have several options:

• you can either create a Kubernetes secret from a X509 cert and a private key procured out-of-band
• or you can use cert-manager to automate provisioning of the TLS cert

TLS certificate procured out-of-band

To provide a TLS certificate procured out-of-band, use:

kubectl create secret tls <onboarding-endpoint-tls-cert> \
  -n istio-system \
  --cert=<path/to/cert/file> \
  --key=<path/to/key/file>

TLS certificate procured by cert-manager

To automate provisioning of the TLS certificate, you can use cert-manager.

E.g., this way you can procure a free TLS certificate signed by a trusted CA such as Let's Encrypt.

In that case, your configuration will look similar to:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: <onboarding-endpoint-tls-cert>
  namespace: istio-system
spec:
  secretName: <onboarding-endpoint-tls-cert>
  duration: 2160h   # 90d
  renewBefore: 360h # 15d
  usages:
  - server auth
  dnsNames:
  - <onboarding-endpoint-dns-name>
  issuerRef:
    name: <your-issuer>
    kind: ClusterIssuer

Refer to the cert-manager documentation for further details.

Workload Onboarding Tokens

Workload Onboarding Token represents a temporary grant to onboard a workload into the service mesh.

Workload Onboarding Token is issued by the Workload Onboarding Endpoint after it verifies platform-specific credentials presented by the Workload Onboarding Agent, e.g. credentials of the VM the workload is running on.

Workload Onboarding Token is used as a session token in subsequent requests from the Workload Onboarding Agent to the Workload Onboarding Endpoint to improve efficiency of authentication and authorization.

By default, Workload Onboarding Token is valid for 1 hour.

Users might choose custom expiration time for the Workload Onboarding Token for several reasons, e.g.:

• they can shorten expiration time to meet stricter security policies established within their organization
• they can extend expiration time to lower the load caused by frequent token renewals

Local Repository

For convenience, DEB/RPM packages of the Workload Onboarding Agent and Istio sidecar can be hosted locally within their network.

Once users enable local repository, they will be able to download DEB/RPM packages from an HTTP server at https://<onboarding-endpoint-dns-name>.

Local repository makes available for download the following artifacts:

URI	Description
/install/deb/amd64/onboarding-agent.deb	DEB package of the Workload Onboarding Agent
/install/deb/amd64/onboarding-agent.deb.sha256	SHA-256 checksum of the DEB package
/install/deb/amd64/istio-sidecar.deb	DEB package of the Istio sidecar
/install/deb/amd64/istio-sidecar.deb.sha256	SHA-256 checksum of the DEB package
/install/rpm/amd64/onboarding-agent.rpm	RPM package of the Workload Onboarding Agent
/install/rpm/amd64/onboarding-agent.rpm.sha256	SHA-256 checksum of the RPM package
/install/rpm/amd64/istio-sidecar.rpm	RPM package of the Istio sidecar
/install/rpm/amd64/istio-sidecar.rpm.sha256	SHA-256 checksum of the RPM package

Create WorkloadGroup

When a workload running outside of a Kubernetes cluster is onboarded into the mesh, it is considered to join a certain WorkloadGroup.

Istio WorkloadGroup resource holds configuration shared by all the workloads that join it.

In a way, Istio WorkloadGroup resource is to individual workloads what Kubernetes Deployment resource is to individual Pods.

To be able to onboard individual workloads into a given Kubernetes cluster, you must first create a respective Istio WorkloadGroup in it.

E.g.,

apiVersion: networking.istio.io/v1alpha3
kind: WorkloadGroup
metadata:
  name: ratings
  namespace: bookinfo
  labels:
    app: ratings
spec:
  template:                           # (1)
    labels:
      app: ratings
      class: vm
    serviceAccount: ratings-vm        # (2)
    network:        virtual-machines  # (3)

where

1. All workloads joining that group inherit configuration specified in spec.template
2. Inside the mesh, onboarded workloads will have the identity of the Kubernetes service account specified in spec.template.serviceAccount. If spec.template.serviceAccount is not set, service account "default" is assumed (this account is guaranteed to exist in every Kubernetes namespace).
3. If workloads in that group have no direct connectivity to Kubernetes Pods, you must set field spec.template.network to a non-empty value.

Allow workloads to join WorkloadGroup

Workloads running outside of a Kubernetes cluster are not allowed to join the mesh unless it is explicitly authorized.

For the purposes of onboarding, a workload is considered to have the identity of the host it is running on. For example, if a workload is running on a AWS EC2 instance, it is considered to have the identity of that AWS EC2 instance.

To allow a workload to onboard into a given cluster, a user must create an onboarding policy in that cluster.

OnboardingPolicy is a Kubernetes resource that authorizes workloads with certain identities to join certain WorkloadGroup(s). OnboardingPolicy must be created in the same namespace as the WorkloadGroup(s) it applies to.

Examples

The example below allows workloads running on any AWS EC2 instance associated with the the given AWS account, and can join any of the available WorkloadGroups in the given Kubernetes namespace:

apiVersion: authorization.onboarding.tetrate.io/v1alpha1
kind: OnboardingPolicy
metadata:
  name: allow-any-aws-ec2-instance-from-given-accounts
  namespace: bookinfo
spec:
  allow:
  - workloads:
    - aws:
        accounts:
        - '123456789012'
        - '234567890123'
        ec2: {} # any AWS EC2 instance from the above account(s)
    onboardTo:
    - workloadGroupSelector: {} # any WorkloadGroup from that namespace

For security reasons, AWS accounts must always be white-listed explicitly. You will not be able to specify that workloads associated with any account may freely join the mesh, as it is never a good practice.

While the previous example may have been a rather "permissive" policy, a more restrictive onboarding policy might only allow onboarding workloads from AWS EC2 instances in a particular AWS region and/or zone, with a particular AWS IAM Role, etc. It might also only allow workloads to join a specific subset of WorkloadGroups.

The following shows an example of a more restrictive policy:

apiVersion: authorization.onboarding.tetrate.io/v1alpha1
kind: OnboardingPolicy
metadata:
  name: allow-narrow-subset-of-aws-ec2-instances
  namespace: bookinfo
spec:
  allow:
  - workloads:
    - aws:
        partitions:
        - aws
        accounts:
        - '123456789012'
        regions:
        - us-east-2
        zones:
        - us-east-2b
        ec2:
          iamRoleNames:
          - ratings-role        # any AWS EC2 instance from the above partitions/accounts/regions/zones
                                # that is associated with one of IAM Roles on that list
    onboardTo:
    - workloadGroupSelector:
        matchLabels:
          app: ratings          # (1)

The above policy authorizes workloads that have the label app=ratings (1) to join those WorkloadGroup(s). As an example, the following group would match the policy, but if you omit or specify different values in the label field, it would not match.

apiVersion: networking.istio.io/v1alpha3
kind: WorkloadGroup
metadata:
  name: ratings
  namespace: bookinfo
  labels:
    app: ratings 
spec:
  ...

Create Sidecar configuration

Workload Onboarding does not currently support the use of Iptables for traffic redirection. Therefore you will need to configure Istio sidecar resource, and reconfigure the application and/or the host environment as necessary.

Workload Configuration (Ingress)

Make sure your workload (i.e., user application) listens on 127.0.0.1 instead of 0.0.0.0.

For example, if your application listens on 0.0.0.0:8080, change its configuration so that it listens on 127.0.0.1:8080 instead.

This has two effects: First, both Istio proxy and the workload will be able to listen on the same port — and thus the proxy will be able to listen on <host IP>:8080. And second, other nodes in the mesh will not be able to connect directly to your application. They would be forced to go through the proxy, who in turn will proxy the traffic to 127.0.0.1:8080.

Workload Configuration (Egress)

Configure your application to refer to dependent services by their DNS names. Otherwise the application would not be able to take full advantage of the mesh.

Specifically, the application should refer to other Kubernetes services by their cluster-local DNS names, such as details.bookinfo, details.bookinfo.svc, or details.bookinfo.svc.cluster.local.

Other mesh-external services should be referred to using their DNS names (e.g. example.org)

You will also need to alias the DNS names of your dependent services to the IP addresses of the egress listeners that you will be using, by editing /etc/hosts file in your VM.

Assuming that you are referring to details.bookinfo.svc and example.org in your application, edit /etc/hosts to contain the following lines, replacing the egress listener IP address with appropriate values:

<egress listner IP address> details.bookinfo.svc
<egress listner IP address> example.org

As a result, when your application attempts to make a request to ratings.bookinfo.svc:8080 or example.org:8080, your application would be connecting to the egress listener, which will proxy the requests to their respective destinations.

note

Alternatively, you might consider specifying the Istio proxy in the http_proxy environment variable.

Sidecar Resource Configuration

You will need to create an Istio sidecar resource on your Kubernetes cluster. The YAML definition will look like the following

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: bookinfo-ratings-no-iptables
  namespace: bookinfo
spec:
  workloadSelector:                  # (1)
    labels:
      app: ratings
      class: vm
  ingress:                           # (2)
  - defaultEndpoint: 127.0.0.1:8080
    port:
      name: http
      number: 8080
      protocol: HTTP
  egress:                            # (3)
  - bind: 127.0.0.2
    port:
      number: 8080
    hosts:
    - ./*

Section (1) defines the WorkloadGroups that this sidecar applies to. In this example this configuration applies to workloads whose labels match app: ratings. This example also specifies that we only apply this to those with the class: vm label, which is intended to be used to distinguish workloads deployed on VMs and those deployed on Kubernetes pods.

Section (2) defines the Ingress listener. This configuration specifies that the Istio proxy will be listening on <host IP>:8080, and will forward traffic received on <host IP>:8080 to 127.0.0.1:8080, which should be where your application will be listening.

Section (3) defines the Egress listener. This configuration specifies that the Egress listener will be listening on 127.0.0.2:8080. It also specifies that the Egress listener will proxy outgoing requests to any service that match the hosts list, and has port 8080.

Install Workload Onboarding Agent on a VM

You will need to install the following components on the VM that you want to onboard:

1. Istio sidecar
2. Workload Onboarding Agent

Use either the DEB or RPM package, depending on your preference. You can download these packages from your local repository at https://<onboarding-endpoint-dns-name> (for more details refer to "Enabling Workload Onboarding").

Installing Istio Sidecar DEB package

Run the following commands. Replace the onboarding-endpoint-dns-name with the appropriate value.

curl -LO "https://<onboarding-endpoint-dns-name>/install/deb/amd64/istio-sidecar.deb"

curl -LO "https://<onboarding-endpoint-dns-name>/install/deb/amd64/istio-sidecar.deb.sha256"

sha256sum --check istio-sidecar.deb.sha256

sudo apt-get install -y ./istio-sidecar.deb

rm istio-sidecar.deb istio-sidecar.deb.sha256

Installing Istio Sidecar RPM Package

Run the following commands. Replace the onboarding-endpoint-dns-name with the appropriate value.

curl -LO "https://<onboarding-endpoint-dns-name>/install/rpm/amd64/istio-sidecar.rpm"

curl -LO "https://<onboarding-endpoint-dns-name>/install/rpm/amd64/istio-sidecar.rpm.sha256"

sha256sum --check istio-sidecar.rpm.sha256

sudo yum install -y ./istio-sidecar.rpm

rm istio-sidecar.rpm istio-sidecar.rpm.sha256

Installing Workload Onboarding Agent DEB Package

Run the following commands. Replace the onboarding-endpoint-dns-name with the appropriate value.

curl -LO "https://<onboarding-endpoint-dns-name>/install/deb/amd64/onboarding-agent.deb"

curl -LO "https://<onboarding-endpoint-dns-name>/install/deb/amd64/onboarding-agent.deb.sha256"

sha256sum --check onboarding-agent.deb.sha256

sudo apt-get install -y ./onboarding-agent.deb

rm onboarding-agent.deb onboarding-agent.deb.sha256

Installing Workload Onboarding Agent RPM Package

Run the following commands. Replace the onboarding-endpoint-dns-name with the appropriate value.

curl -LO "https://<onboarding-endpoint-dns-name>/install/rpm/amd64/onboarding-agent.rpm"

curl -LO "https://<onboarding-endpoint-dns-name>/install/rpm/amd64/onboarding-agent.rpm.sha256"

sha256sum --check onboarding-agent.rpm.sha256

sudo yum install -y ./onboarding-agent.rpm

rm onboarding-agent.rpm onboarding-agent.rpm.sha256