Managing Onboarded Workloads

Workload Naming

Workloads onboarded into the mesh are represented by the Kubernetes resource WorkloadAutoRegistration.

When a new workload is onboarded into the mesh and joins a given WorkloadGroup, the Workload Onboarding Endpoint creates a WorkloadAutoRegistration resource in the namespace of that WorkloadGroup.

Each WorkloadAutoRegistration resource is assigned a unique name in the format:

<workload-group-name>-<workload-identity>

Where workload-identity is a unique name generated by TSB. For workload running on a AWS EC2 instance, its workload-identity would be in the format

aws-<aws-partition>-<aws-account>-<aws-zone>-ec2-<aws-ec2-instance-id>

Put together, a workload's unique name may look like the following:

ratings-aws-aws-123456789012-us-east-2b-ec2-i-1234567890abcdef0

Listing Onboarded Workloads

To list onboarded workloads, issue a kubectl get command for the war resource. war is an alias for WorkloadAutoRegistration.

The following command will list the workloads registered across all Kubernetes namespaces:

kubectl get war -A 

You will see an output similar to the one below:

NAMESPACE   NAME                                                              AGENT CONNECTED   AGE
bookinfo    ratings-aws-aws-123456789012-us-east-2b-ec2-i-1234567890abcdef0   True              1m

The AGENT CONNECTED column is showing the status of the Workload Onboarding Agent. If the value is True, the agent is currently connected to the Workload Onboarding Endpoint, and the workload is considered to be healthy. If the value is False, the agent is no longer connected. The workload itself may or may not be healthy.

Describing Onboarded Workload

To see details of the onboarded workload, run the kubectl describe war command:

kubectl describe war <war-name>

You will see an output similar to the one below:

Name:         ratings-aws-aws-123456789012-us-east-2b-ec2-i-1234567890abcdef0
Namespace:    bookinfo
API Version:  runtime.onboarding.tetrate.io/v1alpha1
Kind:         WorkloadAutoRegistration
Spec:
  Identity:                                # (1)
    Aws:
      Account:  123456789012
      ec2:
        Instance Id:  i-1234567890abcdef0
      Partition:      aws
      Region:         us-east-2
      Zone:           us-east-2b
  Registration:
    Agent:
      Version:  v1.4.0
    Host:
      Addresses:
        Ip:    172.31.5.254
        Type:  VPC
    Settings:
      Connected Over:  VPC
    Sidecar:
      Istio:
        Version:  1.9.8-15bc6e5e32
    Workload:
      Labels:
        Version:  v5
Status:
  Conditions:
    Last Transition Time:  2021-10-09T10:56:41.380102645Z
    Reason:                AgentEstablishedConnection
    Status:                True
    Type:                  AgentConnected

The Spec.Identity section (1) describes the verified identity of the workload, which in this case is the identity of the VM the workload is running on. This information may be useful in verifying the origin of the onboarded workloads, as opposed to trusting the information reported by the workloads themselves.

Checking the Status of Istio Sidecar

You can use istioctl proxy-status command to check status of the Istio sidecar of the onboarded workload.

Run:

istioctl proxy-status

You should get output similar to:

NAME                                                                         CDS        LDS        EDS        RDS        ISTIOD                      VERSION
ratings-aws-aws-123456789012-us-east-2b-ec2-i-1234567890abcdef0.bookinfo     SYNCED     SYNCED     SYNCED     SYNCED     istiod-6449df9b98-prvqd     1.9.8-15bc6e5e32
...

istioctl proxy-status command displays the status of all Istio proxies (both sidecars and gateways) currently connected to the Istio Control Plane.

The name for the sidecars will be the same as the name of the workload.

Auto-removal of Onboarded Workloads

The Workload Onboarding Endpoint component is not aware of the lifecycle of workloads registered through the Workload Onboarding Agent. For example if an AWS EC2 instance the workload is running on is terminated, the Workload Onboarding Endpoint is only aware of the fact that the Workload Onboarding Agent is no longer connected to it.

In order to avoid retaining dangling workload registrations indefinitely, the Workload Onboarding Endpoint considers a workload as being no longer active once a Workload Onboarding Agent disconnects and does not reconnect within a pre-configured grace period. The default value for this grace period is 5 minutes.

Updating WorkloadGroups

Istio WorkloadGroup used in the Workload Onboarding feature plays a similar role to a Kubernetes Deployment. The WorkloadGroup is used to define a template for configuration used in each individual instance in a group.

There is an important difference between Kubernetes Deployments and WorkloadGroups. While the former is backed by a controller logic that knows how to gradually replace Pods with new instances using the new configuration and roll out changes made to a Deployment resource, the latter does not have such capabilities.

This means that while any changes you make to a WorkloadGroup will affect future workloads joining that group, workloads that have previously joined retain their old configuration.

Applying WorkloadGroup Updates

A WorkloadGroup defines the core set of configurations of an Istio sidecar that cannot be updated without a restart of the sidecar.

Therefore if we are to apply new configurations to all workloads at once, all workloads would have to be terminated at the same time. This would not be safe for production environments.

Similarly, it would not be safe to apply configurations when the Istio sidecar reconnects to the Control Plane, since there is a possibility that multiple sidecars reconnect simultaneously due to network hiccups.

On the other hand, an individual workload onboarded into the mesh is represented by the WorkloadAutoRegistration resource.

To guarantee that core configuration of the workload's Istio sidecar remains stable at all times, a WorkloadAutoRegistration carries the snapshot of the WorkloadGroup taken at the moment when the workload joined the mesh.

A WorkloadAutoRegistration captures principal information about the workload, such as IP address(es), use of traffic redirection, etc, which all affect core configuration of the Istio sidecar for that workload.

Therefore, for a workload to observe new changes to the WorkloadGroup after the WorkloadAutoRegistration has been created, the WorkloadAutoRegistration needs to be deleted, and re-created.

In summary, you should delete the WorkloadAutoRegistration resource after you make updates to a WorkloadGroup. Run the following command after making changes to a WorkloadGroup:

kubectl delete war <war-name> 

Removing WorkloadAutoRegistration resource will cause the Workload Onboarding Agent to go through the Workload Onboarding flow again, which in turn will re-create the WorkloadAutoRegistration. This will then capture the latest version of the WorkloadGroup configuration.