Skip to main content
Version: 1.5.x

Canary deployments using ArgoRollout

This document describes how you can configure ArgoCD & integrate ArgoRollout with TSB GitOps support for canary deployment automation

Before you get started, make sure:
ArgoCD is installed in your cluster and ArgoCD CLI is configured to connect to your ArgoCD server
ArgoRollout is installed in your cluster
✓ TSB is up and running, and GitOps has been enabled for the target cluster

Create an Application from a Git repository

Create a sample application using the below command. An example repository containing Istio's bookinfo application and TSB configurations is available at https://github.com/tetrateio/tsb-gitops-demo. You can either use ArgoCD CLI or their web UI to import application configurations directly from Git.

argocd app create bookinfo-app --repo https://github.com/tetrateio/tsb-gitops-demo.git --path application --dest-server https://kubernetes.default.svc --dest-namespace cms-bookinfo

Check the status of your application

argocd app get bookinfo-app
Name:               bookinfo-app
Project: default
Server: https://kubernetes.default.svc
Namespace: cms-bookinfo
URL: https://localhost:8080/applications/bookinfo-app
Repo: https://github.com/tetrateio/tsb-gitops-demo.git
Target:
Path: application
SyncWindow: Sync Allowed
Sync Policy: <none>
Sync Status: Synced to (9d20ee2)
Health Status: Healthy

GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
Namespace cms-bookinfo cms-bookinfo Running Synced namespace/cms-bookinfo configured
ServiceAccount cms-bookinfo bookinfo-details Synced serviceaccount/bookinfo-details configured
ServiceAccount cms-bookinfo bookinfo-productpage Synced serviceaccount/bookinfo-productpage configured
ServiceAccount cms-bookinfo bookinfo-ratings Synced serviceaccount/bookinfo-ratings configured
ServiceAccount cms-bookinfo bookinfo-reviews Synced serviceaccount/bookinfo-reviews configured
Service cms-bookinfo details Synced Healthy service/details configured
Service cms-bookinfo productpage Synced Healthy service/productpage configured
Service cms-bookinfo reviews Synced Healthy service/reviews configured
Service cms-bookinfo ratings Synced Healthy service/ratings configured
apps Deployment cms-bookinfo productpage-v1 Synced Healthy deployment.apps/productpage-v1 configured
apps Deployment cms-bookinfo reviews Synced Healthy deployment.apps/reviews configured
apps Deployment cms-bookinfo ratings-v1 Synced Healthy deployment.apps/ratings-v1 configured
apps Deployment cms-bookinfo details-v1 Synced Healthy deployment.apps/details-v1 configured
Namespace cms-bookinfo Synced

Application Setup

If you already have kubernetes manifests created for deployment and service resource, You can choose to keep the same objects along with Argo Rollout object for facilitating the canary deployments. You can make necessary changes to Rollout object and TSB mesh configuration of Istio VirtualService/DestinationRule to achieve the desired result.

TSB Configuration Setup

Since ArgoRollout require you to make some modifications on VS/DR object according to their Canary deployment strategy convention for Istio, You can use TSB DIRECT mode configuration to achieve the desired result.

  • According to ArgoRollout convention, 2 subsets named stable and canary has been added to subsets with necessary labels to identify canary and stable pods which ArgoRollout inject based on canaryMetadata.
  • Please make sure the version labels eg: version: canary/stable has been added to canaryMetadata according to Istio convention for TSB to recognize the subsets and plot the metrics in service dashboard.
  • When using Istio direct mode resources with GitOps, there is an additional label istio.io/rev: "tsb" that needs to be added to the resources. Please refer here for more details.

Create a bookinfo-tsb-conf app by importing the TSB configurations from tsb-gitops-demo/canary/tsb/conf.yaml. You can also choose to keep it in the same repo.

argocd app create bookinfo-tsb-conf --repo https://github.com/tetrateio/tsb-gitops-demo.git --path canary/tsb --dest-server https://kubernetes.default.svc --dest-namespace cms-bookinfo

Check the status of TSB resources

argocd app get bookinfo-tsb-conf

Name: bookinfo-tsb-conf
Project: default
Server: https://kubernetes.default.svc
Namespace: cms-bookinfo
URL: https://localhost:8080/applications/bookinfo-tsb-conf
Repo: https://github.com/tetrateio/tsb-gitops-demo.git
Target:
Path: canary/tsb
SyncWindow: Sync Allowed
Sync Policy: <none>
Sync Status: OutOfSync from (9d20ee2)
Health Status: Healthy

GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
networking.istio.io VirtualService cms-bookinfo bookinfo Synced virtualservice.networking.istio.io/bookinfo configured
networking.istio.io Gateway cms-bookinfo bookinfo-gateway Synced gateway.networking.istio.io/bookinfo-gateway configured
security.tsb.tetrate.io Group cms-bookinfo bookinfo-security Synced group.security.tsb.tetrate.io/bookinfo-security configured
traffic.tsb.tetrate.io Group cms-bookinfo bookinfo-traffic Synced group.traffic.tsb.tetrate.io/bookinfo-traffic configured
gateway.tsb.tetrate.io Group cms-bookinfo bookinfo-gateway Synced group.gateway.tsb.tetrate.io/bookinfo-gateway configured
tsb.tetrate.io Workspace cms-bookinfo bookinfo-ws Synced workspace.tsb.tetrate.io/bookinfo-ws configured
tsb.tetrate.io Tenant cms-bookinfo cms Synced tenant.tsb.tetrate.io/cms configured
networking.istio.io DestinationRule cms-bookinfo details Synced destinationrule.networking.istio.io/details configured
networking.istio.io VirtualService cms-bookinfo details Synced virtualservice.networking.istio.io/details configured
networking.istio.io DestinationRule cms-bookinfo ratings Synced destinationrule.networking.istio.io/ratings configured
networking.istio.io DestinationRule cms-bookinfo productpage Synced destinationrule.networking.istio.io/productpage configured
networking.istio.io VirtualService cms-bookinfo ratings Synced virtualservice.networking.istio.io/ratings configured
networking.istio.io DestinationRule cms-bookinfo reviews Synced destinationrule.networking.istio.io/reviews configured
networking.istio.io VirtualService cms-bookinfo reviews Synced virtualservice.networking.istio.io/reviews configured
install.tetrate.io IngressGateway cms-bookinfo tsb-gateway-bookinfo Synced ingressgateway.install.tetrate.io/tsb-gateway-bookinfo configured

Verify application

Run the below command to export LB ip of tsb-gateway-bookinfo

export GATEWAY_IP=$(kubectl -n cms-bookinfo get service tsb-gateway-bookinfo -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Confirm that you can access bookinfo application. As you can see in the response, review v1 service which we have currently deployed doesn't call ratings service.

curl -v "http://bookinfo.tetrate.com/api/v1/products/1/reviews" \
--resolve "bookinfo.tetrate.com:80:$GATEWAY_IP"

* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< content-length: 361
< server: istio-envoy
< date: Mon, 22 Aug 2022 06:36:52 GMT
< x-envoy-upstream-service-time: 782
<
* Connection #0 to host bookinfo.tetrate.com left intact
{"id": "1", "podname": "reviews-rollout-56ff4b868c-74d8t", "clustername": "null", "reviews": [{"reviewer": "Reviewer1", "text": "An extremely entertaining play by Shakespeare. The slapstick humour is refreshing!"}, {"reviewer": "Reviewer2", "text": "Absolutely fun and entertaining. The play lacks thematic depth when compared to other plays by Shakespeare."}]}

Setup ArgoRollout

ArgoRollout provides multiple options to migrate your existing kubernetes deployment object into Argo rollout object. You can either convert an existing deployment object to rollout or you can refer your existing deployment from a rollout object using workloadRef. We will be following the latter approach in this example.

In this example we will be doing a canary deployment of reviews service to demonstrate rollout object configurations and how it is facilitating the traffic shifting to both primary and canary deployment of reviews service.

  • Create a Rollout resource and refer your existing deployment using workloadRef.
  • Make sure selector matchLabels has been configured based on your k8s application deployment manifest.
  • Configure rollout strategy to use canary with subset level traffic splitting.
  • Configure canaryMetadata to inject labels and annotations on canary and stable pods.
  • Configure Istio virtualService and destinationRule based on TSB configuration.
  • Once the Rollout object is created, it will spin up the required number of pods side-by-side along with the Deployment pods and then you can scale down your existing deployment to 0 by changing the replicas.
  • Rollout object won't modify your existing Deployment, Traffic would be shifted to the pods managed by Rollout object once the subset is updated in VirtualService.

Example Configuration

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: reviews-rollout
spec:
replicas: 5
selector:
matchLabels:
app: reviews
workloadRef:
apiVersion: apps/v1
kind: Deployment
name: reviews
strategy:
canary:
canaryMetadata:
annotations:
version: canary
labels:
version: canary
service.istio.io/canonical-revision: canary
stableMetadata:
annotations:
version: stable
labels:
version: stable
service.istio.io/canonical-revision: stable
trafficRouting:
istio:
virtualService:
name: reviews
destinationRule:
name: reviews
canarySubsetName: canary
stableSubsetName: stable
steps:
- setWeight: 10
- pause: {}
- setWeight: 20
- pause: {duration: 10}
- setWeight: 40
- pause: {duration: 10}
- setWeight: 60
- pause: {duration: 10}
- setWeight: 80
- pause: {duration: 10}

Create Rollout

Run the below command to create a rollout app

argocd app create reviews-rollout --repo https://github.com/tetrateio/tsb-gitops-demo.git --path canary/rollout --dest-server https://kubernetes.default.svc --dest-namespace cms-bookinfo

Check the status

argocd app get reviews-rollout

Name: reviews-rollout
Project: default
Server: https://kubernetes.default.svc
Namespace: cms-bookinfo
URL: https://localhost:8080/applications/reviews-rollout
Repo: https://github.com/tetrateio/tsb-gitops-demo.git
Target:
Path: canary/rollout
SyncWindow: Sync Allowed
Sync Policy: <none>
Sync Status: Synced to (bab126b)
Health Status: Healthy

GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
argoproj.io Rollout cms-bookinfo reviews-rollout Synced Healthy rollout.argoproj.io/reviews-rollout created

Trigger Canary Deployment

Update the reviews service deployment image to v2 version. This will immediately trigger a canary deployment of reviews v2 and will modify the traffic percentage as 90/10.

kubectl argo rollouts set image reviews-rollout reviews=docker.io/istio/examples-bookinfo-reviews-v2:1.16.4 -n cms-bookinfo

Monitor Canary Deployment

Run the below command to monitor your canary deployment.

kubectl argo rollouts get rollout reviews-rollout --watch -n cms-bookinfo

Name: reviews-rollout
Namespace: cms-bookinfo
Status: ॥ Paused
Message: CanaryPauseStep
Strategy: Canary
Step: 1/10
SetWeight: 10
ActualWeight: 10
Images: docker.io/istio/examples-bookinfo-reviews-v1:1.16.4 (stable)
docker.io/istio/examples-bookinfo-reviews-v2:1.16.4 (canary)
Replicas:
Desired: 5
Current: 6
Updated: 1
Ready: 6
Available: 6

NAME KIND STATUS AGE INFO
⟳ reviews-rollout Rollout ॥ Paused 20m
├──# revision:2
│ └──⧉ reviews-rollout-5986899db8 ReplicaSet ✔ Healthy 80s canary
│ └──□ reviews-rollout-5986899db8-pxnjp Pod ✔ Running 80s ready:2/2
└──# revision:1
└──⧉ reviews-rollout-56ff4b868c ReplicaSet ✔ Healthy 20m stable
├──□ reviews-rollout-56ff4b868c-5vrn6 Pod ✔ Running 20m ready:2/2
├──□ reviews-rollout-56ff4b868c-74d8t Pod ✔ Running 20m ready:2/2
├──□ reviews-rollout-56ff4b868c-fphf5 Pod ✔ Running 20m ready:2/2
├──□ reviews-rollout-56ff4b868c-fsrfd Pod ✔ Running 20m ready:2/2
└──□ reviews-rollout-56ff4b868c-t4299 Pod ✔ Running 20m ready:2/2

Generate traffic

Run the below command to send some requests to bookinfo application.

while true; do curl -m 5 -v "http://bookinfo.tetrate.com/api/v1/products/1/reviews" --resolve "bookinfo.tetrate.com:80:$GATEWAY_IP";  sleep 2 ; done ;

As you can see, some of the response will have the response from ratings service as reviews-v2 calls ratings service.

> GET /api/v1/products/1/reviews HTTP/1.1
> Host: bookinfo.tetrate.com
> User-Agent: curl/7.79.1
> Accept: */*
> Content-Length: 0
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< content-length: 437
< server: istio-envoy
< date: Mon, 22 Aug 2022 06:53:14 GMT
< x-envoy-upstream-service-time: 45
<
* Connection #0 to host bookinfo.tetrate.com left intact
{"id": "1", "podname": "reviews-66f8dddb8c-84pk6", "clustername": "null", "reviews": [{"reviewer": "Reviewer1", "text": "An extremely entertaining play by Shakespeare. The slapstick humour is refreshing!", "rating": {"stars": 5, "color": "black"}}, {"reviewer": "Reviewer2", "text": "Absolutely fun and entertaining. The play lacks thematic depth when compared to other plays by Shakespeare.", "rating": {"stars": 4, "color": "black"}}]}

Monitor Performance Metrics in TSB

You can monitor the health of each service instance of both canary and stable pods from TSB service dashboard.

service dashboard which shows both stable and canary subsets
service dashboard which shows both stable and canary subsets

service instance metrics for subset canary
service instance metrics for subset canary

service instance metrics for subset stable
service instance metrics for subset stable

service topology which shows only reviews-canary is calling details-v1 service
service topology which shows only reviews-canary is calling details-v1 service

service tracability of reviews service calling details and details of the tags
service tracability of reviews service calling details and details of the tags

Promote Canary Deployment

You can either do a step promote which will proceed to the next steps mentioned in the Rollout by changing the traffic weight and it will eventually rollout the new version completely or you can do a full promote to the desired version by skipping analysis, pauses, and steps.

# step promotion
kubectl argo rollouts promote reviews-rollout -n cms-bookinfo

# full promotion
kubectl argo rollouts promote reviews-rollout --full -n cms-bookinfo
kubectl argo rollouts get rollout reviews-rollout --watch -n cms-bookinfo

Name: reviews-rollout
Namespace: cms-bookinfo
Status: ॥ Paused
Message: CanaryPauseStep
Strategy: Canary
Step: 5/10
SetWeight: 40
ActualWeight: 40
Images: docker.io/istio/examples-bookinfo-reviews-v1:1.16.4 (stable)
docker.io/istio/examples-bookinfo-reviews-v2:1.16.4 (canary)
Replicas:
Desired: 5
Current: 7
Updated: 2
Ready: 7
Available: 7

NAME KIND STATUS AGE INFO
⟳ reviews-rollout Rollout ॥ Paused 24m
├──# revision:2
│ └──⧉ reviews-rollout-5986899db8 ReplicaSet ✔ Healthy 6m9s canary
│ ├──□ reviews-rollout-5986899db8-pxnjp Pod ✔ Running 6m9s ready:2/2
│ └──□ reviews-rollout-5986899db8-npgxc Pod ✔ Running 11s ready:2/2
└──# revision:1
└──⧉ reviews-rollout-56ff4b868c ReplicaSet ✔ Healthy 24m stable
├──□ reviews-rollout-56ff4b868c-5vrn6 Pod ✔ Running 24m ready:2/2
├──□ reviews-rollout-56ff4b868c-74d8t Pod ✔ Running 24m ready:2/2
├──□ reviews-rollout-56ff4b868c-fphf5 Pod ✔ Running 24m ready:2/2
├──□ reviews-rollout-56ff4b868c-fsrfd Pod ✔ Running 24m ready:2/2
└──□ reviews-rollout-56ff4b868c-t4299 Pod ✔ Running 24m ready:2/2