Skip to main content
Version: 1.5.x

Canary deployments using ArgoRollout

This document describes how you can configure ArgoCD & integrate ArgoRollout with TSB GitOps support for canary deployment automation

Before you get started, make sure:
✓ ArgoCD is installed in your cluster and ArgoCD CLI is configured to connect to your ArgoCD server
✓ ArgoRollout is installed in your cluster
✓ TSB is up and running, and GitOps has been enabled for the target cluster

Create an Application from a Git repository​

Create a sample application using the below command. An example repository containing Istio's bookinfo application and TSB configurations is available at https://github.com/tetrateio/tsb-gitops-demo. You can either use ArgoCD CLI or their web UI to import application configurations directly from Git. 

argocd app create bookinfo-app --repo https://github.com/tetrateio/tsb-gitops-demo.git --path application --dest-server https://kubernetes.default.svc --dest-namespace cms-bookinfo

Check the status of your application

argocd app get bookinfo-app
Name:               bookinfo-app
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          cms-bookinfo
URL:                https://localhost:8080/applications/bookinfo-app
Repo:               https://github.com/tetrateio/tsb-gitops-demo.git
Target:
Path:               application
SyncWindow:         Sync Allowed
Sync Policy:        <none>
Sync Status:        Synced to  (9d20ee2)
Health Status:      Healthy

GROUP  KIND            NAMESPACE     NAME                  STATUS   HEALTH   HOOK  MESSAGE
       Namespace       cms-bookinfo  cms-bookinfo          Running  Synced         namespace/cms-bookinfo configured
       ServiceAccount  cms-bookinfo  bookinfo-details      Synced                  serviceaccount/bookinfo-details configured
       ServiceAccount  cms-bookinfo  bookinfo-productpage  Synced                  serviceaccount/bookinfo-productpage configured
       ServiceAccount  cms-bookinfo  bookinfo-ratings      Synced                  serviceaccount/bookinfo-ratings configured
       ServiceAccount  cms-bookinfo  bookinfo-reviews      Synced                  serviceaccount/bookinfo-reviews configured
       Service         cms-bookinfo  details               Synced   Healthy        service/details configured
       Service         cms-bookinfo  productpage           Synced   Healthy        service/productpage configured
       Service         cms-bookinfo  reviews               Synced   Healthy        service/reviews configured
       Service         cms-bookinfo  ratings               Synced   Healthy        service/ratings configured
apps   Deployment      cms-bookinfo  productpage-v1        Synced   Healthy        deployment.apps/productpage-v1 configured
apps   Deployment      cms-bookinfo  reviews               Synced   Healthy        deployment.apps/reviews configured
apps   Deployment      cms-bookinfo  ratings-v1            Synced   Healthy        deployment.apps/ratings-v1 configured
apps   Deployment      cms-bookinfo  details-v1            Synced   Healthy        deployment.apps/details-v1 configured
       Namespace                     cms-bookinfo          Synced

Application Setup​

If you already have kubernetes manifests created for deployment and service resource, You can choose to keep the same objects along with Argo Rollout object for facilitating the canary deployments. You can make necessary changes to Rollout object and TSB mesh configuration of Istio VirtualService/DestinationRule to achieve the desired result.

TSB Configuration Setup​

Since ArgoRollout require you to make some modifications on VS/DR object according to their Canary deployment strategy convention for Istio, You can use TSB DIRECT mode configuration to achieve the desired result.

  • According to ArgoRollout convention, 2 subsets named stable and canary has been added to subsets with necessary labels to identify canary and stable pods which ArgoRollout inject based on canaryMetadata.
  • Please make sure the version labels eg: version: canary/stable has been added to canaryMetadata according to Istio convention for TSB to recognize the subsets and plot the metrics in service dashboard.
  • When using Istio direct mode resources with GitOps, there is an additional label istio.io/rev: "tsb" that needs to be added to the resources. Please refer here for more details.

Create a bookinfo-tsb-conf app by importing the TSB configurations from tsb-gitops-demo/canary/tsb/conf.yaml. You can also choose to keep it in the same repo. 

argocd app create bookinfo-tsb-conf --repo https://github.com/tetrateio/tsb-gitops-demo.git --path canary/tsb --dest-server https://kubernetes.default.svc --dest-namespace cms-bookinfo

Check the status of TSB resources

argocd app get bookinfo-tsb-conf

Name:               bookinfo-tsb-conf
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          cms-bookinfo
URL:                https://localhost:8080/applications/bookinfo-tsb-conf
Repo:               https://github.com/tetrateio/tsb-gitops-demo.git
Target:
Path:               canary/tsb
SyncWindow:         Sync Allowed
Sync Policy:        <none>
Sync Status:        OutOfSync from  (9d20ee2)
Health Status:      Healthy

GROUP                    KIND             NAMESPACE     NAME                               STATUS     HEALTH  HOOK  MESSAGE
networking.istio.io      VirtualService   cms-bookinfo  bookinfo                           Synced                   virtualservice.networking.istio.io/bookinfo configured
networking.istio.io      Gateway          cms-bookinfo  bookinfo-gateway                   Synced                   gateway.networking.istio.io/bookinfo-gateway configured
security.tsb.tetrate.io  Group            cms-bookinfo  bookinfo-security                  Synced                   group.security.tsb.tetrate.io/bookinfo-security configured
traffic.tsb.tetrate.io   Group            cms-bookinfo  bookinfo-traffic                   Synced                   group.traffic.tsb.tetrate.io/bookinfo-traffic configured
gateway.tsb.tetrate.io   Group            cms-bookinfo  bookinfo-gateway                   Synced                   group.gateway.tsb.tetrate.io/bookinfo-gateway configured
tsb.tetrate.io           Workspace        cms-bookinfo  bookinfo-ws                        Synced                   workspace.tsb.tetrate.io/bookinfo-ws configured
tsb.tetrate.io           Tenant           cms-bookinfo  cms                                Synced                   tenant.tsb.tetrate.io/cms configured
networking.istio.io      DestinationRule  cms-bookinfo  details                            Synced                   destinationrule.networking.istio.io/details configured
networking.istio.io      VirtualService   cms-bookinfo  details                            Synced                   virtualservice.networking.istio.io/details configured
networking.istio.io      DestinationRule  cms-bookinfo  ratings                            Synced                   destinationrule.networking.istio.io/ratings configured
networking.istio.io      DestinationRule  cms-bookinfo  productpage                        Synced                   destinationrule.networking.istio.io/productpage configured
networking.istio.io      VirtualService   cms-bookinfo  ratings                            Synced                   virtualservice.networking.istio.io/ratings configured
networking.istio.io      DestinationRule  cms-bookinfo  reviews                            Synced                   destinationrule.networking.istio.io/reviews configured
networking.istio.io      VirtualService   cms-bookinfo  reviews                            Synced                   virtualservice.networking.istio.io/reviews configured
install.tetrate.io       IngressGateway   cms-bookinfo  tsb-gateway-bookinfo               Synced                   ingressgateway.install.tetrate.io/tsb-gateway-bookinfo configured

Verify application​

Run the below command to export LB ip of tsb-gateway-bookinfo

export GATEWAY_IP=$(kubectl -n cms-bookinfo get service tsb-gateway-bookinfo -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Confirm that you can access bookinfo application. As you can see in the response, review v1 service which we have currently deployed doesn't call ratings service.

curl -v "http://bookinfo.tetrate.com/api/v1/products/1/reviews" \
    --resolve "bookinfo.tetrate.com:80:$GATEWAY_IP"

* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< content-length: 361
< server: istio-envoy
< date: Mon, 22 Aug 2022 06:36:52 GMT
< x-envoy-upstream-service-time: 782
<
* Connection #0 to host bookinfo.tetrate.com left intact
{"id": "1", "podname": "reviews-rollout-56ff4b868c-74d8t", "clustername": "null", "reviews": [{"reviewer": "Reviewer1", "text": "An extremely entertaining play by Shakespeare. The slapstick humour is refreshing!"}, {"reviewer": "Reviewer2", "text": "Absolutely fun and entertaining. The play lacks thematic depth when compared to other plays by Shakespeare."}]}

Setup ArgoRollout​

ArgoRollout provides multiple options to migrate your existing kubernetes deployment object into Argo rollout object. You can either convert an existing deployment object to rollout or you can refer your existing deployment from a rollout object using workloadRef. We will be following the latter approach in this example.

In this example we will be doing a canary deployment of reviews service to demonstrate rollout object configurations and how it is facilitating the traffic shifting to both primary and canary deployment of reviews service.

  • Create a Rollout resource and refer your existing deployment using workloadRef.
  • Make sure selector matchLabels has been configured based on your k8s application deployment manifest.
  • Configure rollout strategy to use canary with subset level traffic splitting.
  • Configure canaryMetadata to inject labels and annotations on canary and stable pods.
  • Configure Istio virtualService and destinationRule based on TSB configuration.
  • Once the Rollout object is created, it will spin up the required number of pods side-by-side along with the Deployment pods and then you can scale down your existing deployment to 0 by changing the replicas.
  • Rollout object won't modify your existing Deployment, Traffic would be shifted to the pods managed by Rollout object once the subset is updated in VirtualService.

Example Configuration

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: reviews-rollout
spec:
  replicas: 5
  selector:
    matchLabels:
      app: reviews
  workloadRef:
    apiVersion: apps/v1
    kind: Deployment
    name: reviews
  strategy:
    canary:
      canaryMetadata:
        annotations:
          version: canary
        labels:
          version: canary
          service.istio.io/canonical-revision: canary
      stableMetadata:
        annotations:
          version: stable
        labels:
          version: stable
          service.istio.io/canonical-revision: stable
      trafficRouting:
        istio:
          virtualService: 
            name: reviews
          destinationRule:
            name: reviews
            canarySubsetName: canary  
            stableSubsetName: stable
      steps:
      - setWeight: 10
      - pause: {}
      - setWeight: 20
      - pause: {duration: 10}
      - setWeight: 40
      - pause: {duration: 10}
      - setWeight: 60
      - pause: {duration: 10}
      - setWeight: 80
      - pause: {duration: 10}

Create Rollout​

Run the below command to create a rollout app

argocd app create reviews-rollout --repo https://github.com/tetrateio/tsb-gitops-demo.git --path canary/rollout --dest-server https://kubernetes.default.svc --dest-namespace cms-bookinfo

Check the status

argocd app get reviews-rollout

Name:               reviews-rollout
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          cms-bookinfo
URL:                https://localhost:8080/applications/reviews-rollout
Repo:               https://github.com/tetrateio/tsb-gitops-demo.git
Target:
Path:               canary/rollout
SyncWindow:         Sync Allowed
Sync Policy:        <none>
Sync Status:        Synced to  (bab126b)
Health Status:      Healthy

GROUP        KIND     NAMESPACE     NAME             STATUS  HEALTH   HOOK  MESSAGE
argoproj.io  Rollout  cms-bookinfo  reviews-rollout  Synced  Healthy        rollout.argoproj.io/reviews-rollout created

Trigger Canary Deployment​

Update the reviews service deployment image to v2 version. This will immediately trigger a canary deployment of reviews v2 and will modify the traffic percentage as 90/10.

kubectl argo rollouts set image reviews-rollout reviews=docker.io/istio/examples-bookinfo-reviews-v2:1.16.4 -n cms-bookinfo

Monitor Canary Deployment​

Run the below command to monitor your canary deployment. 

kubectl argo rollouts get rollout reviews-rollout --watch -n cms-bookinfo

Name:            reviews-rollout
Namespace:       cms-bookinfo
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          1/10
  SetWeight:     10
  ActualWeight:  10
Images:          docker.io/istio/examples-bookinfo-reviews-v1:1.16.4 (stable)
                 docker.io/istio/examples-bookinfo-reviews-v2:1.16.4 (canary)
Replicas:
  Desired:       5
  Current:       6
  Updated:       1
  Ready:         6
  Available:     6

NAME                                         KIND        STATUS     AGE  INFO
⟳ reviews-rollout                            Rollout     ॥ Paused   20m
├──# revision:2
│  └──⧉ reviews-rollout-5986899db8           ReplicaSet  ✔ Healthy  80s  canary
│     └──□ reviews-rollout-5986899db8-pxnjp  Pod         ✔ Running  80s  ready:2/2
└──# revision:1
   └──⧉ reviews-rollout-56ff4b868c           ReplicaSet  ✔ Healthy  20m  stable
      ├──□ reviews-rollout-56ff4b868c-5vrn6  Pod         ✔ Running  20m  ready:2/2
      ├──□ reviews-rollout-56ff4b868c-74d8t  Pod         ✔ Running  20m  ready:2/2
      ├──□ reviews-rollout-56ff4b868c-fphf5  Pod         ✔ Running  20m  ready:2/2
      ├──□ reviews-rollout-56ff4b868c-fsrfd  Pod         ✔ Running  20m  ready:2/2
      └──□ reviews-rollout-56ff4b868c-t4299  Pod         ✔ Running  20m  ready:2/2

Generate traffic​

Run the below command to send some requests to bookinfo application.

while true; do curl -m 5 -v "http://bookinfo.tetrate.com/api/v1/products/1/reviews" --resolve "bookinfo.tetrate.com:80:$GATEWAY_IP";  sleep 2 ; done ;

As you can see, some of the response will have the response from ratings service as reviews-v2 calls ratings service.

> GET /api/v1/products/1/reviews HTTP/1.1
> Host: bookinfo.tetrate.com
> User-Agent: curl/7.79.1
> Accept: */*
> Content-Length: 0
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< content-length: 437
< server: istio-envoy
< date: Mon, 22 Aug 2022 06:53:14 GMT
< x-envoy-upstream-service-time: 45
<
* Connection #0 to host bookinfo.tetrate.com left intact
{"id": "1", "podname": "reviews-66f8dddb8c-84pk6", "clustername": "null", "reviews": [{"reviewer": "Reviewer1", "text": "An extremely entertaining play by Shakespeare. The slapstick humour is refreshing!", "rating": {"stars": 5, "color": "black"}}, {"reviewer": "Reviewer2", "text": "Absolutely fun and entertaining. The play lacks thematic depth when compared to other plays by Shakespeare.", "rating": {"stars": 4, "color": "black"}}]}

Monitor Performance Metrics in TSB​

You can monitor the health of each service instance of both canary and stable pods from TSB service dashboard.

service dashboard which shows both stable and canary subsets
service dashboard which shows both stable and canary subsets

service instance metrics for subset canary
service instance metrics for subset canary

service instance metrics for subset stable
service instance metrics for subset stable

service topology which shows only reviews-canary is calling details-v1 service
service topology which shows only reviews-canary is calling details-v1 service

service tracability of reviews service calling details and details of the tags
service tracability of reviews service calling details and details of the tags

Promote Canary Deployment​

You can either do a step promote which will proceed to the next steps mentioned in the Rollout by changing the traffic weight and it will eventually rollout the new version completely or you can do a full promote to the desired version by skipping analysis, pauses, and steps.

# step promotion
kubectl argo rollouts promote reviews-rollout -n cms-bookinfo

# full promotion
kubectl argo rollouts promote reviews-rollout --full -n cms-bookinfo
kubectl argo rollouts get rollout reviews-rollout --watch -n cms-bookinfo

Name:            reviews-rollout
Namespace:       cms-bookinfo
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          5/10
  SetWeight:     40
  ActualWeight:  40
Images:          docker.io/istio/examples-bookinfo-reviews-v1:1.16.4 (stable)
                 docker.io/istio/examples-bookinfo-reviews-v2:1.16.4 (canary)
Replicas:
  Desired:       5
  Current:       7
  Updated:       2
  Ready:         7
  Available:     7

NAME                                         KIND        STATUS     AGE   INFO
⟳ reviews-rollout                            Rollout     ॥ Paused   24m
├──# revision:2
│  └──⧉ reviews-rollout-5986899db8           ReplicaSet  ✔ Healthy  6m9s  canary
│     ├──□ reviews-rollout-5986899db8-pxnjp  Pod         ✔ Running  6m9s  ready:2/2
│     └──□ reviews-rollout-5986899db8-npgxc  Pod         ✔ Running  11s   ready:2/2
└──# revision:1
   └──⧉ reviews-rollout-56ff4b868c           ReplicaSet  ✔ Healthy  24m   stable
      ├──□ reviews-rollout-56ff4b868c-5vrn6  Pod         ✔ Running  24m   ready:2/2
      ├──□ reviews-rollout-56ff4b868c-74d8t  Pod         ✔ Running  24m   ready:2/2
      ├──□ reviews-rollout-56ff4b868c-fphf5  Pod         ✔ Running  24m   ready:2/2
      ├──□ reviews-rollout-56ff4b868c-fsrfd  Pod         ✔ Running  24m   ready:2/2
      └──□ reviews-rollout-56ff4b868c-t4299  Pod         ✔ Running  24m   ready:2/2