k8wiz
All graphs

Argo CD graph

Argo Rollouts canary: traffic weights on an HTTPRoute and analysis from Prometheus

How Argo Rollouts runs a canary: a Rollout owns two ReplicaSets, shifts traffic by editing HTTPRoute weights through the Gateway API plugin, and an AnalysisTemplate queries Prometheus to promote or abort.

A Deployment's rolling update replaces Pods at a fixed pace and stops only when Pods fail their probes. Argo Rollouts replaces the Deployment with a Rollout that moves traffic in steps, measures the new version while it takes real requests, and goes back by itself when the numbers are bad.

Here the traffic split lives in a Gateway API HTTPRoute and the measurement is a Prometheus query. Argo CD syncs the Rollout from Git like any other object.

Interactive graph

Click a resource to see what it does, or follow the setup step by step.

You write itCreated by a controllerComes with an add-oncreatesreferencesselects by labeltraffic

How it works

All 10 resources and what they do

Rollout checkoutargoproj.io/v1alpha1
A Deployment-shaped object with a strategy.canary. The steps set a traffic weight, pause, and run an analysis. It names the two Services and the HTTPRoute it may edit.
ReplicaSet checkout-6d5f8c7b9apps/v1
The current version, 2.4.1. It keeps serving most of the traffic during the canary and is scaled down only after the new version is promoted.
ReplicaSet checkout-79c4d6f5bapps/v1
The new version, 2.5.0, created when the Pod template changes. The controller sizes it to match the traffic weight.
Service checkout-stablev1
You write it with the plain app: checkout selector. The controller adds the stable ReplicaSet's rollouts-pod-template-hash to the selector, so it only ever reaches the old Pods.
Service checkout-canaryv1
The same manifest under another name. Its selector gets the canary ReplicaSet's hash, so it reaches only the new Pods.
Gateway publicgateway.networking.k8s.io/v1
The platform team's Gateway. Requests for shop.example.com arrive here and follow the HTTPRoute attached to it.
HTTPRoute checkoutgateway.networking.k8s.io/v1
Two backends with weights. You write 100 and 0; the Gateway API plugin rewrites the two numbers at each setWeight step, and the Gateway splits real requests accordingly.
AnalysisTemplate success-rateargoproj.io/v1alpha1
How to judge a version: a query, how often to run it, and what counts as success. Here the canary's share of non-5xx responses must stay at 99% or more, and two failed measurements abort.
AnalysisRun checkout-79c4d6f5b-2argoproj.io/v1alpha1
One execution of the template for one rollout step. It stores each measurement and ends as Successful, Failed or Inconclusive.
Service prometheus-k8sv1
Answers the analysis query with the canary's success rate over the last two minutes.

Common mistakes

  • Argo CD sees the edited weights as drift and puts 100 and 0 back in the middle of a canary. Add ignoreDifferences for the route's backendRefs weights, or leave the weights out of Git.
  • The Gateway API support is a plugin. The controller only loads it when the argo-rollouts-config ConfigMap lists it under trafficRouterPlugins.
  • With too little traffic on the canary the query returns no data, and the measurement is an error, not a pass. Give the step real traffic, or handle the empty result in the condition.
  • An aborted Rollout stays aborted with the new image still in its spec. Fix forward with a new image, or revert the commit; kubectl argo rollouts retry rollout checkout starts the same version again.
  • A HorizontalPodAutoscaler must target the Rollout, not a Deployment. scaleTargetRef takes apiVersion: argoproj.io/v1alpha1 and kind: Rollout.

The manifests you write

Everything above that is marked "You write it", in one file. The rest comes from add-ons and controllers.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
  namespace: shop
spec:
  replicas: 6
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: ghcr.io/acme/checkout:2.5.0
          ports:
            - containerPort: 8080
  strategy:
    canary:
      stableService: checkout-stable
      canaryService: checkout-canary
      trafficRouting:
        plugins:
          argoproj-labs/gatewayAPI:
            httpRoute: checkout
            namespace: shop
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 50
        - pause: { duration: 10m }
---
apiVersion: v1
kind: Service
metadata:
  name: checkout-stable
  namespace: shop
spec:
  selector:
    app: checkout
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: checkout-canary
  namespace: shop
spec:
  selector:
    app: checkout
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: checkout
  namespace: shop
spec:
  parentRefs:
    - name: public
      namespace: infra
  hostnames:
    - shop.example.com
  rules:
    - backendRefs:
        - name: checkout-stable
          port: 80
          weight: 100
        - name: checkout-canary
          port: 80
          weight: 0
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: shop
spec:
  metrics:
    - name: success-rate
      interval: 1m
      count: 5
      failureLimit: 1
      successCondition: result[0] >= 0.99
      provider:
        prometheus:
          address: http://prometheus-k8s.monitoring.svc:9090
          query: |
            sum(rate(http_requests_total{service="checkout-canary",code!~"5.."}[2m]))
              / sum(rate(http_requests_total{service="checkout-canary"}[2m]))

Keep going

More graphs