Progressive Rollouts in Kubernetes

Flagger Operator — Automated, Flexible, Safer

Kubernetes Operator, Flagger which is OSS Progressive Delivery Operator managing progressive delivery of new deployment versions of application(s) to a Kubernetes.

It is happening through custom resource definition(CRD) of Kubernetes called canary.

Flagger Overview (Source: https://docs.flagger.app/)

Flagger enables us to automate the release process without disrupting daily operations by gradually redirecting traffic to the new release, monitoring metrics, and conducting compliance tests, thus minimising the risk of introducing a new release in production.

It is versatile and can be integrated with various CI/CD platforms such as Flux, etc. It is compatible with different service meshes and ingress controllers. An added advantage is that it does not require using custom types to replace Deployment objects.

Strategies in Deployment

The strategies majorly differ in accomplishing the goal of gradually redirecting traffic to a new release version. They are listed below.

• Canary Releases
• A/B Testing
• Blue/Green Mirroring
• Blue/Green
• Canary Releases with Session Affinity

NOTE:

We apply the below manifests:

- deployment.apps/podinfo
- horizontalpodautoscaler.autoscaling/podinfo
- ingresses.extensions/podinfo
- canary.flagger.app/podinfo

The below manifests are generated:

- deployment.apps/podinfo-primary
- horizontalpodautoscaler.autoscaling/podinfo-primary
- service/podinfo
- service/podinfo-canary
- service/podinfo-primary
- ingresses.extensions/podinfo-canary

Prerequisites

The following components should be deployed and available in the cluster:

• nginx ingress controller
• flagger
• flagger-loadtester
• prometheus

helm repo add flagger https://flagger.app
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

helm upgrade -i flagger flagger/flagger \
--namespace ingress-nginx \
--set prometheus.install=true \
--set meshProvider=nginx

helm install flagger-loadtester flagger/loadtester -n flagger-system

kubectl create ns ingress-nginx
helm upgrade -i ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--set controller.metrics.enabled=true \
--set controller.podAnnotations."prometheus\.io/scrape"=true \
--set controller.podAnnotations."prometheus\.io/port"=10254

helm upgrade -i flagger-grafana flagger/grafana \
--namespace=monitoring-system \
--set url=http://prometheus.monitoring-system:9090 \
--set user=admin \
--set password=change-me

Lets deploy a sample application(podinfo):

kubectl create ns test

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

cat > podinfo-ingress.yaml <<EOF
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: podinfo
  namespace: test
  labels:
    app: podinfo
spec:
  ingressClassName: nginx
  rules:
    - host: "podinfo.mydomain"
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: podinfo
                port:
                  number: 80
EOF

I. Canary Releases

cat > podinfo-canary.yaml <<EOF
---
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: podinfo
  namespace: test
spec:
  provider: nginx

  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo

  autoscalerRef:
    apiVersion: autoscaling/v2beta2
    kind: HorizontalPodAutoscaler
    name: podinfo

  ingressRef:
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    name: podinfo

  progressDeadlineSeconds: 60

  service:
    port: 80
    targetPort: 9898

  skipAnalysis: false

  analysis:
    interval: 1m
    threshold: 10
    maxWeight: 50
    stepWeight: 2
    stepWeightPromotion: 100

    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
      interval: 1m

    webhooks:
      - name: acceptance-test
        type: pre-rollout
        url: http://flagger-loadtester.flagger-system/
        timeout: 30s
        metadata:
          type: bash
          cmd: "curl -sd 'test' http://podinfo-canary/token | grep token"
      - name: load-test
        url: http://flagger-loadtester.flagger-system/
        timeout: 5s
        metadata:
          cmd: "hey -z 1m -q 10 -c 2 http://podinfo.mydomain/"
EOF

cat > podinfo-canary-step.yaml <<EOF
---
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: podinfo
  namespace: test
spec:
  provider: nginx

  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo

  autoscalerRef:
    apiVersion: autoscaling/v2beta2
    kind: HorizontalPodAutoscaler
    name: podinfo

  ingressRef:
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    name: podinfo

  progressDeadlineSeconds: 60

  service:
    port: 80
    targetPort: 9898

  skipAnalysis: false

  analysis:
    interval: 10s
    threshold: 10
    maxWeight: 50
    stepWeight: 5

    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
      interval: 1m

    webhooks:
    - name: acceptance-test
      type: pre-rollout
      url: http://flagger-loadtester.flagger-system/
      timeout: 30s
      metadata:
        type: bash
        cmd: "curl -sd 'test' http://podinfo-canary/token | grep token"
    - name: load-test
      url: http://flagger-loadtester.flagger-system/
      timeout: 5s
      metadata:
        cmd: "hey -z 1m -q 10 -c 2 http://podinfo.mydomain/"
EOF

Detailed information on the options are detailed here

II. A/B Testing

cat > podinfo-canary-ab.yaml <<EOF
---
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: podinfo
  namespace: test
spec:
  provider: nginx

  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo

  autoscalerRef:
    apiVersion: autoscaling/v2beta2
    kind: HorizontalPodAutoscaler
    name: podinfo

  ingressRef:
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    name: podinfo

  progressDeadlineSeconds: 60

  service:
    port: 80
    targetPort: 9898

  skipAnalysis: false

  analysis:
    interval: 1m
    iterations: 10
    threshold: 2
    match:
    - headers:
        x-canary:
          exact: "true"
    - headers:
        cookie:
          exact: "true"
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
      interval: 1m

    webhooks:
    - name: acceptance-test
      type: pre-rollout
      url: http://flagger-loadtester.flagger-system/
      timeout: 30s
      metadata:
        type: bash
        cmd: "curl -sd 'test' http://podinfo-canary/token | grep token"
    - name: load-test
      url: http://flagger-loadtester.flagger-system/
      timeout: 5s
      metadata:
        cmd: "hey -z 1m -q 10 -c 2 http://podinfo.mydomain/"
EOF

Detailed information on the options are detailed here

III. Blue/Green Mirroring

cat > podinfo-canary-bg-mirror.yaml <<EOF
---
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: podinfo
  namespace: test
spec:
  provider: nginx

  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo

  autoscalerRef:
    apiVersion: autoscaling/v2beta2
    kind: HorizontalPodAutoscaler
    name: podinfo

  ingressRef:
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    name: podinfo

  progressDeadlineSeconds: 60

  service:
    port: 80
    targetPort: 9898

  skipAnalysis: false

  analysis:
    interval: 1m
    iterations: 10
    threshold: 2
    mirror: true
    mirrorWeight: 100

    webhooks:
    - name: acceptance-test
      type: pre-rollout
      url: http://flagger-loadtester.flagger-system/
      timeout: 30s
      metadata:
        type: bash
        cmd: "curl -sd 'test' http://podinfo-canary/token | grep token"
    - name: load-test
      url: http://flagger-loadtester.flagger-system/
      timeout: 5s
      metadata:
        cmd: "hey -z 1m -q 10 -c 2 http://podinfo.mydomain/"
EOF

Detailed information on the options are detailed here

IV. Blue/Green

cat > podinfo-canary-bg.yaml <<EOF
---
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: podinfo
  namespace: test
spec:
  provider: nginx

  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo

  autoscalerRef:
    apiVersion: autoscaling/v2beta2
    kind: HorizontalPodAutoscaler
    name: podinfo

  ingressRef:
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    name: podinfo

  progressDeadlineSeconds: 60

  service:
    port: 80
    targetPort: 9898

  skipAnalysis: false

  analysis:
    interval: 1m
    iterations: 10
    threshold: 2

    webhooks:
    - name: acceptance-test
      type: pre-rollout
      url: http://flagger-loadtester.flagger-system/
      timeout: 30s
      metadata:
        type: bash
        cmd: "curl -sd 'test' http://podinfo-canary/token | grep token"
    - name: load-test
      url: http://flagger-loadtester.flagger-system/
      timeout: 5s
      metadata:
        cmd: "hey -z 1m -q 10 -c 2 http://podinfo.mydomain/"
EOF

Detailed information on the options are detailed here

V. Canary Releases with Session Affinity

cat > podinfo-canary-affinity.yaml <<EOF
---
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: podinfo
  namespace: test
spec:
  provider: nginx

  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo

  autoscalerRef:
    apiVersion: autoscaling/v2beta2
    kind: HorizontalPodAutoscaler
    name: podinfo

  ingressRef:
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    name: podinfo

  progressDeadlineSeconds: 60

  service:
    port: 80
    targetPort: 9898

  skipAnalysis: false

  analysis:
    interval: 1m
    threshold: 2
    maxWeight: 50
    stepWeight: 2
    sessionAffinity:
      cookieName: canary
      maxAge: 21600

    webhooks:
    - name: acceptance-test
      type: pre-rollout
      url: http://flagger-loadtester.flagger-system/
      timeout: 30s
      metadata:
        type: bash
        cmd: "curl -sd 'test' http://podinfo-canary/token | grep token"
    - name: load-test
      url: http://flagger-loadtester.flagger-system/
      timeout: 5s
      metadata:
        cmd: "hey -z 1m -q 10 -c 2 http://podinfo.mydomain/"
EOF

Detailed information on the options are detailed here

Reporting via Slack

You can configure Flagger to send its logs to your Slack workspace. To use Slack integration, you’ll need to have an incoming webhook on Slack for your workspace.

App creation → <Name in select the desired workspace> → Create App

On the settings page for the new app, click on Incoming Webhooks on the left navigation bar. Enable webhooks by flipping the switch button next to the title Activate Incoming Webhooks.

More detailed explanation is available here