Progressive Rollouts in Kubernetes

Yashwanth Lakkaraju
5 min readJan 23, 2023

Flagger Operator — Automated, Flexible, Safer

Kubernetes Operator, Flagger which is OSS Progressive Delivery Operator managing progressive delivery of new deployment versions of application(s) to a Kubernetes.

It is happening through custom resource definition(CRD) of Kubernetes called canary.

Flagger Overview (Source: https://docs.flagger.app/)

Flagger enables us to automate the release process without disrupting daily operations by gradually redirecting traffic to the new release, monitoring metrics, and conducting compliance tests, thus minimising the risk of introducing a new release in production.

It is versatile and can be integrated with various CI/CD platforms such as Flux, etc. It is compatible with different service meshes and ingress controllers. An added advantage is that it does not require using custom types to replace Deployment objects.

Strategies in Deployment

The strategies majorly differ in accomplishing the goal of gradually redirecting traffic to a new release version. They are listed below.

  • Canary Releases
  • A/B Testing
  • Blue/Green Mirroring
  • Blue/Green
  • Canary Releases with Session Affinity

NOTE:

We apply the below manifests:

- deployment.apps/podinfo
- horizontalpodautoscaler.autoscaling/podinfo
- ingresses.extensions/podinfo
- canary.flagger.app/podinfo

The below manifests are generated:

- deployment.apps/podinfo-primary
- horizontalpodautoscaler.autoscaling/podinfo-primary
- service/podinfo
- service/podinfo-canary
- service/podinfo-primary
- ingresses.extensions/podinfo-canary

Prerequisites

The following components should be deployed and available in the cluster:

  • nginx ingress controller
  • flagger
  • flagger-loadtester
  • prometheus
helm repo add flagger https://flagger.app
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

helm upgrade -i flagger flagger/flagger \
--namespace ingress-nginx \
--set prometheus.install=true \
--set meshProvider=nginx

helm install flagger-loadtester flagger/loadtester -n flagger-system

kubectl create ns ingress-nginx
helm upgrade -i ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--set controller.metrics.enabled=true \
--set controller.podAnnotations."prometheus\.io/scrape"=true \
--set controller.podAnnotations."prometheus\.io/port"=10254

helm upgrade -i flagger-grafana flagger/grafana \
--namespace=monitoring-system \
--set url=http://prometheus.monitoring-system:9090 \
--set user=admin \
--set password=change-me

Lets deploy a sample application(podinfo):

kubectl create ns test

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

cat > podinfo-ingress.yaml <<EOF
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: podinfo
namespace: test
labels:
app: podinfo
spec:
ingressClassName: nginx
rules:
- host: "podinfo.mydomain"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: podinfo
port:
number: 80
EOF

I. Canary Releases

cat > podinfo-canary.yaml <<EOF
---
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: podinfo
namespace: test
spec:
provider: nginx

targetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo

autoscalerRef:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
name: podinfo

ingressRef:
apiVersion: networking.k8s.io/v1
kind: Ingress
name: podinfo

progressDeadlineSeconds: 60

service:
port: 80
targetPort: 9898

skipAnalysis: false

analysis:
interval: 1m
threshold: 10
maxWeight: 50
stepWeight: 2
stepWeightPromotion: 100

metrics:
- name: request-success-rate
thresholdRange:
min: 99
interval: 1m

webhooks:
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.flagger-system/
timeout: 30s
metadata:
type: bash
cmd: "curl -sd 'test' http://podinfo-canary/token | grep token"
- name: load-test
url: http://flagger-loadtester.flagger-system/
timeout: 5s
metadata:
cmd: "hey -z 1m -q 10 -c 2 http://podinfo.mydomain/"
EOF
cat > podinfo-canary-step.yaml <<EOF
---
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: podinfo
namespace: test
spec:
provider: nginx

targetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo

autoscalerRef:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
name: podinfo

ingressRef:
apiVersion: networking.k8s.io/v1
kind: Ingress
name: podinfo

progressDeadlineSeconds: 60

service:
port: 80
targetPort: 9898

skipAnalysis: false

analysis:
interval: 10s
threshold: 10
maxWeight: 50
stepWeight: 5

metrics:
- name: request-success-rate
thresholdRange:
min: 99
interval: 1m

webhooks:
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.flagger-system/
timeout: 30s
metadata:
type: bash
cmd: "curl -sd 'test' http://podinfo-canary/token | grep token"
- name: load-test
url: http://flagger-loadtester.flagger-system/
timeout: 5s
metadata:
cmd: "hey -z 1m -q 10 -c 2 http://podinfo.mydomain/"
EOF

Detailed information on the options are detailed here

II. A/B Testing

cat > podinfo-canary-ab.yaml <<EOF
---
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: podinfo
namespace: test
spec:
provider: nginx

targetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo

autoscalerRef:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
name: podinfo

ingressRef:
apiVersion: networking.k8s.io/v1
kind: Ingress
name: podinfo

progressDeadlineSeconds: 60

service:
port: 80
targetPort: 9898

skipAnalysis: false

analysis:
interval: 1m
iterations: 10
threshold: 2
match:
- headers:
x-canary:
exact: "true"
- headers:
cookie:
exact: "true"
metrics:
- name: request-success-rate
thresholdRange:
min: 99
interval: 1m

webhooks:
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.flagger-system/
timeout: 30s
metadata:
type: bash
cmd: "curl -sd 'test' http://podinfo-canary/token | grep token"
- name: load-test
url: http://flagger-loadtester.flagger-system/
timeout: 5s
metadata:
cmd: "hey -z 1m -q 10 -c 2 http://podinfo.mydomain/"
EOF

Detailed information on the options are detailed here

III. Blue/Green Mirroring

cat > podinfo-canary-bg-mirror.yaml <<EOF
---
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: podinfo
namespace: test
spec:
provider: nginx

targetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo

autoscalerRef:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
name: podinfo

ingressRef:
apiVersion: networking.k8s.io/v1
kind: Ingress
name: podinfo

progressDeadlineSeconds: 60

service:
port: 80
targetPort: 9898

skipAnalysis: false

analysis:
interval: 1m
iterations: 10
threshold: 2
mirror: true
mirrorWeight: 100

webhooks:
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.flagger-system/
timeout: 30s
metadata:
type: bash
cmd: "curl -sd 'test' http://podinfo-canary/token | grep token"
- name: load-test
url: http://flagger-loadtester.flagger-system/
timeout: 5s
metadata:
cmd: "hey -z 1m -q 10 -c 2 http://podinfo.mydomain/"
EOF

Detailed information on the options are detailed here

IV. Blue/Green

cat > podinfo-canary-bg.yaml <<EOF
---
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: podinfo
namespace: test
spec:
provider: nginx

targetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo

autoscalerRef:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
name: podinfo

ingressRef:
apiVersion: networking.k8s.io/v1
kind: Ingress
name: podinfo

progressDeadlineSeconds: 60

service:
port: 80
targetPort: 9898

skipAnalysis: false

analysis:
interval: 1m
iterations: 10
threshold: 2

webhooks:
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.flagger-system/
timeout: 30s
metadata:
type: bash
cmd: "curl -sd 'test' http://podinfo-canary/token | grep token"
- name: load-test
url: http://flagger-loadtester.flagger-system/
timeout: 5s
metadata:
cmd: "hey -z 1m -q 10 -c 2 http://podinfo.mydomain/"
EOF

Detailed information on the options are detailed here

V. Canary Releases with Session Affinity

cat > podinfo-canary-affinity.yaml <<EOF
---
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: podinfo
namespace: test
spec:
provider: nginx

targetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo

autoscalerRef:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
name: podinfo

ingressRef:
apiVersion: networking.k8s.io/v1
kind: Ingress
name: podinfo

progressDeadlineSeconds: 60

service:
port: 80
targetPort: 9898

skipAnalysis: false

analysis:
interval: 1m
threshold: 2
maxWeight: 50
stepWeight: 2
sessionAffinity:
cookieName: canary
maxAge: 21600

webhooks:
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.flagger-system/
timeout: 30s
metadata:
type: bash
cmd: "curl -sd 'test' http://podinfo-canary/token | grep token"
- name: load-test
url: http://flagger-loadtester.flagger-system/
timeout: 5s
metadata:
cmd: "hey -z 1m -q 10 -c 2 http://podinfo.mydomain/"
EOF

Detailed information on the options are detailed here

Reporting via Slack

You can configure Flagger to send its logs to your Slack workspace. To use Slack integration, you’ll need to have an incoming webhook on Slack for your workspace.

App creation → <Name in select the desired workspace> → Create App

On the settings page for the new app, click on Incoming Webhooks on the left navigation bar. Enable webhooks by flipping the switch button next to the title Activate Incoming Webhooks.

More detailed explanation is available here

--

--

Yashwanth Lakkaraju

📌 🇩🇪 • 🇮🇳 ➡️ 🇺🇲 ➡️ 🇩🇪 | Learner | Engineer | Techie | Nethead | Medium Addict | Avid Movie & TV Buff | Cricket & Soccer Onlooker