A canary deployment routes a small percentage of production traffic to a new version while the stable version continues serving the majority. If the canary shows healthy metrics after a defined observation period, traffic gradually shifts to the new version. If metrics degrade, traffic routes back to the stable version automatically. This approach reduces blast radius from 100% of users to as low as 1%.
Netflix popularized canary deployments in 2013, and by 2025, 41% of organizations with 100+ engineers use canary releases for production deployments according to the Accelerate State of DevOps report. The median error detection time with canary deployments is 4.2 minutes compared to 23 minutes with traditional full-rollout deployments.
---
Canary Deployment Flow
The canary lifecycle follows five stages:
| Phase | Traffic Split | Duration | Decision Criteria |
| Initial canary | 5% canary / 95% stable | 5-10 min | Error rate < 1%, p99 latency < 500ms |
| Expanded canary | 25% canary / 75% stable | 10-15 min | Error rate < 0.5%, p99 latency < 400ms |
| Majority canary | 75% canary / 25% stable | 15-30 min | Error rate < 0.1%, p99 unchanged |
| Full promotion | 100% canary | - | All metrics healthy |
---
Method 1: Nginx Upstream Weights
For VPS or bare-metal deployments without a service mesh, nginx upstream weights provide simple traffic splitting. This approach works with any backend, not just Kubernetes.
<h1>/etc/nginx/conf.d/canary.conf</h1>
<h1>Stable version (95% of traffic)</h1>
upstream stable_backend {
server 127.0.0.1:3000 weight=95;
}
<h1>Canary version (5% of traffic)</h1>
upstream canary_backend {
server 127.0.0.1:3001 weight=5;
}
<h1>Combined upstream with weighted distribution</h1>
upstream app_backend {
server 127.0.0.1:3000 weight=95;
server 127.0.0.1:3001 weight=5;
}
server {
listen 443 ssl http2;
server_name yourdomain.com;
location / {
proxy_pass http://app_backend;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# Add header to identify which version handled the request
add_header X-Served-By $upstream_addr always;
}
}
To adjust traffic weights without downtime:
<h1>Update weights in the config file</h1>
sudo sed -i "s/weight=95/weight=75/" /etc/nginx/conf.d/canary.conf
sudo sed -i "s/weight=5/weight=25/" /etc/nginx/conf.d/canary.conf
<h1>Reload nginx (zero-downtime)</h1>
sudo nginx -t && sudo systemctl reload nginx
The `X-Served-By` header helps you verify traffic is actually splitting correctly. Inspect responses with `curl -I https://yourdomain.com` to confirm the upstream address alternates between ports.
---
Method 2: Istio VirtualService
For Kubernetes environments with Istio service mesh, VirtualService resources provide fine-grained traffic control with no application code changes.
<h1>canary-virtualservice.yaml</h1>
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: my-app
namespace: production
spec:
hosts:
- my-app
http:
- route:
- destination:
host: my-app
subset: stable
weight: 95
- destination:
host: my-app
subset: canary
weight: 5
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: my-app
namespace: production
spec:
host: my-app
subsets:
- name: stable
labels:
version: v1.2.0
- name: canary
labels:
version: v1.3.0-rc1
The Deployment for each version carries a `version` label that the DestinationRule uses for subset selection:
<h1>canary-deployment.yaml</h1>
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app-canary
namespace: production
spec:
replicas: 1
selector:
matchLabels:
app: my-app
version: v1.3.0-rc1
template:
metadata:
labels:
app: my-app
version: v1.3.0-rc1
spec:
containers:
- name: app
image: ghcr.io/my-org/my-app:v1.3.0-rc1
ports:
- containerPort: 3000
---
Metrics Validation with PromQL
Canary decisions must be data-driven. Query Prometheus to compare canary metrics against the stable baseline. The three critical metrics are error rate, latency, and saturation.
Error Rate Comparison
<h1>Canary error rate (5xx responses)</h1>
sum(rate(http_requests_total{version="v1.3.0-rc1", status=~"5.."}[5m]))
/
sum(rate(http_requests_total{version="v1.3.0-rc1"}[5m]))
<h1>Stable error rate for comparison</h1>
sum(rate(http_requests_total{version="v1.2.0", status=~"5.."}[5m]))
/
sum(rate(http_requests_total{version="v1.2.0"}[5m]))
Latency Comparison
<h1>Canary p99 latency</h1>
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket{version="v1.3.0-rc1"}[5m]))
by (le)
)
<h1>Stable p99 latency</h1>
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket{version="v1.2.0"}[5m]))
by (le)
)
Saturation Check
<h1>Canary pod CPU usage</h1>
sum(rate(container_cpu_usage_seconds_total{pod=~"my-app-canary.*"}[5m]))
/
sum(kube_pod_container_resource_limits{pod=~"my-app-canary.*", resource="cpu"})
| Metric | Canary Threshold | Action if Exceeded |
| Error rate | > 1% (or 2x stable) | Immediate rollback |
| P99 latency | > 500ms (or 1.5x stable) | Pause and investigate |
| CPU saturation | > 80% | Scale up or rollback |
| Memory usage | > 90% of limit | Scale up or rollback |
---
Automated Rollback Script
This script queries Prometheus, evaluates canary health, and automatically rolls back if metrics exceed thresholds.
#!/bin/bash
<h1>canary-check.sh - Automated canary validation and rollback</h1>
set -e
PROMETHEUS_URL="http://prometheus:9090"
CANARY_VERSION="v1.3.0-rc1"
STABLE_VERSION="v1.2.0"
ERROR_THRESHOLD=0.01 # 1% error rate
LATENCY_THRESHOLD=0.5 # 500ms p99
<h1>Query canary error rate</h1>
CANARY_ERROR_RATE=$(curl -s "$PROMETHEUS_URL/api/v1/query" \
--data-urlencode "query=sum(rate(http_requests_total{version=\"$CANARY_VERSION\",status=~\"5..\"}[5m])) / sum(rate(http_requests_total{version=\"$CANARY_VERSION\"}[5m]))" \
| jq -r ".data.result[0].value[1] // 0")
<h1>Query canary p99 latency</h1>
CANARY_LATENCY=$(curl -s "$PROMETHEUS_URL/api/v1/query" \
--data-urlencode "query=histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{version=\"$CANARY_VERSION\"}[5m])) by (le))" \
| jq -r ".data.result[0].value[1] // 0")
echo "Canary error rate: $CANARY_ERROR_RATE (threshold: $ERROR_THRESHOLD)"
echo "Canary p99 latency: ${CANARY_LATENCY}s (threshold: ${LATENCY_THRESHOLD}s)"
<h1>Check error rate</h1>
if (( $(echo "$CANARY_ERROR_RATE > $ERROR_THRESHOLD" | bc -l) )); then
echo "ERROR: Canary error rate exceeds threshold. Rolling back."
kubectl set image deployment/my-app-canary app=ghcr.io/my-org/my-app:$STABLE_VERSION -n production
kubectl patch virtualservice my-app -n production --type merge \
-p '{"spec":{"http":[{"route":[{"destination":{"host":"my-app","subset":"stable"},"weight":100}]}]}}'
echo "Rollback complete. All traffic on stable version."
exit 1
fi
<h1>Check latency</h1>
if (( $(echo "$CANARY_LATENCY > $LATENCY_THRESHOLD" | bc -l) )); then
echo "WARNING: Canary latency exceeds threshold. Pausing promotion."
exit 1
fi
echo "Canary metrics healthy. Safe to increase traffic."
Run this script on a cron schedule (every 2-3 minutes) during the canary observation period. Combine it with a traffic stepping script that gradually increases the canary weight from 5% to 25% to 75% to 100% at each healthy checkpoint.
---
Progressive Traffic Stepping
#!/bin/bash
<h1>progressive-canary.sh - Step traffic from 5% to 100%</h1>
STEPS=(5 25 50 75 100)
OBSERVATION_MINUTES=10
for WEIGHT in "${STEPS[@]}"; do
STABLE_WEIGHT=$((100 - WEIGHT))
echo "Setting canary traffic to ${WEIGHT}%"
kubectl patch virtualservice my-app -n production --type merge \
-p '{"spec":{"http":[{"route":[{"destination":{"host":"my-app","subset":"stable"},"weight":$STABLE_WEIGHT},{"destination":{"host":"my-app","subset":"canary"},"weight":$WEIGHT}]}]}}'
echo "Observing for $OBSERVATION_MINUTES minutes..."
sleep $((OBSERVATION_MINUTES * 60))
# Run validation
if ! ./canary-check.sh; then
echo "Canary failed at ${WEIGHT}% traffic. Rolled back."
exit 1
fi
echo "Checkpoint passed at ${WEIGHT}%"
done
echo "Canary promoted to 100%. Cleaning up old deployment."
kubectl delete deployment my-app-canary -n production
---
Canary with Flagger (Automated Progressive Delivery)
For fully automated canary deployments on Kubernetes, Flagger integrates with Istio, Linkerd, or nginx ingress to handle the entire promotion lifecycle.
<h1>flagger-canary.yaml</h1>
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: my-app
namespace: production
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: my-app
service:
port: 80
targetPort: 3000
analysis:
interval: 1m
threshold: 5
maxWeight: 50
stepWeight: 10
metrics:
- name: request-success-rate
thresholdRange:
min: 99
interval: 1m
- name: request-duration
thresholdRange:
max: 500
interval: 1m
Flagger automates what the manual scripts above do: deploy canary, step traffic, validate metrics, and promote or rollback. It reduces a 30-line bash script to a 20-line Kubernetes resource.
For the underlying CI/CD pipeline that feeds into canary deployments, see our CI/CD pipeline guide for startups. For an alternative deployment strategy with instant cutover, read our blue-green deployment checklist. To understand how ArgoCD integrates with canary workflows, check our GitOps for small teams guide.
---
Frequently Asked Questions
What percentage of traffic should the initial canary receive?
Start with 5% for most applications. If your daily traffic is under 1,000 requests, start with 10-20% to ensure the canary receives enough traffic for statistically meaningful metric comparisons. The minimum viable sample is approximately 100 requests to calculate a reliable error rate.
How long should the canary observation period last?
Minimum 10 minutes per traffic step. This covers at least two Prometheus scrape intervals (default 15-second intervals means 40+ data points). For applications with strong time-of-day patterns (e.g., e-commerce), extend to 1-2 hours to capture traffic variability.
Can I do canary deployments without a service mesh?
Yes. Nginx upstream weights (shown above) provide basic traffic splitting without any service mesh. You lose per-request metrics and fine-grained routing, but gain simplicity. For VPS deployments, nginx-based canary is the most practical approach.
What metrics should I monitor during a canary?
The four essential metrics are: error rate (5xx responses), p99 latency, CPU/memory saturation, and at least one business metric (conversion rate, signup completions, API response correctness). Error rate catches obvious failures. Latency catches performance regressions. Business metrics catch subtle logic bugs that do not trigger errors.
How do I handle database schema changes with canary deployments?
Both the stable and canary versions must work with the same database schema. Use expand-and-contract migrations: first add new columns/tables (expand), deploy the canary that uses both old and new schemas, then after full promotion, remove old columns (contract). Never deploy a canary that requires a breaking schema change.
What is the difference between canary and blue-green deployments?
Canary gradually shifts traffic (5%, 25%, 50%, 100%) over 30-60 minutes with continuous metric validation. Blue-green switches 100% of traffic instantly between two identical environments. Canary has lower blast radius but longer deployment time. Blue-green has instant rollback but higher blast radius. Choose canary for high-traffic applications where gradual validation is worth the extra deployment time.