Configure Argo Rollouts for canary deployments with automated analysis, design safe database migrations using the expand-contract pattern, write rollback scripts that handle both application and schema changes, and use Helm rollback for emergency recovery. This guide covers the deployment strategies that prevent bad releases from becoming outages.
Every deployment is a risk. The 2024 DORA State of DevOps report found that elite teams deploy 973x more frequently than low performers, but they also have 3x lower change failure rates. The difference is not that elite teams write perfect code. The difference is that they have robust rollback strategies. When a deployment fails, they recover in minutes instead of hours. This guide gives you the exact configurations and scripts to achieve that.
---
1. Deployment Strategy Comparison
Before choosing a strategy, understand the trade-offs. Each approach balances deployment speed, risk, infrastructure cost, and complexity.
Strategy comparison matrix
| Strategy | Risk Level | Rollback Time | Extra Infra Cost | Complexity | Best For |
| Rolling update | Medium | 1-5 min | None | Low | Stateless services, low risk |
| Blue-green | Low | Instant | 2x resources | Medium | Critical services, zero-downtime |
| Canary | Very low | 1-2 min | 10-20% extra | High | High-traffic services, gradual rollout |
| Feature flags | Very low | Instant | None | Medium | Individual features, A/B testing |
For most production services, canary deployments are the optimal strategy. They provide the lowest risk with moderate infrastructure overhead and enable automated rollback based on real metrics.
---
2. Argo Rollouts for Canary Deployments
Argo Rollouts extends Kubernetes with advanced deployment strategies. Unlike a standard Kubernetes Deployment (which does rolling updates), Argo Rollouts supports canary analysis, traffic splitting, and automated rollback based on Prometheus metrics.
Install Argo Rollouts
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
<h1>Install the kubectl plugin for rollout management</h1>
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x kubectl-argo-rollouts-linux-amd64
sudo mv kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts
Canary Rollout with automated analysis
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: api-server
namespace: production
spec:
replicas: 10
selector:
matchLabels:
app: api-server
template:
metadata:
labels:
app: api-server
spec:
containers:
- name: api
image: registry.example.com/api:v2.0.0
ports:
- containerPort: 8080
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
memory: 512Mi
strategy:
canary:
canaryService: api-server-canary
stableService: api-server-stable
trafficRouting:
nginx:
stableIngress: api-server-ingress
steps:
- setWeight: 5
- pause:
duration: 2m
- analysis:
templates:
- templateName: canary-analysis
- setWeight: 20
- pause:
duration: 5m
- analysis:
templates:
- templateName: canary-analysis
- setWeight: 50
- pause:
duration: 5m
- analysis:
templates:
- templateName: canary-analysis
- setWeight: 100
rollbackWindow:
revisions: 3
AnalysisTemplate for automated rollback
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: canary-analysis
namespace: production
spec:
metrics:
- name: error-rate
interval: 30s
failureLimit: 3
successCondition: result[0] < 0.01
provider:
prometheus:
address: http://prometheus-server.monitoring:9090
query: |
sum(rate(http_requests_total{app="api-server",status=~"5..",rollouts_pod_template_hash="{{args.canary-hash}}"}[2m]))
/
sum(rate(http_requests_total{app="api-server",rollouts_pod_template_hash="{{args.canary-hash}}"}[2m]))
- name: latency-p99
interval: 30s
failureLimit: 3
successCondition: result[0] < 1.0
provider:
prometheus:
address: http://prometheus-server.monitoring:9090
query: |
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket{app="api-server",rollouts_pod_template_hash="{{args.canary-hash}}"}[2m])) by (le)
)
- name: pod-restarts
interval: 60s
failureLimit: 1
successCondition: result[0] == 0
provider:
prometheus:
address: http://prometheus-server.monitoring:9090
query: |
sum(increase(kube_pod_container_status_restarts_total{namespace="production",pod=~"api-server-.*"}[5m]))
This configuration:
- Starts at 5% traffic to the canary, catching catastrophic failures with minimal user impact
- Runs automated analysis at each step, checking error rate (<1%), P99 latency (<1s), and pod stability
- Automatically rolls back if any metric fails 3 consecutive checks
- Progresses through 5%, 20%, 50%, 100% over approximately 15-20 minutes
Managing rollouts
<h1>Watch rollout progress</h1>
kubectl argo rollouts get rollout api-server -w
<h1>Manual rollback (immediate)</h1>
kubectl argo rollouts undo api-server
<h1>Abort a rollout in progress</h1>
kubectl argo rollouts abort api-server
<h1>Promote canary to 100% (skip remaining steps)</h1>
kubectl argo rollouts promote api-server
<h1>Retry a failed rollout</h1>
kubectl argo rollouts retry rollout api-server
<h1>View rollout history</h1>
kubectl argo rollouts history api-server
Teams using Arxentra's deployment platform get canary deployments with automated analysis built into the service management dashboard.
---
3. Safe Database Migration Pattern (Expand-Contract)
Application rollbacks are useless if the database schema has been changed in a way that is incompatible with the previous version. The expand-contract pattern ensures that database migrations are always backward-compatible, allowing safe rollback at any point.
The expand-contract phases
| Phase | Action | Old Code Works? | New Code Works? |
| 1. Expand | Add new column/table (nullable or with default) | Yes | Yes |
| 2. Migrate data | Backfill new column from old | Yes | Yes |
| 3. Deploy new code | Application reads/writes both columns | Yes | Yes |
| 4. Contract | Remove old column (after all code uses new) | No | Yes |
Example: Renaming a column safely
Suppose you need to rename `user_name` to `display_name` in the users table.
-- Phase 1: Expand (safe to run, does not break old code)
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);
-- Phase 2: Backfill data
UPDATE users SET display_name = user_name WHERE display_name IS NULL;
-- Phase 3: Deploy new code that reads/writes both columns
-- Application code:
-- INSERT INTO users (user_name, display_name, ...) VALUES ($1, $1, ...)
-- SELECT COALESCE(display_name, user_name) AS display_name FROM users
-- Phase 4: Contract (only after all code uses display_name)
-- Wait for at least one deployment cycle to confirm rollback is not needed
ALTER TABLE users DROP COLUMN user_name;
Migration script template
-- Migration: 018_rename_user_name_to_display_name.sql
-- Phase: EXPAND (backward-compatible)
-- Rollback: DROP COLUMN display_name
BEGIN;
-- Add the new column
ALTER TABLE users ADD COLUMN IF NOT EXISTS display_name VARCHAR(255);
-- Backfill in batches to avoid locking
DO $do$
DECLARE
batch_size INT := 1000;
rows_updated INT;
BEGIN
LOOP
UPDATE users
SET display_name = user_name
WHERE display_name IS NULL
AND id IN (
SELECT id FROM users
WHERE display_name IS NULL
LIMIT batch_size
FOR UPDATE SKIP LOCKED
);
GET DIAGNOSTICS rows_updated = ROW_COUNT;
EXIT WHEN rows_updated = 0;
COMMIT;
BEGIN;
END LOOP;
END $do$;
-- Create trigger to keep columns in sync during transition
CREATE OR REPLACE FUNCTION sync_display_name()
RETURNS TRIGGER AS $func$
BEGIN
IF NEW.display_name IS NULL THEN
NEW.display_name := NEW.user_name;
END IF;
IF NEW.user_name IS NULL THEN
NEW.user_name := NEW.display_name;
END IF;
RETURN NEW;
END;
$func$ LANGUAGE plpgsql;
CREATE TRIGGER trg_sync_display_name
BEFORE INSERT OR UPDATE ON users
FOR EACH ROW EXECUTE FUNCTION sync_display_name();
COMMIT;
For database backup and recovery in case migrations go wrong, see our PostgreSQL backup and PITR guide.
---
4. Rollback Scripts
Every deployment should have a corresponding rollback script. This script should be written, reviewed, and tested before the deployment, not during the incident.
Application rollback script
#!/bin/bash
<h1>rollback.sh - Application rollback script</h1>
<h1>Usage: ./rollback.sh [target-revision]</h1>
set -euo pipefail
NAMESPACE="production"
ROLLOUT_NAME="api-server"
TARGET_REVISION="${1:-}"
echo "=== Application Rollback ==="
echo "Time: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "Namespace: $NAMESPACE"
echo "Rollout: $ROLLOUT_NAME"
echo ""
<h1>Step 1: Check current status</h1>
echo "--- Current Status ---"
kubectl argo rollouts get rollout "$ROLLOUT_NAME" -n "$NAMESPACE" --no-color
echo ""
<h1>Step 2: Perform rollback</h1>
if [ -n "$TARGET_REVISION" ]; then
echo "Rolling back to revision: $TARGET_REVISION"
kubectl argo rollouts undo "$ROLLOUT_NAME" -n "$NAMESPACE" --to-revision="$TARGET_REVISION"
else
echo "Rolling back to previous revision"
kubectl argo rollouts undo "$ROLLOUT_NAME" -n "$NAMESPACE"
fi
<h1>Step 3: Wait for rollback to complete</h1>
echo "Waiting for rollback..."
kubectl argo rollouts status "$ROLLOUT_NAME" -n "$NAMESPACE" --timeout=300s
<h1>Step 4: Verify health</h1>
echo ""
echo "--- Post-Rollback Verification ---"
kubectl get pods -n "$NAMESPACE" -l app=api-server
echo ""
<h1>Step 5: Quick health check</h1>
HEALTH=$(kubectl exec -n "$NAMESPACE" deploy/api-server -- curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health 2>/dev/null || echo "failed")
echo "Health check: $HEALTH"
if [ "$HEALTH" = "200" ]; then
echo "Rollback successful."
else
echo "WARNING: Health check failed after rollback. Manual investigation required."
exit 1
fi
Database rollback script
#!/bin/bash
<h1>db-rollback.sh - Database migration rollback</h1>
<h1>Usage: ./db-rollback.sh <migration-number></h1>
set -euo pipefail
MIGRATION="${1:?Usage: db-rollback.sh <migration-number>}"
DB_HOST="${DB_HOST:-localhost}"
DB_PORT="${DB_PORT:-5432}"
DB_NAME="${DB_NAME:-app}"
DB_USER="${DB_USER:-postgres}"
ROLLBACK_FILE="migrations/${MIGRATION}_rollback.sql"
if [ ! -f "$ROLLBACK_FILE" ]; then
echo "ERROR: Rollback file not found: $ROLLBACK_FILE"
exit 1
fi
echo "=== Database Rollback ==="
echo "Migration: $MIGRATION"
echo "Rollback file: $ROLLBACK_FILE"
echo "Database: $DB_NAME on $DB_HOST:$DB_PORT"
echo ""
<h1>Confirm before proceeding</h1>
read -p "Are you sure you want to rollback migration $MIGRATION? (yes/no): " CONFIRM
if [ "$CONFIRM" != "yes" ]; then
echo "Aborted."
exit 0
fi
<h1>Run rollback</h1>
echo "Running rollback..."
psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" \
-v ON_ERROR_STOP=1 \
-f "$ROLLBACK_FILE"
echo "Rollback complete."
<h1>Update migration tracking table</h1>
psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" \
-c "UPDATE schema_migrations SET rolled_back_at = now() WHERE version = '$MIGRATION';"
echo "Migration tracking updated."
---
5. Helm Rollback for Emergency Recovery
When Argo Rollouts is not in use, or for infrastructure-level deployments managed by Helm, Helm's built-in rollback provides a fast recovery option.
<h1>View Helm release history</h1>
helm history api-server -n production
<h1>Rollback to previous revision</h1>
helm rollback api-server -n production
<h1>Rollback to a specific revision</h1>
helm rollback api-server 5 -n production
<h1>Rollback with timeout and wait</h1>
helm rollback api-server -n production --timeout 5m --wait
<h1>Dry-run to see what would change</h1>
helm rollback api-server -n production --dry-run
Helm rollback limitations
| Limitation | Impact | Workaround |
| Does not rollback databases | Schema changes persist | Use expand-contract pattern (Section 3) |
| Does not rollback ConfigMaps created outside Helm | Config drift possible | Manage all config through Helm values |
| Does not rollback PVCs | Data changes persist | Use database backup/PITR |
| Revision history limited | Old revisions pruned | Set --history-max=10 on install |
---
6. Rollback Runbook Template
Every team should have a documented rollback runbook. This template covers the decision tree and communication steps that are often missed during incidents.
Rollback decision tree
| Signal | Severity | Action |
| Error rate >5% for 2+ minutes | Critical | Immediate rollback, no waiting |
| Error rate 1-5% for 5+ minutes | High | Rollback, investigate after |
| P99 latency >2x baseline for 10+ minutes | Medium | Rollback if not resolving |
| Pod crash loops after deploy | High | Rollback, check logs |
| Customer reports, no metric signal | Low | Investigate first, rollback if confirmed |
Communication during rollback
For deployment monitoring, see our Kubernetes observability stack guide. For secure deployment practices, see the Kubernetes security baseline. On Arxentra's platform, rollback is available with one click from the service management dashboard.
---
Frequently Asked Questions
How fast should a rollback be?
Target under 5 minutes from decision to completion. Argo Rollouts achieves this by keeping the previous ReplicaSet warm. Helm rollback typically completes in 1-3 minutes for most deployments. The decision-making time (detecting the problem and deciding to rollback) is usually longer than the actual rollback. Automated canary analysis removes the decision-making bottleneck.
Should I rollback or roll forward?
Rollback when: the issue is unclear, the impact is high, or the fix will take more than 15 minutes. Roll forward when: the fix is a one-line change, the issue is well understood, and the fix can be deployed in under 10 minutes. When in doubt, always rollback first, then fix and deploy normally. The Arxentra blog covers incident response patterns in detail.
How do I handle rollback when the database migration cannot be reversed?
This is why the expand-contract pattern (Section 3) is critical. If you follow expand-contract, the old code always works with the new schema because columns are added, never removed, until the old code is fully decommissioned. For truly irreversible migrations (data deletion, schema changes without expand-contract), take a database snapshot before deploying and be prepared for PITR. See our PostgreSQL backup guide.
What is the difference between Argo Rollouts and Flagger?
Both provide progressive delivery for Kubernetes. Argo Rollouts replaces the Deployment resource with a Rollout resource and manages the canary directly. Flagger works as a sidecar to existing Deployments and manages canary through separate Deployments. Argo Rollouts has more features (blue-green, experiment CRD) and a larger community. Flagger is lighter-weight and integrates well with Flux CD.
How many revisions should I keep for rollback?
Keep 3-5 revisions for Argo Rollouts and Helm. This covers the most recent deployments where you might need to rollback. Keeping more consumes cluster resources (old ReplicaSets with their pods) and complicates the rollback decision. For database migrations, keep rollback scripts forever in version control.
Can I automate rollback without human intervention?
Yes, and you should for well-defined failure modes. The Argo Rollouts AnalysisTemplate in Section 2 automatically rolls back when error rate or latency exceeds thresholds. The key is choosing metrics that have clear pass/fail criteria and low false-positive rates. Start with automated rollback on error rate (most reliable signal), then add latency and pod health as you build confidence.