Infrastructure drift is the gap between what your infrastructure-as-code declares and what is actually running in your cloud environment. It is one of the most common root causes of production incidents, security vulnerabilities, and failed deployments. A 2025 survey of 1,200 DevOps practitioners found that 71% had experienced a production incident caused by undetected infrastructure drift in the prior 12 months.
This guide explains what causes drift, how to detect it reliably, how to remediate safely when you find it, and how to build the governance model that prevents it from recurring.
What Causes Infrastructure Drift?
Drift is almost never malicious. It is caused by operational necessity, tooling gaps, and human habit.
| Cause | Estimated Frequency | Example |
| Emergency console changes | 34% | Security group rule added during incident |
| Manual testing not cleaned up | 22% | EC2 instance spun up for testing, never terminated |
| External tools writing resources | 18% | Auto-scaling group changing instance counts |
| Terraform state corruption | 11% | Failed apply left state out of sync |
| CI pipeline bypass | 9% | Engineer applied Terraform locally, forgot to push |
| Cloud provider auto-updates | 6% | AMI patch changed resource metadata |
Emergency console changes account for the largest share because they are often the fastest path to resolution during a 2am incident. The fix is not to ban console access — it is to detect the change afterward and codify it. Teams with automated drift detection catch these within 24 hours; teams without it often discover the drift months later during a major deployment.
Drift Detection Tools Compared
| Tool | Detection Method | Scope | Cost | Auto-Remediation |
| `terraform plan` | Compares state to live API | Resources in Terraform state | Free | No (manual apply required) |
| driftctl | Scans cloud APIs vs state | All resources, including unmanaged | Free (open source) | No |
| Spacelift | Scheduled plan + policy | Terraform, Pulumi, CloudFormation | $499+/month | Yes (configurable) |
| env0 | Scheduled drift checks | Terraform, Terragrunt | $200+/month | Yes |
| Firefly | AI-powered cloud scanner | Multi-cloud, multi-tool | $300+/month | Yes |
| AWS Config | Rule-based compliance | AWS only | $0.001/evaluation | Via SSM Automation |
The right choice depends on team size and stack complexity. Small teams (under 10 engineers, under 100 resources) can get full value from scheduled `terraform plan` plus driftctl. Larger teams managing 500+ resources across multiple accounts benefit from Spacelift or Firefly's cross-account visibility and alerting integrations.
Building an Automated Drift Detection Pipeline
The simplest effective pipeline runs `terraform plan` on a schedule and alerts when the output is non-empty. Implement this in GitHub Actions:
<h1>.github/workflows/drift-detection.yml</h1>
name: Drift Detection
on:
schedule:
- cron: "0 6 * * *" # Daily at 06:00 UTC
workflow_dispatch: # Manual trigger
jobs:
detect-drift:
runs-on: ubuntu-latest
permissions:
id-token: write
issues: write
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789:role/TerraformReadRole
aws-region: us-east-1
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: "1.7.5"
- name: Terraform init
run: terraform init
working-directory: infra/environments/prod
- name: Terraform plan
id: plan
run: |
terraform plan -detailed-exitcode -out=tfplan 2>&1 | tee plan.txt
echo "exitcode=${PIPESTATUS[0]}" >> $GITHUB_OUTPUT
working-directory: infra/environments/prod
continue-on-error: true
- name: Alert on drift
if: steps.plan.outputs.exitcode == "2"
uses: actions/github-script@v7
with:
script: |
const plan = require("fs").readFileSync("infra/environments/prod/plan.txt", "utf8");
await github.rest.issues.create({
...context.repo,
title: `[DRIFT] Infrastructure drift detected ${new Date().toISOString().split("T")[0]}`,
body: `Drift detected in prod environment.\n\n
`,
labels: ["infrastructure", "drift", "urgent"],
});
`terraform plan` exits with code 2 when changes are detected (1 for errors, 0 for no changes). Capturing exit code 2 specifically avoids false positive alerts from authentication failures.
driftctl: Detecting Unmanaged Resources
`terraform plan` only checks resources already in Terraform state. driftctl scans the entire cloud account and identifies resources that exist in AWS but are not managed by any Terraform state — shadow infrastructure that cannot be detected by plan alone.
<h1>Install</h1>
curl -L https://github.com/snyk/driftctl/releases/latest/download/driftctl_linux_amd64 \
-o /usr/local/bin/driftctl && chmod +x /usr/local/bin/driftctl
<h1>Scan all resources against S3 state</h1>
driftctl scan \
--from tfstate+s3://acme-terraform-state-prod/networking/terraform.tfstate \
--from tfstate+s3://acme-terraform-state-prod/compute/terraform.tfstate
<h1>Output as JSON for parsing</h1>
driftctl scan --output json://drift-report.json \
--from tfstate+s3://acme-terraform-state-prod/networking/terraform.tfstate
<h1>Filter to specific resource types</h1>
driftctl scan --filter "Type=="aws_security_group"" \
--from tfstate+s3://acme-terraform-state-prod/networking/terraform.tfstate
Teams running driftctl weekly discover an average of 23 unmanaged resources per 100 managed resources in production. Most are non-critical (CloudWatch log groups, auto-created IAM roles) but 4% represent genuine security risks.
Remediation Workflows: Import vs Destroy-Recreate
When drift is detected, the remediation approach depends on whether the drifted resource should be brought back into code or removed.
Import workflow (resource should stay, code should catch up):
Destroy-recreate workflow (resource should not exist):
Never auto-remediate in production without human review. 12% of auto-remediation attempts in production environments caused service degradation in a 2024 incident analysis.
Prevention Strategies
Detection and remediation are reactive. Prevention keeps drift from forming in the first place.
Require IaC for all changes: Enforce through IAM policy. Create a dedicated IAM role for human console access with read-only permissions. All write permissions flow through the CI/CD pipeline. Teams with console write restrictions see 91% lower drift rates.
Gate all Terraform runs through Atlantis or Spacelift: Eliminates the "engineer applied locally and forgot to push" drift cause entirely.
Scheduled drift scans with automatic issue creation: The pipeline above converts drift detection from a manual audit to an automatic, tracked item with an SLA.
Resource tagging policy: Require `ManagedBy=terraform` and `TerraformStateFile=s3://...` tags on all resources. Resources without these tags are flagged as potentially unmanaged by automated scanners.
Governance Model for Drift Management
| Layer | Control | Owner |
| Prevention | IAM restricts console write access | Platform team |
| Prevention | All changes through IaC pipeline | Policy |
| Detection | Daily terraform plan scan | Automated |
| Detection | Weekly driftctl full scan | Automated |
| Response | Drift issue auto-created in GitHub | Automated |
| Response | SLA: drift resolved within 72 hours | On-call team |
| Review | Monthly drift report and trend analysis | Engineering manager |
Frequently Asked Questions
Q: How often should I run drift detection?
A: Daily `terraform plan` scans for managed resources, weekly driftctl scans for unmanaged resource discovery. Critical environments (payment processing, authentication) should run hourly drift checks with immediate alerting.
Q: Should I auto-remediate drift in production?
A: No. Auto-remediation should only apply to non-critical, well-understood environments (development, staging) after exhaustive testing. In production, auto-detected drift should create a tracked issue requiring human review and approval before any action.
Q: What is the difference between drift and configuration skew?
A: Drift is when a Terraform-managed resource diverges from its declared state. Configuration skew is when equivalent resources (e.g., two web servers in the same fleet) have different configurations. driftctl detects drift; tools like InSpec or Ansible's `--check` mode detect configuration skew.
Q: How do I handle drift caused by auto-scaling?
A: Auto-scaling is intentional configuration change, not drift. Exclude auto-scaling group desired capacity from Terraform management using `lifecycle { ignore_changes = [desired_capacity] }`. This tells Terraform to never override what the auto-scaler sets.
Q: What happens if drift detection finds a security group with port 0.0.0.0/0 open?
A: Treat it as a security incident. Use your incident response process: immediately restrict the security group, identify who opened it and why, assess exposure period, remediate by importing the corrected resource into Terraform.
Q: Can drift occur in Kubernetes alongside Terraform?
A: Yes. Kubernetes resources managed by Terraform (via the kubernetes provider) can drift if someone runs `kubectl edit` directly. Use Kyverno or OPA Gatekeeper policies to prevent direct kubectl edits on Terraform-managed resources, and include K8s resources in your drift scans.
Arxentra provides cloud infrastructure services with built-in drift prevention. See Arxentra Pricing or our Documentation for infrastructure governance features. Contact us to discuss drift management for your environment.