Infrastructure code has the same failure modes as application code: regressions, misconfigurations, and unintended side effects. Yet most teams test application code rigorously while deploying infrastructure changes without any automated testing. This guide covers the infrastructure testing pyramid from unit tests (policy checks) through integration tests (Terratest) to compliance verification (OPA), plus drift detection to catch unauthorized changes.
A 2025 HashiCorp State of Cloud Strategy survey found that 68% of organizations experienced at least one infrastructure misconfiguration incident in the past year. Teams with automated infrastructure testing reduced misconfiguration incidents by 81% compared to teams relying on manual review alone.
---
Infrastructure Testing Pyramid
| Level | Tool | Speed | Coverage | Cost |
| Static analysis | tflint, checkov | Seconds | Syntax, patterns | Free |
| Unit tests | Terraform validate, OPA | Seconds | Plan validation | Free |
| Integration tests | Terratest, Kitchen-Terraform | Minutes | Real resource creation | Cloud cost |
| Compliance tests | OPA, Sentinel | Seconds | Policy enforcement | Free |
| Drift detection | driftctl, tfstate | Minutes | State vs reality | Free |
---
Terratest: Integration Testing for Infrastructure
Terratest is a Go testing framework that deploys real infrastructure, validates it, and destroys it after testing.
package test
import (
"testing"
"time"
"net/http"
"github.com/gruntwork-io/terratest/modules/terraform"
"github.com/stretchr/testify/assert"
)
func TestWebServer(t *testing.T) {
t.Parallel()
terraformOptions := &terraform.Options{
TerraformDir: "../modules/web-server",
Vars: map[string]interface{}{
"instance_type": "t3.micro",
"environment": "test",
},
// Retry up to 3 times on transient errors
MaxRetries: 3,
TimeBetweenRetries: 5 * time.Second,
}
// Clean up after test
defer terraform.Destroy(t, terraformOptions)
// Deploy infrastructure
terraform.InitAndApply(t, terraformOptions)
// Get outputs
publicIP := terraform.Output(t, terraformOptions, "public_ip")
url := "http://" + publicIP + "/health"
// Validate: HTTP health check returns 200
statusCode := httpGetStatusCode(t, url)
assert.Equal(t, 200, statusCode)
}
func httpGetStatusCode(t *testing.T, url string) int {
client := http.Client{Timeout: 10 * time.Second}
resp, err := client.Get(url)
if err != nil {
t.Fatalf("Failed to GET %s: %v", url, err)
}
defer resp.Body.Close()
return resp.StatusCode
}
<h1>Run Terratest</h1>
cd test
go test -v -timeout 30m -run TestWebServer
---
Policy-as-Code with Open Policy Agent (OPA)
OPA evaluates Terraform plans against policy rules before any infrastructure changes are applied. This prevents policy violations from ever reaching production.
<h1>policy/terraform.rego</h1>
package terraform.analysis
import input as tfplan
<h1>Deny instances larger than t3.xlarge in non-production</h1>
deny[msg] {
resource := tfplan.resource_changes[_]
resource.type == "aws_instance"
resource.change.after.instance_type != "t3.micro"
resource.change.after.instance_type != "t3.small"
resource.change.after.instance_type != "t3.medium"
resource.change.after.instance_type != "t3.large"
resource.change.after.instance_type != "t3.xlarge"
not contains(resource.address, "production")
msg := sprintf("Instance %s uses %s which exceeds non-production limit",
[resource.address, resource.change.after.instance_type])
}
<h1>Require encryption on all S3 buckets</h1>
deny[msg] {
resource := tfplan.resource_changes[_]
resource.type == "aws_s3_bucket"
not has_encryption(resource)
msg := sprintf("S3 bucket %s must have encryption enabled",
[resource.address])
}
has_encryption(resource) {
resource.change.after.server_side_encryption_configuration[_]
}
<h1>Enforce tagging policy</h1>
deny[msg] {
resource := tfplan.resource_changes[_]
required_tags := {"Environment", "Team", "CostCenter"}
provided_tags := {tag | resource.change.after.tags[tag]}
missing := required_tags - provided_tags
count(missing) > 0
msg := sprintf("Resource %s missing required tags: %v",
[resource.address, missing])
}
<h1>Generate Terraform plan as JSON</h1>
terraform plan -out=tfplan.binary
terraform show -json tfplan.binary > tfplan.json
<h1>Evaluate against OPA policies</h1>
opa eval --data policy/ --input tfplan.json \
"data.terraform.analysis.deny"
<h1>In CI pipeline: fail if any deny rules match</h1>
VIOLATIONS=$(opa eval --data policy/ --input tfplan.json \
"data.terraform.analysis.deny" --format raw)
if [ "$VIOLATIONS" != "[]" ]; then
echo "Policy violations found:"
echo "$VIOLATIONS"
exit 1
fi
---
Drift Detection
Infrastructure drift occurs when the actual state of resources diverges from the declared state in code. Common causes are manual console changes, direct API calls, and untracked scripts.
<h1>Terraform refresh and plan to detect drift</h1>
terraform refresh
terraform plan -detailed-exitcode
<h1>Exit code 2 = drift detected (changes needed)</h1>
<h1>Automated drift detection script</h1>
#!/bin/bash
cd /path/to/terraform
terraform init -backend=true
terraform refresh -input=false
DRIFT=$(terraform plan -detailed-exitcode -input=false 2>&1)
EXIT_CODE=$?
if [ $EXIT_CODE -eq 2 ]; then
echo "DRIFT DETECTED"
echo "$DRIFT"
# Send alert to Slack/PagerDuty
curl -X POST "$SLACK_WEBHOOK" \
-d "{"text": "Infrastructure drift detected in production"}"
fi
| Drift Source | Detection Method | Prevention |
| Console changes | Terraform plan | RBAC: restrict console write access |
| Direct API calls | CloudTrail + alerts | Enforce IaC-only changes |
| Untracked scripts | State comparison | Code review for all infra changes |
| Auto-scaling events | Expected drift | Exclude from drift checks |
| Third-party integrations | Tag-based filtering | Document expected changes |
For CI/CD pipelines that run infrastructure tests, see our GitHub Actions advanced workflows guide. Teams implementing GitOps for infrastructure should read our GitOps for small teams guide. For Kubernetes-specific infrastructure security, check our Kubernetes security baseline guide.
---
Frequently Asked Questions
How long do infrastructure integration tests take?
Terratest creates real resources, so test duration depends on resource provisioning time. A simple EC2 instance test takes 3-5 minutes. An RDS instance test takes 10-15 minutes. A full VPC with subnets, NAT gateways, and load balancers takes 15-25 minutes. Run tests in parallel to reduce total pipeline time, and use smaller resource types in tests (t3.micro instead of production sizes).
Is OPA difficult to learn?
Rego (OPA policy language) has a learning curve because it uses a declarative, logic-programming style rather than imperative code. Most teams can write basic policies (tag enforcement, size limits, encryption checks) within 2-3 hours. Complex policies (cross-resource relationships, conditional rules) take 1-2 days to master. Start with the OPA Playground (play.openpolicyagent.org) for interactive learning.
How often should I run drift detection?
Run drift detection at least every 4 hours in production environments. Critical environments (financial systems, healthcare) should check every 30-60 minutes. Schedule drift checks as a cron job in your CI system or use a dedicated tool like driftctl. Alert on drift immediately but do not auto-remediate without human review.
Can I test Terraform modules without deploying real resources?
Yes, using terraform plan and OPA for plan validation. This catches configuration errors, policy violations, and logical issues without provisioning anything. However, plan-based testing cannot verify runtime behavior (security groups actually blocking traffic, health checks passing, DNS resolving). Use plan validation for fast feedback and Terratest for thorough verification.
How do I handle test infrastructure costs?
Use the smallest resource sizes available (t3.micro, db.t3.micro). Set aggressive auto-cleanup with defer terraform.Destroy() in Terratest and a nightly cleanup script that destroys all resources tagged with environment=test. Monitor your test AWS account with billing alerts. Most teams spend $50-200 per month on infrastructure testing, which is negligible compared to the cost of production misconfigurations.