Infrastructure monitoring is the practice of collecting, aggregating, and alerting on metrics, logs, and traces from every layer of your technology stack. Effective monitoring answers two questions in real time: is the system working, and if not, where is it broken? Without monitoring, teams discover outages from customer complaints. With monitoring, teams detect and resolve issues before users notice.
The 2025 DORA State of DevOps report found that elite-performing teams have a mean time to recovery (MTTR) of less than one hour, compared to over one month for low performers. Monitoring and observability are the primary differentiators.
The Three Pillars of Observability
| Pillar | What It Captures | Storage | Query Pattern | Key Tools |
| Metrics | Numeric measurements over time (CPU%, request rate, error count) | Time-series database | Aggregation, trending | Prometheus, Datadog, CloudWatch |
| Logs | Timestamped text records of events | Log aggregation platform | Search, filter, correlate | Loki, Elasticsearch, CloudWatch Logs |
| Traces | Request flow across distributed services | Trace storage backend | Latency breakdown, dependency mapping | Jaeger, Tempo, Zipkin, Datadog APM |
Metrics tell you something is wrong. Logs tell you what went wrong. Traces tell you where in the request flow it went wrong. A mature monitoring stack uses all three pillars together.
Step 1: Deploy Prometheus for Metrics Collection
Prometheus is the CNCF-graduated monitoring standard used by 78% of Kubernetes organizations (CNCF 2025 survey). It scrapes metrics from application endpoints at configurable intervals and stores them in a local time-series database.
Installation via Helm (recommended for Kubernetes):
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--create-namespace \
--set prometheus.prometheusSpec.retention=30d \
--set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi
The kube-prometheus-stack includes Prometheus, Grafana, Alertmanager, and pre-configured dashboards for Kubernetes components. It is production-ready out of the box.
Step 2: Instrument Your Application
Expose custom metrics from your application using Prometheus client libraries.
Key metrics every application should expose:
| Metric | Type | Purpose | Example |
| http_requests_total | Counter | Total request count by status code and method | http_requests_total{method="GET", status="200"} |
| http_request_duration_seconds | Histogram | Request latency distribution | http_request_duration_seconds_bucket{le="0.5"} |
| active_connections | Gauge | Current number of open connections | active_connections{service="api"} |
| errors_total | Counter | Total error count by type | errors_total{type="database_timeout"} |
| queue_depth | Gauge | Items waiting to be processed | queue_depth{queue="email"} |
These metrics follow the RED method (Rate, Errors, Duration) for request-driven services and the USE method (Utilization, Saturation, Errors) for resources.
Step 3: Build Grafana Dashboards
Grafana visualizes Prometheus metrics through configurable dashboards. The kube-prometheus-stack installs Grafana automatically. Access it via kubectl port-forward or Ingress.
Essential dashboards to create:
Overview Dashboard
Shows the health of all services at a glance. Include:
- Total request rate across all services
- Global error rate (percentage of 5xx responses)
- p50 and p99 latency for top 5 services by traffic
- Node CPU and memory utilization
- Pod count by namespace
Service Detail Dashboard
One per critical service, showing:
- Request rate by endpoint and status code
- Latency histogram (p50, p90, p99)
- Error rate by type
- Pod CPU and memory usage vs. limits
- Replica count over time
Step 4: Essential PromQL Queries
PromQL is Prometheus's query language. These queries cover the most common monitoring needs.
| Purpose | PromQL Query |
| Request rate (per second) | `rate(http_requests_total[5m])` |
| Error rate (percentage) | `sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100` |
| p99 latency | `histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))` |
| p50 latency | `histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))` |
| CPU utilization by pod | `sum(rate(container_cpu_usage_seconds_total[5m])) by (pod) / sum(container_spec_cpu_quota / container_spec_cpu_period) by (pod) * 100` |
| Memory utilization by pod | `sum(container_memory_working_set_bytes) by (pod) / sum(container_spec_memory_limit_bytes) by (pod) * 100` |
| Pod restart rate | `increase(kube_pod_container_status_restarts_total[1h])` |
| Disk usage percentage | `(node_filesystem_size_bytes - node_filesystem_avail_bytes) / node_filesystem_size_bytes * 100` |
Step 5: Configure Alert Rules
Alerts notify teams when metrics cross predefined thresholds. Good alerts are actionable (someone can do something about it), urgent (it needs attention now), and specific (the alert message explains what is wrong).
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: application-alerts
namespace: monitoring
spec:
groups:
- name: application
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "Error rate exceeds 5%"
description: "{{ $value | humanizePercentage }} of requests are returning 5xx errors."
runbook: "https://wiki.internal/runbooks/high-error-rate"
- alert: HighLatency
expr: |
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
) > 2
for: 10m
labels:
severity: warning
annotations:
summary: "p99 latency exceeds 2 seconds"
description: "p99 latency is {{ $value }}s."
runbook: "https://wiki.internal/runbooks/high-latency"
- alert: PodCrashLooping
expr: increase(kube_pod_container_status_restarts_total[1h]) > 5
for: 5m
labels:
severity: critical
annotations:
summary: "Pod {{ $labels.pod }} is crash-looping"
description: "Pod has restarted {{ $value }} times in the last hour."
runbook: "https://wiki.internal/runbooks/crashloop"
Alert Severity Levels
| Severity | Condition | Response | Notification |
| Critical | Service down or data loss risk | Immediate (wake people up) | PagerDuty/OpsGenie page |
| Warning | Degraded performance or approaching threshold | Next business day | Slack channel |
| Info | Informational, no action needed | Review in weekly meeting | Dashboard only |
Step 6: Create Incident Runbooks
Every alert should link to a runbook that tells the on-call engineer exactly what to do. Runbooks reduce MTTR by removing the need to diagnose from scratch every time.
Runbook template:
Runbook: High Error Rate
Alert: HighErrorRate (error rate > 5% for 5+ minutes)
Impact: Users are receiving error responses. Revenue-impacting if sustained.
Diagnosis steps:
Resolution steps:
- If caused by recent deployment: rollback with `kubectl rollout undo deployment/web-app -n production`
- If caused by database: check connection pool exhaustion, restart if needed
- If caused by external dependency: enable circuit breaker, return cached/degraded responses
Escalation: If not resolved within 30 minutes, page the service owner.
Step 7: Configure Alert Routing
Alertmanager routes alerts to the right team through the right channel.
route:
group_by: [alertname, namespace]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: slack-default
routes:
- match:
severity: critical
receiver: pagerduty-oncall
- match:
severity: warning
receiver: slack-engineering
receivers:
- name: pagerduty-oncall
pagerduty_configs:
- routing_key: YOUR_PAGERDUTY_KEY
- name: slack-engineering
slack_configs:
- api_url: YOUR_SLACK_WEBHOOK
channel: "#engineering-alerts"
- name: slack-default
slack_configs:
- api_url: YOUR_SLACK_WEBHOOK
channel: "#monitoring"
Monitoring Stack Cost Comparison
| Solution | Type | Cost (100 hosts) | Retention | Setup Effort |
| Prometheus + Grafana | Self-hosted (open source) | Infrastructure cost only (~$200/mo) | Configurable (30d default) | Medium |
| Datadog | SaaS | $1,500-4,500/mo | 15 months | Low |
| New Relic | SaaS | $700-2,100/mo | 8-13 months | Low |
| Grafana Cloud | Managed open source | $300-1,500/mo | 13 months | Low |
| CloudWatch | AWS native | $300-1,000/mo | 15 months | Low (AWS only) |
Self-hosted Prometheus + Grafana is the most cost-effective option and provides the most flexibility. SaaS solutions reduce operational overhead but cost 3-10x more at scale. For organizations spending $10K+/month on monitoring, self-hosted becomes increasingly attractive. For context on managing the infrastructure that monitoring runs on, see our cloud infrastructure management guide. For setting up the CI/CD pipeline that deploys your monitoring stack, our CI/CD guide covers the workflow.
Frequently Asked Questions
How many metrics should I collect?
Collect the metrics you will actually use for dashboards and alerts. A typical production service exposes 50-200 custom metrics. Avoid high-cardinality labels (user IDs, request IDs) in metrics -- these belong in logs and traces. High cardinality causes Prometheus memory usage to explode: each unique label combination creates a new time series.
How long should I retain monitoring data?
Keep high-resolution metrics (15s intervals) for 15-30 days. Downsample to 5-minute intervals for 6-12 months. Archive daily aggregates for 2-5 years. Prometheus handles short-term storage; use Thanos or Cortex for long-term retention. The 2025 Grafana Labs survey found that 60% of organizations retain metrics for 30-90 days.
What is the difference between monitoring and observability?
Monitoring tells you when predefined conditions are met (CPU > 80%, error rate > 5%). Observability enables you to ask arbitrary questions about your system without knowing the questions in advance. Observability requires all three pillars (metrics, logs, traces) with correlation between them. Monitoring is a subset of observability.
How do I monitor Kubernetes clusters?
The kube-prometheus-stack provides comprehensive Kubernetes monitoring out of the box, including dashboards for nodes, pods, namespaces, and control plane components. Add custom ServiceMonitors for your application metrics. Monitor etcd latency, API server request rate, and scheduler queue depth for cluster health.
How do I prevent alert fatigue?
Alert fatigue occurs when too many non-actionable alerts desensitize the on-call team. Prevention strategies: only create alerts for conditions that require human action, use warning severity for non-urgent issues (Slack only, no page), tune thresholds based on historical data (avoid alerts that fire and self-resolve in under 5 minutes), and review alert frequency monthly.