Kubernetes monitoring is the practice of collecting, aggregating, and alerting on metrics, logs, and traces from containerized workloads and cluster infrastructure. A production-ready observability stack provides sub-minute incident detection, actionable dashboards for on-call engineers, and SLO tracking that drives reliability improvements. Teams with mature observability reduce mean time to detection (MTTD) by 70% and mean time to recovery (MTTR) by 50%.
The Three Pillars of Kubernetes Observability
Effective Kubernetes monitoring combines three signal types. Each serves a different purpose in the incident detection and resolution workflow.
| Pillar | Purpose | Key Tools | Storage Cost |
| Metrics | Trend analysis, SLO tracking, alerting | Prometheus, Thanos, Mimir | Low |
| Logs | Root cause investigation, audit trail | Loki, Elasticsearch, CloudWatch | High |
| Traces | Request flow visibility, latency attribution | Jaeger, Tempo, Zipkin | Medium |
Metrics
Metrics are numeric measurements sampled at regular intervals. They are the foundation of alerting and SLO tracking because they are cheap to store and fast to query.
Critical Kubernetes metrics to collect:
- Cluster level: Node CPU/memory/disk utilization, pod scheduling latency, API server request rates
- Workload level: Pod CPU/memory usage vs requests/limits, restart counts, OOMKill events
- Application level: Request rate, error rate, latency percentiles (p50, p95, p99)
- Business level: Transactions per second, queue depth, user-facing error rates
<h1>Prometheus ServiceMonitor example</h1>
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: api-server-metrics
namespace: monitoring
spec:
selector:
matchLabels:
app: api-server
endpoints:
- port: metrics
interval: 15s
path: /metrics
namespaceSelector:
matchNames:
- production
Logs
Logs provide the context that metrics cannot. When an alert fires, engineers use logs to understand what happened and why. However, logs are expensive to store at scale, so retention and filtering strategies are essential.
Log management best practices:
- Use structured logging (JSON) with consistent field names across all services
- Include correlation IDs (trace IDs) in every log line for cross-service tracing
- Set retention tiers: 7 days hot, 30 days warm, 90 days cold
- Filter out health check logs and other high-volume, low-value entries
- Never log sensitive data: passwords, tokens, PII
// Structured log format example
{
"timestamp": "2025-03-15T10:30:00Z",
"level": "error",
"service": "api-server",
"trace_id": "abc123def456",
"method": "POST",
"path": "/api/services",
"status": 500,
"duration_ms": 1523,
"error": "database connection timeout"
}
Traces
Distributed tracing shows the complete path of a request across services. This is critical for diagnosing latency issues in microservice architectures where a single user request may touch 5-10 services.
Tracing implementation priorities:
For infrastructure options that include built-in observability, review Services Overview.
Designing Your Monitoring Stack
Recommended Open Source Stack
| Component | Tool | Purpose |
| Metrics collection | Prometheus + kube-state-metrics | Cluster and app metrics |
| Metrics storage | Thanos or Mimir | Long-term, multi-cluster storage |
| Log collection | Promtail or Fluent Bit | Ship logs to central store |
| Log storage | Loki | Cost-effective log aggregation |
| Trace collection | OpenTelemetry Collector | Vendor-neutral trace pipeline |
| Trace storage | Tempo or Jaeger | Distributed trace backend |
| Visualization | Grafana | Unified dashboards for all signals |
| Alerting | Alertmanager | Alert routing, grouping, silencing |
Installation with Helm
<h1>Install kube-prometheus-stack (Prometheus + Grafana + Alertmanager)</h1>
helm repo add prometheus-community \
https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack \
--namespace monitoring --create-namespace \
--set prometheus.prometheusSpec.retention=15d \
--set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi
<h1>Install Loki for log aggregation</h1>
helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki-stack \
--namespace monitoring \
--set promtail.enabled=true
<h1>Install Tempo for distributed tracing</h1>
helm install tempo grafana/tempo \
--namespace monitoring
Alerting Strategy
Bad alerting is worse than no alerting. Alert fatigue causes engineers to ignore notifications, which leads to missed incidents. Design your alerting with these principles.
Alert on Symptoms, Not Causes
Alert when users are affected (high error rate, high latency), not when a potential cause is detected (high CPU usage). High CPU that does not impact users does not need a 2 AM page.
Severity Levels
| Severity | Criteria | Response | Notification |
| P1 Critical | Service down, data loss risk | Immediate, all-hands | PagerDuty + phone |
| P2 High | Degraded performance, SLO breach | Within 30 minutes | PagerDuty + Slack |
| P3 Medium | Non-critical component issue | Next business day | Slack channel |
| P4 Low | Cosmetic or minor issue | Within sprint | Ticket creation |
Alert Template
Every alert should include:
- What is broken (specific metric and threshold)
- Impact assessment (which users or services are affected)
- Runbook link (step-by-step investigation and remediation)
- Dashboard link (pre-filtered to the relevant time window)
- Escalation path (who to contact if the primary on-call cannot resolve)
<h1>PrometheusRule example with runbook</h1>
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: api-server-alerts
spec:
groups:
- name: api-server
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "API error rate above 5%"
runbook_url: "https://docs.internal/runbooks/high-error-rate"
dashboard_url: "https://grafana.internal/d/api-overview"
Dashboard Design
Organize dashboards by audience and purpose. Each dashboard should answer a specific set of questions without requiring the viewer to click through multiple pages.
Dashboard Hierarchy
Dashboard design guidelines:
- Use consistent color coding across all dashboards (red = critical, yellow = warning, green = healthy)
- Include time range selectors defaulting to 6 hours for triage and 30 days for trends
- Add annotation overlays for deployments, incidents, and configuration changes
- Keep each dashboard to 8-12 panels maximum. Information density beyond this reduces usability.
For platform-level observability tools and dashboards, explore Cloud Computing for architecture patterns.
SLO-Based Monitoring
Service Level Objectives (SLOs) define the reliability targets your service must meet. SLO-based monitoring shifts focus from "is anything broken?" to "are we meeting our reliability commitments?"
Example SLOs:
- 99.9% of API requests complete successfully (< 500 status code) within 30 days
- 99.5% of API requests complete within 500ms (p99 latency) within 30 days
- 99.99% data durability for stored objects within 365 days
Error budget = 100% minus SLO target. For a 99.9% availability SLO over 30 days, the error budget is 43.2 minutes of downtime. When the error budget is exhausted, freeze feature releases and focus on reliability.
<h1>Calculate remaining error budget</h1>
1 - (
sum(rate(http_requests_total{status=~"5.."}[30d]))
/ sum(rate(http_requests_total[30d]))
) / (1 - 0.999)
Instrumentation Best Practices
Good monitoring starts with good instrumentation. Follow the RED method for services and the USE method for infrastructure.
RED method (for services):
- Rate: Requests per second
- Errors: Failed requests per second
- Duration: Latency distribution (histogram)
USE method (for infrastructure):
- Utilization: Percentage of resource capacity in use
- Saturation: Amount of queued or deferred work
- Errors: Error events from the resource
<h1>Python/FastAPI instrumentation example</h1>
from prometheus_client import Histogram, Counter
REQUEST_DURATION = Histogram(
"http_request_duration_seconds",
"HTTP request duration",
["method", "endpoint", "status"],
buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0]
)
REQUEST_COUNT = Counter(
"http_requests_total",
"Total HTTP requests",
["method", "endpoint", "status"]
)
Review Documentation for detailed instrumentation guides and integration patterns.
Cost Management for Observability
Observability infrastructure can become expensive if not managed intentionally. The biggest cost drivers are high-cardinality metrics, verbose logging, and full-fidelity tracing.
Cost control strategies:
- Drop unused metrics with Prometheus relabeling rules
- Reduce metric cardinality by aggregating high-cardinality labels
- Set log retention tiers based on service criticality
- Sample traces: 100% for errors, 5-20% for successful requests
- Use recording rules to pre-compute frequently queried aggregations
Compare platform options that include built-in observability at Pricing.
Frequently Asked Questions
Q: How much storage do I need for Prometheus metrics?
A rough formula: storage = ingestion_rate x retention_period x bytes_per_sample. For a cluster with 100 services and 15-second scrape interval, expect 5-10 GB per day. With 15-day retention, plan for 75-150 GB. Thanos or Mimir with object storage backend is more cost-effective for retention beyond 15 days.
Q: Should I use the ELK stack or Loki for logs?
Loki is significantly cheaper to operate because it indexes only labels, not full log content. Choose Loki if your primary use case is correlating logs with metrics in Grafana. Choose Elasticsearch if you need complex full-text search, structured query language, or integration with existing ELK tooling.
Q: How do I monitor Kubernetes control plane components?
The API server, etcd, scheduler, and controller manager all expose Prometheus metrics endpoints. Use kube-prometheus-stack, which includes pre-built ServiceMonitors and dashboards for all control plane components. Key metrics: API server request latency, etcd leader changes, scheduler queue depth.
Q: What is the right alert threshold for error rates?
Base thresholds on your SLO, not on arbitrary percentages. If your SLO is 99.9% availability, alert when the error rate exceeds 0.1% sustained over 5 minutes. Use multi-window, multi-burn-rate alerting for more sophisticated SLO-based alerts that balance sensitivity with noise.
Q: How do I handle monitoring across multiple Kubernetes clusters?
Use Thanos or Grafana Mimir for cross-cluster metric aggregation. Deploy a Prometheus instance per cluster for local collection and an aggregation layer that queries across all clusters. Use consistent labeling (cluster name, environment) to filter and correlate data across clusters. Contact us for multi-cluster architecture guidance.