Deploy kube-prometheus-stack via Helm, write PromQL recording rules for dashboard performance, configure Loki for centralized log aggregation, and create alert rules that page on symptoms instead of causes. This guide builds a complete observability platform that handles metrics, logs, and alerts for production Kubernetes clusters.
Observability is not optional in production. The 2024 Grafana Labs State of Observability report found that organizations with mature observability practices resolve incidents 60% faster and experience 40% fewer outages. Yet many teams run Prometheus with default configurations, missing critical alerts and drowning in noise. This guide builds an opinionated, production-tested stack from scratch.
---
1. Deploy kube-prometheus-stack
The kube-prometheus-stack Helm chart bundles Prometheus, Grafana, Alertmanager, node-exporter, and kube-state-metrics into a single deployment. It is the de facto standard for Kubernetes monitoring, used by 78% of organizations running Prometheus on Kubernetes according to the 2024 CNCF survey.
Install with production-ready values
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install monitoring prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--create-namespace \
-f monitoring-values.yaml
monitoring-values.yaml
prometheus:
prometheusSpec:
retention: 30d
retentionSize: 50GB
storageSpec:
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 100Gi
resources:
requests:
cpu: 500m
memory: 2Gi
limits:
memory: 4Gi
ruleSelectorNilUsesHelmValues: false
serviceMonitorSelectorNilUsesHelmValues: false
podMonitorSelectorNilUsesHelmValues: false
grafana:
adminPassword: "changeme-use-sealed-secret"
persistence:
enabled: true
size: 10Gi
dashboardProviders:
dashboardproviders.yaml:
apiVersion: 1
providers:
- name: default
orgId: 1
folder: ""
type: file
disableDeletion: false
editable: true
options:
path: /var/lib/grafana/dashboards/default
alertmanager:
alertmanagerSpec:
storage:
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
nodeExporter:
enabled: true
kubeStateMetrics:
enabled: true
Critical settings explained:
- retention: 30d with retentionSize: 50GB uses whichever limit is hit first, preventing disk exhaustion.
- ruleSelectorNilUsesHelmValues: false tells Prometheus to discover PrometheusRule resources from all namespaces, not just the Helm release namespace. Without this, custom alert rules in application namespaces are silently ignored.
- Persistent storage for Prometheus, Grafana, and Alertmanager ensures data survives pod restarts.
---
2. Write PromQL Recording Rules
Recording rules pre-compute expensive queries and store results as new time series. This dramatically improves dashboard load times, especially for queries that aggregate across many pods. Without recording rules, complex Grafana dashboards can take 10-30 seconds to load; with them, they load in under 1 second.
PrometheusRule for API latency and availability
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: api-recording-rules
namespace: monitoring
labels:
release: monitoring
spec:
groups:
- name: api.rules
interval: 30s
rules:
- record: api:http_request_duration_seconds:p99
expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
- record: api:http_request_duration_seconds:p95
expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
- record: api:http_request_duration_seconds:p50
expr: histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
- record: api:http_requests:rate5m
expr: sum(rate(http_requests_total[5m])) by (service, status_code)
- record: api:http_requests:error_rate5m
expr: |
sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)
- record: api:http_requests:availability5m
expr: |
1 - (
sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)
)
- name: node.rules
interval: 60s
rules:
- record: node:cpu:utilization
expr: 1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)
- record: node:memory:utilization
expr: |
1 - (
node_memory_MemAvailable_bytes
/
node_memory_MemTotal_bytes
)
- record: node:disk:utilization
expr: |
1 - (
node_filesystem_avail_bytes{mountpoint="/"}
/
node_filesystem_size_bytes{mountpoint="/"}
)
Key PromQL patterns every team needs
| Use Case | PromQL Query | Description |
| Error rate | `sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))` | Percentage of 5xx responses |
| Request rate | `sum(rate(http_requests_total[5m])) by (service)` | Requests per second per service |
| P99 latency | `histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))` | 99th percentile response time |
| Pod restarts | `increase(kube_pod_container_status_restarts_total[1h]) > 3` | Pods restarting frequently |
| CPU saturation | `1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)` | Per-node CPU utilization |
| Memory pressure | `node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1` | Nodes with less than 10% free memory |
| PVC usage | `kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes > 0.85` | Volumes above 85% full |
---
3. Deploy Loki for Log Aggregation
Prometheus handles metrics; Loki handles logs. Loki is designed by Grafana Labs as "like Prometheus, but for logs." It indexes log metadata (labels) rather than full text, making it 10-100x cheaper to operate than Elasticsearch for the same log volume. The 2024 Grafana survey found that Loki adoption grew 45% year-over-year, now running in 40% of organizations with Kubernetes.
Install Loki with Promtail
<h1>Install Loki stack (Loki + Promtail)</h1>
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install loki grafana/loki-stack \
--namespace monitoring \
-f loki-values.yaml
loki-values.yaml
loki:
enabled: true
persistence:
enabled: true
size: 50Gi
config:
limits_config:
retention_period: 14d
max_entries_limit_per_query: 5000
max_query_series: 500
chunk_store_config:
max_look_back_period: 14d
table_manager:
retention_deletes_enabled: true
retention_period: 14d
promtail:
enabled: true
config:
snippets:
pipelineStages:
- docker: {}
- match:
selector: '{app="api-server"}'
stages:
- json:
expressions:
level: level
msg: message
trace_id: trace_id
- labels:
level:
- output:
source: msg
Add Loki as a datasource in Grafana
<h1>Add to monitoring-values.yaml under grafana.additionalDataSources</h1>
grafana:
additionalDataSources:
- name: Loki
type: loki
access: proxy
url: http://loki:3100
jsonData:
maxLines: 1000
Useful LogQL queries
| Use Case | LogQL Query |
| All errors in production | `{namespace="production"} \ | = "error" \ | logfmt \ | level="error"` |
| Slow requests (>1s) | `{app="api-server"} \ | json \ | duration > 1s` | |
| Pod crash logs | `{namespace="production"} \ | = "OOMKilled" or "CrashLoopBackOff"` | ||
| Request count by status | `sum by (status) (count_over_time({app="api-server"} \ | json \ | __error__="" [5m]))` | |
| Trace correlation | `{app="api-server"} \ | = "trace_id=abc123"` |
On Arxentra's infrastructure platform, logs are automatically aggregated and searchable through the built-in dashboard, eliminating the need to manage your own Loki installation.
---
4. Create Actionable Alert Rules
The most important principle in alerting is: alert on symptoms, not causes. Users care that the API is slow, not that CPU is at 80%. Alert on SLO violations (error rate, latency, availability) and investigate causes during incident response. According to Google's SRE book, teams with symptom-based alerting have 50% fewer false positives than those alerting on resource utilization.
PrometheusRule for production alerts
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: production-alerts
namespace: monitoring
labels:
release: monitoring
spec:
groups:
- name: slo.alerts
rules:
- alert: HighErrorRate
expr: api:http_requests:error_rate5m > 0.01
for: 5m
labels:
severity: critical
annotations:
summary: "Error rate above 1% for {{ $labels.service }}"
description: "Service {{ $labels.service }} has {{ $value | humanizePercentage }} error rate"
runbook: "https://wiki.example.com/runbooks/high-error-rate"
- alert: HighLatency
expr: api:http_request_duration_seconds:p99 > 1
for: 5m
labels:
severity: warning
annotations:
summary: "P99 latency above 1s for {{ $labels.service }}"
description: "Service {{ $labels.service }} p99 latency is {{ $value | humanizeDuration }}"
- alert: LowAvailability
expr: api:http_requests:availability5m < 0.999
for: 10m
labels:
severity: critical
annotations:
summary: "Availability below 99.9% for {{ $labels.service }}"
- name: infrastructure.alerts
rules:
- alert: PodCrashLooping
expr: increase(kube_pod_container_status_restarts_total[1h]) > 5
for: 10m
labels:
severity: warning
annotations:
summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} crash-looping"
- alert: PVCAlmostFull
expr: kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes > 0.85
for: 15m
labels:
severity: warning
annotations:
summary: "PVC {{ $labels.persistentvolumeclaim }} is {{ $value | humanizePercentage }} full"
- alert: NodeHighMemory
expr: node:memory:utilization > 0.9
for: 10m
labels:
severity: warning
annotations:
summary: "Node {{ $labels.instance }} memory at {{ $value | humanizePercentage }}"
- alert: NodeDiskPressure
expr: node:disk:utilization > 0.85
for: 10m
labels:
severity: critical
annotations:
summary: "Node {{ $labels.instance }} disk at {{ $value | humanizePercentage }}"
Configure Alertmanager routing
<h1>Add to monitoring-values.yaml</h1>
alertmanager:
config:
global:
resolve_timeout: 5m
route:
group_by: ["alertname", "namespace"]
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: default
routes:
- match:
severity: critical
receiver: pagerduty
repeat_interval: 1h
- match:
severity: warning
receiver: slack
receivers:
- name: default
slack_configs:
- api_url: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
channel: "#alerts"
title: '{{ .GroupLabels.alertname }}'
text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
- name: pagerduty
pagerduty_configs:
- service_key: "your-pagerduty-key"
- name: slack
slack_configs:
- api_url: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
channel: "#alerts-warning"
---
5. Essential Grafana Dashboards
Every Kubernetes team needs at minimum three dashboards: cluster overview, namespace overview, and application-specific. Rather than building from scratch, import community dashboards and customize them.
Must-have dashboard IDs for Grafana
| Dashboard | Grafana ID | Description |
| Kubernetes Cluster Overview | 7249 | Node health, pod counts, resource utilization |
| Namespace Resources | 15758 | Per-namespace CPU, memory, network |
| CoreDNS | 15762 | DNS query latency and error rates |
| NGINX Ingress | 14314 | Request rates, latency by ingress |
| Node Exporter | 1860 | Detailed per-node system metrics |
| Pod Resources | 15760 | Per-pod resource usage and limits |
<h1>Import a dashboard via Grafana API</h1>
curl -X POST http://admin:password@grafana:3000/api/dashboards/import \
-H "Content-Type: application/json" \
-d '{
"dashboard": {"id": 7249},
"overwrite": true,
"inputs": [{
"name": "DS_PROMETHEUS",
"type": "datasource",
"pluginId": "prometheus",
"value": "Prometheus"
}]
}'
---
6. Operational Checklist
| Task | Frequency | Command/Action |
| Check Prometheus disk usage | Weekly | `kubectl exec -n monitoring prometheus-0 -- df -h /prometheus` |
| Review firing alerts | Daily | Alertmanager UI or Slack channel |
| Validate recording rules | Monthly | `promtool check rules rules.yaml` |
| Update dashboards | Quarterly | Review for new metrics, remove unused panels |
| Test alert routing | Monthly | `amtool alert add test severity=critical` |
| Review Loki retention | Monthly | Check disk usage, adjust retention_period |
For teams using Arxentra's managed infrastructure, monitoring is pre-configured with battle-tested alert rules and dashboards. Read our security baseline guide for securing the monitoring stack itself.
---
Frequently Asked Questions
How much storage does Prometheus need?
A rough formula: storage = retention_days x ingestion_rate x bytes_per_sample x 1.5 (overhead). For a 50-node cluster with 500 pods, expect about 1-2GB per day of storage. With 30-day retention, allocate 50-100GB. The kube-prometheus-stack defaults often undersize storage; always use persistent volumes.
Should I use Loki or Elasticsearch for logs?
Loki for most Kubernetes workloads. It is 5-10x cheaper to operate because it only indexes labels, not full text. Elasticsearch is better when you need full-text search across millions of log lines (security forensics, compliance). For 90% of operational logging needs, Loki is sufficient and dramatically simpler to manage.
How do I prevent alert fatigue?
Three rules: (1) Alert on symptoms, not causes. (2) Every alert must have a runbook URL in its annotations. (3) If nobody acts on an alert within 30 days, delete it. The Google SRE book recommends that every page should require immediate human action; if it does not, it should be a warning (Slack) or a ticket, not a page.
Can I use this stack for multi-cluster monitoring?
Yes, with additions. Deploy Thanos or Victoria Metrics as a long-term storage layer that aggregates metrics from multiple Prometheus instances. For logs, run Loki in multi-tenant mode with one tenant per cluster. Grafana natively supports multiple datasources, so a single Grafana instance can query all clusters. Check the Arxentra documentation for multi-cluster patterns.
What metrics should every application expose?
At minimum, the four golden signals from the Google SRE book: (1) Latency - histogram of request duration, (2) Traffic - request count, (3) Errors - count by error type, (4) Saturation - queue depth or thread pool usage. Use the Prometheus client library for your language and expose metrics on a /metrics endpoint. Most frameworks (Express, FastAPI, Spring) have middleware that does this automatically.
How do I migrate from an existing ELK stack to Loki?
Run both in parallel during the transition. Deploy Promtail alongside your existing Filebeat/Fluentd. Route the same logs to both Elasticsearch and Loki for 2-4 weeks. Compare query results and validate that your critical log queries work in LogQL. Once validated, remove the Elasticsearch pipeline. The cost savings typically pay for the migration effort within the first month.