Logs are the most fundamental observability signal but also the most expensive to store and query at scale. The average production application generates 10-50 GB of logs per day, and uncontrolled log growth is the primary driver of observability costs. This guide compares ELK (Elasticsearch, Logstash, Kibana), Grafana Loki, and Datadog for log management, covers structured logging implementation, and provides cost optimization strategies.
The 2025 Datadog State of Cloud Costs report found that log management accounts for 35-45% of total observability spending. Organizations that implement structured logging, appropriate log levels, and tiered retention reduce log costs by 60-70% without losing diagnostic capability.
---
Platform Comparison
| Feature | ELK (Self-Hosted) | Grafana Loki | Datadog |
| Price | Infrastructure cost | Infrastructure cost | $0.10/GB ingested + $1.70/M events |
| Storage model | Full-text index | Label index + chunks | Managed (opaque) |
| Query language | KQL / Lucene | LogQL | Custom query syntax |
| Storage per GB | 1.5-2x raw size (indexed) | 0.8-1.2x raw size (compressed) | N/A (managed) |
| Query speed | Sub-second (indexed fields) | Seconds (grep-like on chunks) | Sub-second |
| Operational complexity | High (cluster management) | Low-Medium | Zero (SaaS) |
| Best for | Full-text search at scale | Cost-effective K8s logging | Teams with budget, no ops |
ELK Stack Architecture
<h1>Filebeat -> Logstash -> Elasticsearch -> Kibana</h1>
<h1>filebeat.yml</h1>
filebeat.inputs:
<ul><li>type: container</li>
</ul> paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
output.logstash:
hosts: ["logstash:5044"]
Grafana Loki with Promtail
<h1>promtail-config.yaml</h1>
server:
http_listen_port: 9080
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: containers
static_configs:
- targets:
- localhost
labels:
job: containerlogs
__path__: /var/log/containers/*.log
pipeline_stages:
- json:
expressions:
level: level
message: msg
- labels:
level:
---
Structured Logging
Structured logging outputs machine-parseable JSON instead of unstructured text. This enables fast field-based queries and automated analysis.
<h1>Python: structlog for structured logging</h1>
import structlog
structlog.configure(
processors=[
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.add_log_level,
structlog.processors.JSONRenderer(),
],
)
logger = structlog.get_logger()
<h1>Structured log entry</h1>
logger.info("order_processed",
order_id=12345,
user_id=789,
amount=99.99,
payment_method="card",
processing_time_ms=142
)
<h1>Output:</h1>
<h1>{"event": "order_processed", "order_id": 12345, "user_id": 789,</h1>
<h1> "amount": 99.99, "payment_method": "card", "processing_time_ms": 142,</h1>
<h1> "level": "info", "timestamp": "2026-03-10T14:30:00Z"}</h1>
// Node.js: pino for structured logging
const pino = require("pino");
const logger = pino({ level: "info" });
logger.info({
event: "order_processed",
orderId: 12345,
userId: 789,
amount: 99.99,
processingTimeMs: 142
}, "Order processed successfully");
---
Log Levels
| Level | Purpose | Production Volume | Example |
| ERROR | Unexpected failures requiring action | Low (< 1%) | Database connection failed |
| WARN | Degraded behavior, not failure | Low (1-5%) | Retry succeeded on third attempt |
| INFO | Business events and state changes | Medium (10-30%) | Order created, user logged in |
| DEBUG | Diagnostic detail for development | High (50-80%) | SQL query text, request body |
| TRACE | Extremely verbose debugging | Very high | Function entry/exit |
<h1>Production: INFO and above</h1>
<h1>Staging: DEBUG and above</h1>
<h1>Development: TRACE and above</h1>
import os
import structlog
LOG_LEVEL = os.getenv("LOG_LEVEL", "INFO")
structlog.configure(wrapper_class=structlog.make_filtering_bound_logger(LOG_LEVEL))
---
Retention Policies
| Log Type | Hot Storage | Warm Storage | Cold/Archive | Total |
| Application errors | 30 days | 90 days | 1 year | 1 year |
| Access logs | 7 days | 30 days | 90 days | 90 days |
| Debug logs | 3 days | None | None | 3 days |
| Audit logs | 90 days | 1 year | 7 years | 7 years |
| Security logs | 30 days | 1 year | 3 years | 3 years |
<h1>Elasticsearch ILM (Index Lifecycle Management) policy</h1>
PUT _ilm/policy/app-logs-policy
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": { "max_size": "50gb", "max_age": "1d" }
}
},
"warm": {
"min_age": "7d",
"actions": {
"shrink": { "number_of_shards": 1 },
"forcemerge": { "max_num_segments": 1 }
}
},
"cold": {
"min_age": "30d",
"actions": {
"freeze": {}
}
},
"delete": {
"min_age": "90d",
"actions": { "delete": {} }
}
}
}
}
---
Cost Optimization
| Strategy | Savings | Implementation |
| Drop debug logs in production | 40-60% | Set LOG_LEVEL=INFO |
| Sample high-volume logs | 20-40% | Log 1 in 10 health checks |
| Compress before shipping | 15-25% | Enable gzip in shipper |
| Tiered retention | 30-50% | Hot/warm/cold lifecycle |
| Drop noisy fields | 10-20% | Strip stack traces from INFO |
| Use Loki instead of ELK | 50-70% | Label-based vs full-text index |
For distributed tracing that complements your log management, see our distributed tracing guide. Teams building alerting on log patterns should read our alerting best practices guide. For the metrics layer of your observability stack, check our Kubernetes observability stack guide.
---
Frequently Asked Questions
Should I use ELK or Loki?
Use Loki if you primarily query logs by label (service name, namespace, log level) and occasionally grep through log content. Loki costs 50-70% less than ELK because it does not index log content. Use ELK if you need fast full-text search across all log fields, complex aggregations, or if your team already has Elasticsearch expertise.
How do I prevent log volume from growing uncontrollably?
Three techniques: (1) Set LOG_LEVEL to INFO in production and DEBUG only in staging. (2) Sample high-volume, low-value logs (health checks, heartbeats) at 1-in-100 rate. (3) Implement log quotas per service, and alert when a service exceeds its daily budget. The most common log explosion comes from error retry loops that log the same error thousands of times.
Is Datadog worth the cost?
Datadog is worth it for teams under 20 engineers who want zero operational overhead. Above 20 engineers or 100 GB/day log volume, the cost becomes significant ($3,000-10,000 per month for logs alone). At that scale, self-hosted Loki or ELK with dedicated ops support is more cost-effective.
How do I handle multiline log entries?
Structured JSON logging eliminates the multiline problem because each log entry is a single JSON line. For legacy applications that output multiline stack traces, configure your log shipper (Filebeat, Promtail) with multiline rules that join lines starting with whitespace or specific patterns back to the previous log entry.
Should I log request and response bodies?
Log request bodies at DEBUG level only, and never in production. Never log response bodies containing user data. For debugging API issues, log a hash of the request body at INFO level so you can correlate requests without storing sensitive data. For compliance-sensitive systems (healthcare, finance), consult your security team before logging any request data.