Distributed tracing follows a single request across multiple services, revealing exactly where time is spent and where errors originate. Without tracing, debugging a slow request in a microservices architecture requires correlating logs across 5-20 services manually. With tracing, you see the complete request timeline in one view. This guide covers OpenTelemetry instrumentation, backend deployment, sampling strategies, and practical debugging workflows.
The 2025 CNCF Observability Survey found that 74% of organizations running microservices have adopted distributed tracing, up from 29% in 2021. Teams with tracing resolve production incidents 3.2x faster than teams relying on logs alone. The median time to identify the root cause of a cross-service latency issue drops from 47 minutes to 8 minutes.
---
OpenTelemetry Setup
OpenTelemetry (OTel) is the vendor-neutral standard for instrumentation. It provides SDKs for all major languages and exports to any tracing backend.
Python (FastAPI)
<h1>requirements.txt</h1>
<h1>opentelemetry-api==1.27.0</h1>
<h1>opentelemetry-sdk==1.27.0</h1>
<h1>opentelemetry-instrumentation-fastapi==0.48b0</h1>
<h1>opentelemetry-instrumentation-httpx==0.48b0</h1>
<h1>opentelemetry-instrumentation-asyncpg==0.48b0</h1>
<h1>opentelemetry-exporter-otlp==1.27.0</h1>
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.asyncpg import AsyncPGInstrumentor
<h1>Configure tracer</h1>
provider = TracerProvider()
processor = BatchSpanProcessor(
OTLPSpanExporter(endpoint="http://jaeger:4317", insecure=True)
)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
<h1>Auto-instrument FastAPI and database</h1>
FastAPIInstrumentor.instrument_app(app)
AsyncPGInstrumentor().instrument()
<h1>Manual span for custom business logic</h1>
tracer = trace.get_tracer("my-service")
async def process_order(order_id: int):
with tracer.start_as_current_span("process_order") as span:
span.set_attribute("order.id", order_id)
result = await validate_payment(order_id)
span.set_attribute("payment.status", result.status)
return result
Node.js (Express)
// tracing.js - Load before application code
const { NodeSDK } = require("@opentelemetry/sdk-node");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-grpc");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const sdk = new NodeSDK({
traceExporter: new OTLPTraceExporter({
url: "http://jaeger:4317",
}),
instrumentations: [getNodeAutoInstrumentations()],
serviceName: "order-service",
});
sdk.start();
---
Deploying Jaeger
<h1>docker-compose.yml for local development</h1>
services:
jaeger:
image: jaegertracing/all-in-one:1.57
ports:
- "16686:16686" # Jaeger UI
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
environment:
COLLECTOR_OTLP_ENABLED: "true"
SPAN_STORAGE_TYPE: "badger"
BADGER_EPHEMERAL: "false"
BADGER_DIRECTORY_VALUE: "/badger/data"
BADGER_DIRECTORY_KEY: "/badger/key"
volumes:
- jaeger_data:/badger
volumes:
jaeger_data:
| Backend | Storage | Retention | Best For |
| Jaeger (Badger) | Local disk | Days | Development, small production |
| Jaeger (Elasticsearch) | Elasticsearch cluster | Weeks-months | Production |
| Jaeger (Cassandra) | Cassandra cluster | Weeks-months | High-volume production |
| Zipkin (MySQL) | MySQL | Weeks | Simple deployments |
| Tempo (Grafana) | Object storage (S3) | Months | Cost-effective production |
---
Trace Context Propagation
Trace context must be propagated across service boundaries via HTTP headers. OpenTelemetry uses the W3C Trace Context standard (traceparent and tracestate headers).
<h1>Automatic propagation with instrumented HTTP client</h1>
import httpx
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
HTTPXClientInstrumentor().instrument()
async def call_downstream_service(order_id: int):
# Trace context automatically injected into headers
async with httpx.AsyncClient() as client:
response = await client.post(
"http://payment-service/api/charge",
json={"order_id": order_id}
)
return response.json()
<h1>W3C Trace Context header format</h1>
<h1>traceparent: {version}-{trace-id}-{span-id}-{flags}</h1>
<h1>Example:</h1>
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
---
Sampling Strategies
At high traffic volumes, tracing every request is prohibitively expensive. Sampling reduces the volume while preserving representative data.
from opentelemetry.sdk.trace.sampling import (
TraceIdRatioBased,
ParentBased,
)
<h1>Sample 10% of traces</h1>
sampler = ParentBased(root=TraceIdRatioBased(0.1))
provider = TracerProvider(sampler=sampler)
| Strategy | Coverage | Cost | Use Case |
| Always on (1.0) | 100% | High | Development, low traffic |
| Ratio-based (0.1) | 10% | Low | Production, high traffic |
| Parent-based | Inherits | Variable | Cross-service consistency |
| Tail-based | Errors + slow | Medium | Focus on problems |
| Rate-limited | N per second | Predictable | Budget-constrained |
---
Debugging with Traces
The most valuable tracing workflow is investigating slow requests and errors.
<h1>Find slow traces in Jaeger</h1>
<h1>1. Open Jaeger UI (http://localhost:16686)</h1>
<h1>2. Select service: "api-gateway"</h1>
<h1>3. Set operation: "POST /api/orders"</h1>
<h1>4. Set min duration: 2s</h1>
<h1>5. Click "Find Traces"</h1>
<h1>Jaeger API: query slow traces programmatically</h1>
curl "http://localhost:16686/api/traces?service=api-gateway&operation=POST%20/api/orders&minDuration=2s&limit=10"
For the log management strategy that complements tracing, see our log management guide. Teams building alerting on trace data should read our alerting best practices guide. For the complete observability stack including metrics, traces, and logs, check our Kubernetes observability stack guide.
---
Frequently Asked Questions
What is the performance overhead of distributed tracing?
OpenTelemetry auto-instrumentation adds 1-3% latency overhead per service and 2-5% CPU overhead for span processing. At 10% sampling rate, the overhead drops to under 0.5%. The BatchSpanProcessor exports spans asynchronously, so span export does not block request processing. Monitor the otel_sdk_span_exported_total metric to verify span export is not falling behind.
How do I correlate traces with logs?
Inject the trace ID and span ID into your log context. OpenTelemetry provides log integrations for most logging frameworks. Once correlated, clicking a trace in Jaeger shows all related log entries, and searching for a trace ID in your log system shows the matching trace.
Should I use Jaeger or Zipkin?
Jaeger for most new deployments. It has better scalability (Cassandra/Elasticsearch backends), a more feature-rich UI, and native OpenTelemetry support. Zipkin is simpler to deploy (single binary, MySQL backend) and has a larger ecosystem of community integrations. If you use Grafana, consider Tempo, which integrates natively with Grafana dashboards.
How much trace data storage do I need?
At 1,000 requests per second with 10% sampling, each trace averaging 10 spans at 1 KB per span, you generate approximately 86 GB per day (100 sampled traces/sec x 10 KB/trace). With 30-day retention, plan for roughly 2.6 TB of storage. At 10,000 RPS with 5% sampling, plan for roughly 13 TB. Use Grafana Tempo with S3 backend for cost-effective storage at high volumes.
Can I add tracing to a monolith?
Yes. OpenTelemetry auto-instrumentation captures HTTP handler spans, database query spans, and external HTTP call spans automatically. You do not need microservices to benefit from tracing. In a monolith, tracing reveals which database queries are slow, which external API calls timeout, and which code paths dominate request latency.