Distributed Tracing: Debug Requests Across Services

Distributed tracing is a method of tracking a single request as it flows through multiple services in a distributed system, recording timing and metadata at each step so you can see the complete execution path in one view. Without it, debugging latency in microservices means guessing which of your five services is responsible for a four-second checkout.

A user reports checkout is slow. You check your services — order service looks fine, payment service looks fine, inventory service looks fine. But the checkout takes 4 seconds.

Where is the time going?

In a monolith, you'd add a profiler and see the call stack. In microservices, a single request touches 5 services. Logs are scattered across machines. Metrics show aggregate data, not individual requests. Distributed tracing gives you the call stack back.

What Is Distributed Tracing?

Distributed tracing follows a single request as it flows through multiple services. Every service records how long it took to handle its part. You see the full journey in one view.

User request: checkout
 
Order Service (200ms)
 └── Inventory Service (50ms)          ← called by Order
 └── Payment Service (3500ms) ← SLOW  ← called by Order
      └── Fraud Detection (3200ms) ← VERY SLOW ← called by Payment
 └── Notification Service (100ms)      ← called by Order
 
Total: 4 seconds. Problem: Fraud Detection.

Without tracing, you'd see "checkout is slow" and guess. With tracing, you see exactly where 80% of the time went.

Traces and Spans

Trace: The entire journey of one request. Has a unique trace ID.

Span: One unit of work within a trace (one service call, one DB query, one HTTP request). Has a span ID and records start time, end time, service name, operation name.

Parent span / child span: Spans are nested. The order service span is parent to the inventory service span.

Context: Metadata propagated between services — primarily the trace ID and parent span ID — so each downstream service knows which trace it belongs to.

Trace ID: abc-123
 
Span: order-service/checkout        [0ms ————————————— 4000ms]
  Span: inventory-service/reserve   [10ms — 60ms]
  Span: payment-service/charge      [70ms ————————— 3570ms]
    Span: fraud-service/check       [80ms ———————— 3280ms]
  Span: notification-service/send   [3600ms — 3700ms]

This waterfall view immediately shows where time is spent. Reading it like a flame graph: wide spans are slow, nested spans are child calls. When a child span is nearly as wide as its parent, the parent is spending almost all its time waiting on that child.

Propagating Context

For tracing to work, each service must pass the trace ID to the next service. This is called context propagation.

When the order service calls the payment service, it adds trace headers:

HTTP request:
  POST /charge
  traceparent: 00-abc123-def456-01
               ↑   ↑       ↑     ↑
               version trace-id span-id flags

The payment service reads traceparent, creates a child span under the same trace, and passes it along when calling fraud detection.

Without propagation, each service would start a new trace — you'd lose the connection between spans. This is the most common tracing mistake: forgetting to propagate context when making async calls, spawning goroutines, or publishing to a message queue.

OpenTelemetry: The Standard for Distributed Tracing

OpenTelemetry (OTel) is the open standard for distributed tracing, metrics, and logs. It is vendor-neutral: you instrument your code once and send data to any backend — Jaeger, Zipkin, Datadog, Honeycomb, or your own collector. The CNCF governs it, and it has replaced older standards like OpenTracing and OpenCensus.

OTel has two instrumentation modes:

Auto-instrumentation patches popular frameworks and libraries automatically — zero changes to business logic. A single setup call instruments every HTTP route, database query, and cache operation.

Manual spans give you control to trace your own business logic — checkout flows, background jobs, third-party API calls that auto-instrumentation doesn't know about.

Python Instrumentation

# Install dependencies
# pip install opentelemetry-sdk opentelemetry-exporter-otlp \
#   opentelemetry-instrumentation-fastapi \
#   opentelemetry-instrumentation-httpx \
#   opentelemetry-instrumentation-sqlalchemy
 
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
 
# One-time setup at application startup
def setup_tracing(service_name: str):
    exporter = OTLPSpanExporter(endpoint="http://localhost:4317")
    provider = TracerProvider()
    provider.add_span_processor(BatchSpanProcessor(exporter))
    trace.set_tracer_provider(provider)
 
setup_tracing("order-service")
tracer = trace.get_tracer("order-service")

Auto-instrumentation for FastAPI:

from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
 
app = FastAPI()
 
# These auto-create spans for every request, HTTP call, and DB query
FastAPIInstrumentor.instrument_app(app)
HTTPXClientInstrumentor().instrument()
SQLAlchemyInstrumentor().instrument()

Manual spans for business logic:

def checkout(order_id: str):
    with tracer.start_as_current_span("checkout") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("user.id", get_user_id())
 
        with tracer.start_as_current_span("reserve-inventory"):
            inventory_result = reserve_inventory(order_id)
 
        with tracer.start_as_current_span("charge-payment") as payment_span:
            try:
                payment_result = charge_payment(order_id)
                payment_span.set_attribute("payment.id", payment_result.id)
                payment_span.set_attribute("payment.status", "success")
            except PaymentError as e:
                payment_span.set_attribute("payment.status", "failed")
                payment_span.record_exception(e)
                payment_span.set_status(trace.Status(trace.StatusCode.ERROR))
                raise
 
        return {"status": "success"}

Node.js Instrumentation

// npm install @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-grpc
// @opentelemetry/auto-instrumentations-node
 
// tracing.js — load before your app
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
 
const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'http://localhost:4317',
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': { enabled: false }, // too noisy
    }),
  ],
});
 
sdk.start();

// Manual spans in business logic
const { trace, context } = require('@opentelemetry/api');
 
const tracer = trace.getTracer('payment-service');
 
async function chargePayment(orderId, amount) {
  return tracer.startActiveSpan('charge-payment', async (span) => {
    span.setAttribute('order.id', orderId);
    span.setAttribute('payment.amount', amount);
 
    try {
      const result = await stripe.charges.create({ amount, currency: 'usd' });
      span.setAttribute('payment.id', result.id);
      span.setAttribute('payment.status', 'success');
      return result;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      throw err;
    } finally {
      span.end();
    }
  });
}

Go Instrumentation

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
    "go.opentelemetry.io/otel/trace"
)
 
var tracer = otel.Tracer("payment-service")
 
func ChargePayment(ctx context.Context, orderID string, amount int) error {
    ctx, span := tracer.Start(ctx, "charge-payment")
    defer span.End()
 
    span.SetAttributes(
        attribute.String("order.id", orderID),
        attribute.Int("payment.amount", amount),
    )
 
    // Pass ctx downstream — the context carries the trace
    result, err := fraudService.Check(ctx, orderID)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return err
    }
 
    span.SetAttributes(attribute.String("fraud.status", result.Status))
    return nil
}

The ctx carries the trace ID. When fraudService.Check creates its own span using the same context, it automatically becomes a child of this span. If you forget to pass ctx and create a new context.Background(), you break the trace chain.

Setting Up Jaeger or Zipkin

Both Jaeger and Zipkin are open-source distributed tracing backends. Run them locally in minutes.

Jaeger Quick Start

# docker-compose.yml
services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"   # Jaeger UI
      - "4317:4317"     # OTLP gRPC (receive spans from OTel SDK)
      - "4318:4318"     # OTLP HTTP
      - "6831:6831/udp" # Jaeger Thrift (legacy)
    environment:
      - COLLECTOR_OTLP_ENABLED=true

docker-compose up -d
# Open http://localhost:16686

Zipkin Quick Start

services:
  zipkin:
    image: openzipkin/zipkin:latest
    ports:
      - "9411:9411"  # Zipkin UI + API

Point your OTel exporter at Zipkin's endpoint:

from opentelemetry.exporter.zipkin.json import ZipkinExporter
 
exporter = ZipkinExporter(endpoint="http://localhost:9411/api/v2/spans")

Reading a Flame Graph in Jaeger

Open the Jaeger UI at http://localhost:16686. Select a service, click Find Traces, then click any trace to open the waterfall view.

How to read it:

• Horizontal axis = time. Wider bars = more time spent.
• Vertical axis = depth. Child spans appear below their parent.
• Red spans = errors. Click to see the exception.
• Tags tab = all attributes you set on that span.
• Logs tab = any structured log events recorded on the span.

A healthy trace has narrow bars at each level. A problematic trace has one very wide bar that dominates its siblings — that is the bottleneck.

Jaeger vs Zipkin

For new projects starting today, Jaeger with OTLP is the safe default.

Distributed Tracing with Kubernetes

In Kubernetes, you have two approaches: instrument each service manually, or use a service mesh to auto-inject tracing at the infrastructure layer.

Sidecar Injection with OpenTelemetry Operator

The OpenTelemetry Operator runs in your cluster and auto-injects the OTel collector as a sidecar container into your pods. No code changes required.

# Install the operator
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
 
# Create a collector instance
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel-collector
spec:
  mode: sidecar
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
    exporters:
      jaeger:
        endpoint: jaeger-collector:14250
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [jaeger]

# Annotate your Deployment to get the sidecar injected
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  template:
    metadata:
      annotations:
        sidecar.opentelemetry.io/inject: "true"  # ← this triggers injection

Service Mesh Auto-Tracing with Istio

Istio injects an Envoy sidecar proxy into every pod. Envoy handles mTLS, traffic policies, and — relevant here — automatic distributed tracing without touching application code.

# Install Istio with Jaeger integration
istioctl install --set profile=demo
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/jaeger.yaml
 
# Label namespace for automatic sidecar injection
kubectl label namespace default istio-injection=enabled

Istio automatically:

• Generates trace IDs for inbound requests
• Propagates traceparent / b3 headers between services
• Reports span data to Jaeger

Caveat: Istio handles network-level spans (HTTP calls between services), but it cannot create spans for in-process operations like database queries or business logic. For full coverage you still need OTel SDK instrumentation inside your app. Treat Istio tracing as the outer frame, OTel as the inner detail.

Linkerd Auto-Tracing

Linkerd is a lighter alternative to Istio. Its tracing integration requires adding a few annotations:

# values.yaml for Linkerd installation
tracing:
  enabled: true
  collector:
    name: otel-collector
    namespace: tracing

Linkerd propagates b3 headers and emits spans for all proxied traffic. Same caveat as Istio: only inter-service spans, not in-process operations.

Sampling Strategies

You cannot trace 100% of requests in production. At 10,000 requests per second, storing every span is prohibitively expensive — storage costs spike, and span ingestion adds latency. Sampling is not optional; it is a design decision.

Head-Based Sampling

The decision to sample is made at the start of the request, before processing begins. Simple and low-overhead.

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased, ParentBased, ALWAYS_ON, ALWAYS_OFF
 
# Sample 5% of all requests
sampler = ParentBased(root=TraceIdRatioBased(0.05))
 
provider = TracerProvider(sampler=sampler)

ParentBased respects the sampling decision from the upstream service. If the caller sampled the trace, downstream services continue sampling it even if they would otherwise drop it. This keeps traces complete.

Limitation: You decide at the start whether to sample. You cannot keep a trace because it turned out to be slow or errored — that information is not yet available.

Tail-Based Sampling

The decision is made after the request completes, based on its properties (error status, latency). This requires buffering spans until the full trace arrives.

OTel Collector supports tail-based sampling:

# otel-collector-config.yaml
processors:
  tail_sampling:
    decision_wait: 10s       # wait up to 10s for all spans in a trace
    num_traces: 50000        # buffer size
    policies:
      - name: errors-policy
        type: status_code
        status_code: {status_codes: [ERROR]}   # always keep errors
 
      - name: slow-requests
        type: latency
        latency: {threshold_ms: 1000}          # always keep requests > 1s
 
      - name: probabilistic-base
        type: probabilistic
        probabilistic: {sampling_percentage: 5} # sample 5% of everything else

Recommended Sampling Rates

Always-sample rules for errors and slow requests ensure that the exact traces you care most about are never dropped.

What to Tag on Spans

Attributes make traces searchable. Tag things you will want to filter on later:

with tracer.start_as_current_span("charge-payment") as span:
    span.set_attribute("payment.amount", amount)
    span.set_attribute("payment.currency", "USD")
    span.set_attribute("user.id", user_id)
    span.set_attribute("order.id", order_id)
 
    try:
        result = payment_service.charge(amount)
        span.set_attribute("payment.id", result.id)
        span.set_attribute("payment.status", "success")
    except PaymentError as e:
        span.set_attribute("payment.status", "failed")
        span.set_attribute("error.message", str(e))
        span.record_exception(e)
        span.set_status(trace.Status(trace.StatusCode.ERROR))
        raise

Now you can search Jaeger for payment.status=failed and see every failed payment trace with full context.

Correlating Traces with Logs and Metrics

Tracing is one of the three pillars of observability. The other two are logs and metrics. Used together, they give you complete visibility:

• Metrics alert you: "Payment error rate crossed 2% at 14:00."
• Traces locate it: "These 40 requests all failed in fraud-service/check."
• Logs explain it: "Fraud API returned 503 — upstream timeout after 3000ms."

The link between traces and logs is the trace ID. Inject it into every log line:

import logging
from opentelemetry import trace
 
class TraceIDFilter(logging.Filter):
    def filter(self, record):
        span = trace.get_current_span()
        ctx = span.get_span_context()
        if ctx.is_valid:
            record.trace_id = format(ctx.trace_id, '032x')
            record.span_id = format(ctx.span_id, '016x')
        else:
            record.trace_id = "none"
            record.span_id = "none"
        return True
 
# Apply to your logger
logging.getLogger().addFilter(TraceIDFilter())
 
# Now every log line includes trace_id and span_id
logger.info("Payment processing started", extra={
    "order_id": order_id,
    "amount": amount,
})
# Output: {"message": "Payment processing started", "trace_id": "abc123...", "span_id": "def456..."}

With trace IDs in logs, the workflow becomes:

1. Alert fires — metric crosses threshold.
2. Open Jaeger, filter by time range and service.
3. Find the slow/failed traces.
4. Click a trace, note the span where the error occurred, copy its trace ID.
5. Search your log aggregator (Loki, Elasticsearch) for that trace ID.
6. See all log lines from all services for that exact request.

This is why structured logging matters — you cannot grep a trace ID out of unstructured log text at scale.

Connecting Metrics to Traces with Exemplars

Prometheus supports exemplars: sample trace IDs attached to metric data points. When your dashboard shows a spike in payment latency, an exemplar links directly to a specific trace that was slow at that moment.

from prometheus_client import Histogram
from opentelemetry import trace
 
payment_duration = Histogram('payment_duration_seconds', 'Payment processing time')
 
def charge_payment(order_id):
    with payment_duration.time():
        with tracer.start_as_current_span("charge") as span:
            ctx = span.get_span_context()
            # Prometheus will record the trace ID alongside this observation
            payment_duration.observe(elapsed, exemplar={'traceID': format(ctx.trace_id, '032x')})

Common Tracing Mistakes

1. Missing context propagation in async code

The most common mistake. If you create a goroutine, thread pool task, or async job and do not pass the context, the new work starts a fresh trace with no parent.

# WRONG — context is lost
async def process_order(order_id):
    asyncio.create_task(send_notification(order_id))  # no context
 
# CORRECT — propagate context explicitly
async def process_order(order_id):
    ctx = context.copy_context()
    asyncio.create_task(ctx.run(send_notification, order_id))

2. Wrong span naming

Span names drive search and aggregation. Generic names like "http-request" or "db-call" are useless.

# WRONG
with tracer.start_as_current_span("db-call"):
    ...
 
# CORRECT — operation type + resource
with tracer.start_as_current_span("db.query users.find_by_email"):
    ...

OTel semantic conventions define standard names: db.query, http.request, rpc.call. Follow them so tools like Jaeger can display them correctly.

3. Not logging the trace ID

You have traces in Jaeger and logs in Elasticsearch but no way to connect them. Always inject the trace ID into log records (see the logging section above).

4. Tracing 100% of requests without sampling

At any meaningful scale this destroys your budget and adds latency. Start with 5–10% and adjust.

5. Creating spans that are too granular

Spans for every function call produce millions of spans per request and make the waterfall unreadable. Create spans for I/O boundaries (HTTP calls, DB queries, cache operations) and significant business operations (checkout, payment, fraud check). Skip pure computation.

6. Not marking error spans correctly

A span that throws an exception should have its status set to ERROR. Without this, Jaeger cannot highlight errors in red and you cannot filter for failed traces.

except Exception as e:
    span.record_exception(e)
    span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
    raise

Tracing vs Logging vs Metrics

These three are complementary, not alternatives:

Metrics: Aggregate numbers over time. "Payment failures increased to 2% at 14:00." Fast to query, no per-request detail.

Logs: Text output from services. "Payment failed for order-123." Good detail but hard to correlate across services without a trace ID.

Traces: Follow one request across all services. "This specific checkout took 4s because fraud check took 3.2s." Essential for multi-service debugging.

The ideal setup: metrics alert you ("something is slow"), traces help you find it ("here's which service and which span"), logs give you the details ("here's the exact error").

Key Takeaways

• Distributed tracing follows a single request through multiple services — you see the full timeline
• Traces consist of spans; spans are nested (parent/child) to show call hierarchy
• Context propagation passes trace IDs between services via HTTP headers
• OpenTelemetry is the standard SDK — instrument once, send to any backend
• Jaeger (open source) or Datadog/Honeycomb (commercial) visualize traces
• Sample 1–10% of traffic — always sample errors and slow requests
• Tag spans with IDs and status so you can search for specific failed/slow requests
• Inject trace IDs into logs to correlate traces with log output

Distributed tracing turns "something is slow somewhere" into "this specific service, this specific operation, at this exact time." Set it up before you need it.

FAQ

What is distributed tracing and why do I need it?

Distributed tracing tracks a single request as it flows through multiple services in a system, recording timing and metadata at each hop. You need it because logs and metrics alone cannot tell you which service or operation caused a slowdown for a specific request. When a user reports slow checkout, tracing shows you the exact service and the exact span where time was lost — without tracing, you are guessing.

What is the difference between a trace, span, and context?

A trace is the complete record of one request's journey through your system, identified by a unique trace ID. A span is one unit of work within that trace — a single service call, database query, or HTTP request — with its own start time, end time, and metadata. Context is the in-process and over-the-wire representation of which trace and span are currently active, propagated via HTTP headers like traceparent so downstream services know they are part of the same trace.

What is OpenTelemetry and should I use it?

OpenTelemetry is the CNCF-governed open standard for distributed tracing, metrics, and logs. It provides SDKs for all major languages, a vendor-neutral wire protocol (OTLP), and a collector that routes data to any backend. Yes, you should use it. It replaced OpenTracing and OpenCensus, is now the industry default, and prevents vendor lock-in — switching from Jaeger to Datadog requires changing one line of exporter configuration, not re-instrumenting all your code.

How does distributed tracing work in microservices?

Each service participating in a request creates one or more spans. When service A calls service B, it attaches the current trace ID and span ID as HTTP headers (traceparent). Service B reads those headers, creates a child span under the same trace, and continues the chain. A central backend (Jaeger, Zipkin, Datadog) collects all spans and assembles them into a single trace tree using the shared trace ID. The result is a waterfall view of every service and operation involved in handling the request.

What sampling rate should I use for production tracing?

Start at 5–10% for services under 1000 requests/second. At higher traffic, 1–5% is typical. Always override sampling for high-value cases: sample 100% of error traces and 100% of requests that exceed your SLA threshold (e.g., > 1 second). Use tail-based sampling via the OTel Collector if you need to make sampling decisions based on outcome rather than at request start.

What is the difference between Jaeger and Zipkin?

Both are open-source distributed tracing backends that collect, store, and visualize traces. Jaeger was built at Uber, is now a CNCF project, and has native OTLP support, better Kubernetes integration, and a richer UI. Zipkin originated at Twitter, is mature and stable, and is well-supported in Java ecosystems (Spring Boot has built-in Zipkin support). For new projects, Jaeger is the more common choice. For teams already on Spring Boot or older Java stacks, Zipkin may require less configuration.

How do I correlate traces with logs?

Inject the trace ID and span ID from the active span into every structured log record. In Python, use a logging.Filter that reads trace.get_current_span().get_span_context() and adds trace_id and span_id fields to each log record. In Node.js, use trace.getActiveSpan()?.spanContext(). Ship structured JSON logs to a log aggregator (Loki, Elasticsearch). When investigating a trace in Jaeger, copy the trace ID and search for it in your log tool — every log line from every service for that request will appear.

Related reading: Circuit Breaker Pattern · Message Queues Explained · API Gateway Pattern