Distributed Tracing

Server-side metrics measure what the database did. Tracing measures what the request experienced — and the gap between the two is where connection pools, retries and N+1 queries hide.

What a database span should carry

with tracer.start_as_current_span("db.query") as span:
    span.set_attribute("db.system", "postgresql")
    span.set_attribute("db.name", "shop")
    span.set_attribute("db.operation", "SELECT")
    span.set_attribute("db.sql.table", "orders")
    # The normalised statement, never the one with literal values.
    span.set_attribute("db.statement", "SELECT * FROM orders WHERE customer_id = $1")
    span.set_attribute("db.pool.wait_ms", pool_wait_ms)

Instrument the pool separately

The single most valuable database tracing attribute is connection acquisition time, as a span or an attribute of its own. It distinguishes three situations that look identical from the outside:

Finding N+1 patterns

A trace showing one request with 200 nearly identical short database spans is the clearest possible diagnosis of an N+1 query. No server-side metric reveals it — each individual query is fast, and the total looks like ordinary traffic.

Alert on span count per request rather than only on duration.

Correlating with the database

Propagate the trace id into the database session so slow query logs and traces can be joined:

-- PostgreSQL: appears in log_line_prefix with %a
SET application_name = 'checkout-service:trace-4f2a9c';

-- Or as a comment the engine records in its statistics.
SELECT /* trace_id=4f2a9c */ * FROM orders WHERE customer_id = $1;

Comment-based propagation is supported by several APM tools and lets a slow query in pg_stat_statements or the slow log be traced back to the request that issued it.

Sampling

Tracing every request is expensive. A workable arrangement:

• Sample a small percentage of normal traffic for baseline behaviour.
• Always sample errors.
• Always sample requests exceeding a latency threshold — tail-based sampling, where the decision is made after the request completes.

Tail-based sampling is what makes tracing useful for performance work: you keep the slow requests, which are the ones worth looking at.