Explain Tempo — Distributed Tracing in Observability

3 / 4

Grafana (LGTM)

Tempo — Distributed Tracing

Store huge volumes of traces cheaply and find them by ID or by trace query.

Tempo is Grafana's distributed tracing backend, built on the same cost philosophy as Loki: store everything in object storage with minimal indexing. Tempo accepts traces in all the common formats (OpenTelemetry/OTLP, Jaeger, Zipkin) and historically focused on retrieval by trace ID — you find the trace ID from a log or a metric exemplar, then Tempo fetches the full trace. TraceQL adds the ability to search for traces by attributes (duration, status, span tags) without a separate index store.

A vast warehouse that keeps every shipment record boxed and cheap. You usually arrive already knowing the tracking number (trace_id) from a receipt (a log or metric), and the warehouse pulls that exact box instantly.

Key Concepts

Tempo's value is the glue in the LGTM stack. Exemplars on Prometheus/Mimir histograms carry sample trace IDs, so a latency panel links straight to an example slow trace; logs in Loki carry trace_id for the same jump. Because it indexes little and leans on object storage, Tempo can retain large trace volumes affordably, with sampling applied upstream (in the app or the OpenTelemetry Collector) to control how much is sent.

What the interviewer is probing.

What the interviewer is probing.

1. "What is distinctive about Tempo's design?" *Probing:* the architecture. *Stalls:* "Nothing." *Moves up:* it needs no index — traces are retrieved by id from object storage, which makes retaining 100% of traces affordable.

1. "What is distinctive about Tempo's design?"

2. "How do you find a trace without an index?" *Probing:* the obvious follow-up. *Stalls:* "You cannot." *Moves up:* by pivoting from metrics or logs that carry the trace id, or through TraceQL search where enabled.

2. "How do you find a trace without an index?"

3. "What are exemplars?" *Probing:* the connection. *Stalls:* "Examples." *Moves up:* a trace id attached to a metric sample, so a latency spike on a graph links straight to a trace that caused it.

3. "What are exemplars?"

4. "What does it accept as input?" *Probing:* the practical detail. *Stalls:* "Its own format." *Moves up:* OTLP and the common tracing protocols, so instrumentation is not tied to Tempo specifically.

4. "What does it accept as input?"