Grafana Tempo
Grafana Tempo is a distributed tracing backend developed by Grafana Labs, designed to store and query trace data at scale with an emphasis on cost efficiency and operational simplicity. Unlike earlier tracing systems such as Jaeger or Zipkin, which required extensive indexing of trace attributes and span metadata, Tempo adopts a storage model similar to Grafana Loki: it stores traces as compressed objects and retrieves them by trace ID, relying on external correlation mechanisms rather than internal indexing to connect traces with metrics and logs. This design reflects a growing consensus in the observability community that traces are not meant to be searched arbitrarily — they are meant to be retrieved precisely, in response to a specific need, and then correlated with the metrics and logs that provide context.
Architecture: Object Storage and the Trace ID
Tempo's architecture is built around a single insight: most trace queries are lookups by trace ID, not searches across arbitrary span attributes. When an operator sees a spike in error rate in Prometheus metrics or an anomalous log line in Grafana Loki, they do not need to search all traces for spans with a particular tag. They need to retrieve the specific trace associated with that error or log line and examine its full span tree. Tempo optimizes for this workflow by storing traces as compressed objects in cheap object storage — Amazon S3, Azure Blob Storage, or Google Cloud Storage — and indexing only the trace ID to object mapping.
This approach reduces storage costs by an order of magnitude compared to systems that index every span attribute. But it introduces a dependency: Tempo is not a standalone observability tool. It requires integration with metrics and logs to be useful. A trace without a corresponding metric or log entry is a needle in a haystack that cannot be found. Tempo assumes that the operator discovers the need for a trace through some other signal — an alert, a dashboard panel, a log line — and then retrieves the trace by ID. This is not a limitation; it is a design philosophy. Tempo does not try to be the center of the observability universe. It tries to be a cheap, reliable repository that other tools can reference.
The trace data is ingested through the OpenTelemetry Protocol (OTLP) or via Tempo's native protocols, and the system supports trace aggregation through the OpenTelemetry Collector, which can batch, sample, and route trace data before it reaches Tempo. Sampling is critical because distributed tracing generates data at volumes that can dwarf metrics and logs combined. A single request through a microservices architecture with fifty services might generate hundreds of spans, each with dozens of attributes. Without sampling, the storage costs become prohibitive regardless of the storage model.
TraceQL and the Limits of Span Search
Tempo provides TraceQL, a query language designed specifically for trace data. TraceQL allows operators to search for traces based on span attributes, duration, and structural properties — for example, finding all traces where a database span took longer than 500ms and was followed by an error span. The language is powerful but constrained by the storage model: because Tempo does not index all span attributes, TraceQL queries that filter on unindexed fields require scanning large amounts of data, which is slow and expensive.
The recommended practice is to use TraceQL for targeted queries — find traces by service name, by time range, by trace ID — and to rely on correlation with metrics and logs for discovery. This workflow is enabled by Grafana's unified querying interface, which allows operators to move from a metric panel to a log query to a trace view without switching contexts. The trace view in Grafana displays the span tree, the timing relationships, and the associated log lines, creating a coherent narrative of a single request's journey through the system.
But this coherence is fragile. It depends on the instrumentation being consistent across all services — the same trace IDs propagated through headers, the same span attributes attached to logs, the same service names used in metrics and traces. In practice, achieving this consistency requires organizational discipline that many teams lack. A trace that spans services with inconsistent instrumentation is a broken narrative: the spans exist, but they cannot be connected to the metrics and logs that would make them meaningful. Tempo does not solve this problem. It provides the storage. The integration is the team's responsibility.
Tempo and the Politics of the Grafana Stack
Tempo is part of the broader Grafana observability stack, alongside Grafana Loki for logs, Grafana Mimir for metrics, and Grafana OnCall for incident response. The strategic logic is clear: provide a complete observability platform that competes with proprietary solutions like Datadog, New Relic, and Dynatrace, but built on open-source components and monetized through managed services. Tempo's cost efficiency is a key selling point for this platform, because trace storage has historically been the most expensive component of observability infrastructure.
The risk is the same as with other components of the Grafana stack: vendor coupling. Tempo can export data to other systems, and its APIs are open, but the optimal experience — the correlated metrics-logs-traces workflow that justifies Tempo's existence — is achieved through Grafana. Teams that adopt Tempo are implicitly adopting Grafana as their observability frontend, and the switching costs accumulate in dashboard definitions, alert rules, and operator training. Open source does not prevent lock-in; it merely relocates it from data formats to user experience.
Grafana Tempo is a bet that distributed tracing has been over-engineered. The tracing community spent a decade building systems that could index and search every span attribute, only to discover that most operators never used those capabilities. Tempo says: store traces cheaply, retrieve them by ID, and let metrics and logs do the discovery. It is a minimalist intervention in a field that had become maximalist. Whether minimalism wins depends on whether organizations can achieve the instrumentation consistency that Tempo requires but does not enforce. The tool is simple. The organizational problem it addresses is not.