Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud in 2012 and now a graduated project of the Cloud Native Computing Foundation. It is designed for monitoring dynamic, cloud-native environments — particularly those built on containers and microservices — where traditional host-centric monitoring tools fail to capture the ephemeral, auto-scaled nature of modern infrastructure. Prometheus does not merely collect metrics; it imposes a specific epistemology on observability, one in which the pull model, dimensional time-series data, and the PromQL query language together constitute a theory of how distributed systems should be known.

Architecture: Pull, Store, Query

Prometheus's architecture rests on three design decisions that distinguish it from earlier monitoring systems. First, it uses a pull model: the Prometheus server periodically scrapes HTTP endpoints exposed by targets, rather than waiting for targets to push data to it. This inversion has profound consequences. A push model makes the monitored system responsible for delivery, which means failures in the monitoring pipeline are indistinguishable from failures in the monitored system. A pull model makes the monitoring server responsible for collection, which means a failed target simply stops appearing in the scrape results — a silent absence that is itself a signal. The pull model also simplifies service discovery: when a container orchestrator like Kubernetes launches a new pod, Prometheus discovers it through the orchestrator's API and begins scraping automatically.

Second, Prometheus stores all data as time series — sequences of timestamped values identified by a metric name and a set of key-value labels. This dimensional data model is more expressive than the hierarchical model of tools like Graphite, where a metric is identified by a single dot-separated path. In Prometheus, the metric can be broken down by , , , and , enabling ad-hoc aggregation and filtering that would require pre-aggregation in hierarchical systems. The dimensional model is powerful but costly: it generates a combinatorial explosion of time series when labels have high cardinality, and high cardinality is the leading cause of Prometheus performance degradation in production.

Third, Prometheus provides PromQL, a functional query language designed specifically for time-series computation. PromQL enables operators to aggregate, filter, and transform metrics in real time, but it is not SQL and does not behave like SQL. Its most powerful operations — range vectors, instant vectors, and subqueries — require a mental model of time-series data that is unfamiliar to engineers trained on relational databases. The learning curve is steep, and the errors are subtle: a poorly written PromQL query can appear to return correct results while silently omitting data or applying the wrong aggregation.

The Alertmanager and the Sociology of Alerting

Prometheus includes an Alertmanager component that evaluates alert rules written in PromQL and routes notifications to receivers such as email, Slack, or PagerDuty. The alert lifecycle is deceptively simple: a rule fires when a query returns data for a sustained duration, and the Alertmanager deduplicates, groups, and routes the notification. But the simplicity conceals a deep epistemic problem: an alert is a claim that something is wrong, and the claim is only as good as the query that generated it.

Alert rules in Prometheus are typically written by the same engineers who operate the systems being monitored. This creates a feedback loop in which the system's operators encode their assumptions about failure into the monitoring system, and the monitoring system then confirms those assumptions by alerting on them. An alert rule that triggers when CPU usage exceeds 80% encodes the assumption that high CPU usage is a problem; but in auto-scaled environments, high CPU usage is the desired signal that triggers scale-out. The alert may be measuring the wrong thing, or measuring the right thing at the wrong abstraction level, and Prometheus has no mechanism to detect this. The Alertmanager routes the alert; it does not validate it.

The result is that Prometheus deployments often suffer from the same alert fatigue that plagues other monitoring systems. The problem is not the tool but the epistemic architecture: when the same people who build the system also define what constitutes failure, the monitoring system becomes a mirror of the builders' biases rather than a window into the system's behavior.

Prometheus and the Cloud Native Stack

Prometheus is deeply embedded in the cloud-native ecosystem. It is the default monitoring system for Kubernetes, which exposes cluster metrics through the kubelet's endpoint. The Prometheus Operator automates deployment and configuration in Kubernetes clusters. Grafana Mimir, a commercial project by Grafana Labs, provides long-term storage for Prometheus metrics at scale, addressing the fundamental limitation that Prometheus stores data only on local disk and is not designed for horizontal scalability of its storage layer.

This ecosystem dependency is both a strength and a vulnerability. Prometheus is excellent at monitoring Kubernetes and terrible at monitoring legacy systems that do not expose HTTP endpoints or do not restart quickly enough to survive the scrape interval. The cloud-native world has built its observability around Prometheus, and in doing so has made Prometheus-shaped assumptions about what systems look like. A mainframe, a factory floor controller, or a sensor network with intermittent connectivity does not fit the Prometheus model, and the cloud-native community has largely ignored these systems rather than adapting the model to accommodate them.

The pull model is not a neutral architectural choice. It is a political statement: it says that the monitoring system is the center of gravity, that the monitored systems are its satellites, and that a target that cannot be scraped is not worth knowing. This is a defensible position in a datacenter full of containers. It is an imperial position when applied to the world beyond the datacenter. Prometheus is not merely a tool. It is a worldview, and like all worldviews, it sees clearly what it was designed to see and remains blind to everything else.