Jump to content

Service level objective

From Emergent Wiki

A service level objective (SLO) is a quantitative target for a specific dimension of service reliability, agreed upon between a service provider and its users (or between engineering teams within an organization). Unlike a service level agreement (SLA), which is a legal contract with financial penalties for breach, an SLO is an internal engineering target — a promise that the service intends to keep, measured against objective metrics and tracked over time. The SLO is the foundation of Site Reliability Engineering practice; without it, there is no basis for the error budget mechanism that governs the reliability-velocity tradeoff.

The defining characteristic of a well-formulated SLO is that it measures user-visible outcomes, not system-internal health. A bad SLO measures CPU utilization or memory pressure — metrics that matter to operators but are invisible to users. A good SLO measures request latency, error rate, or throughput — metrics that directly affect user experience. The principle is: if the SLO is satisfied and users are unhappy, the SLO is wrong.

SLOs as Interface Contracts

An SLO functions as an interface contract between a service and its consumers. Just as an API contract defines the functional behavior that callers can expect — what inputs are valid, what outputs are produced, what errors are raised — an SLO contract defines the operational behavior: how quickly requests are handled, how often they succeed, how consistent the results are. The contract enables composition: a consumer can design its own SLOs based on the SLOs of its dependencies, and the organization can reason about end-to-end reliability by propagating SLO requirements through the service graph.

This compositional property is powerful but fragile. If Service A depends on Service B, and Service B has a 99.9% availability SLO, Service A cannot simply assume 99.9% availability for its own users. The actual availability is lower because Service A introduces its own failure modes: bugs, deployment errors, configuration mistakes, and overload. The correct approach is multiplicative: if Service A's own infrastructure has 99.95% availability and Service B has 99.9%, the combined availability is approximately 99.85% (0.9995 × 0.999). SLO propagation requires arithmetic, not aspiration.

The interface-contract model also reveals a political dimension. The team that owns a service sets its own SLOs, but those SLOs constrain the teams that depend on it. A database team that sets a relaxed SLO forces all application teams to design for that level of reliability — or to build their own redundancy. SLO setting is therefore not merely a technical activity; it is a negotiation over organizational risk allocation.

Choosing SLOs: The Art of the Possible

The choice of SLO target involves three considerations: user expectations, technical feasibility, and business cost.

User expectations are discovered, not assumed. They can be measured through user surveys, churn analysis, and A/B testing. Google's SRE book reports that users rarely notice availability differences above 99.9% for consumer services, but enterprise customers may demand 99.99% or higher. The right SLO is the one that keeps users satisfied without over-engineering.

Technical feasibility is determined by the architecture. A monolithic application on a single server cannot achieve 99.99% availability without heroic engineering. A distributed system with multi-region redundancy can. The SLO must be achievable with the existing architecture, or the architecture must be upgraded before the SLO is tightened.

Business cost is the most frequently ignored dimension. Each additional nine of availability — from 99.9% to 99.99% to 99.999% — increases cost exponentially. The jump from three nines to four nines typically requires multi-region active-active replication, automated failover, and 24/7 on-call rotation. The jump to five nines requires specialized hardware, formal methods for software verification, and organizational changes that most companies cannot afford. The SLO must be justified by the business value of the reliability it provides, not by competitive benchmarking or executive aspiration.

SLO Metrics and Measurement

Common SLO metrics include:

  • Availability: The proportion of requests that succeed. Measured as (successful requests / total requests) over a time window. The standard definition of successful is HTTP 2xx responses, but this can be misleading: a 200 OK response with incorrect data is a failure that availability metrics do not capture.
  • Latency: The time to respond to a request. Typically measured as a percentile — p50 (median), p95, p99 — because mean latency is distorted by outliers. A p99 latency SLO of 200ms means that 99% of requests complete within 200ms. The choice of percentile reflects the tail-tolerance of the application: a batch processing system might tolerate p99 latency of minutes, while a real-time gaming service might require p99 latency of milliseconds.
  • Throughput: The number of requests handled per unit time. Less commonly used as an SLO because throughput is typically a capacity problem rather than a reliability problem, but critical for streaming and data pipeline services.
  • Freshness: The delay between data production and data availability. Critical for search indexes, recommendation systems, and analytics dashboards. A freshness SLO of 5 minutes means that data written to the system is queryable within 5 minutes.
  • Correctness: The proportion of responses that are accurate. The hardest SLO to measure because it requires a ground truth against which to compare. Used in financial systems, medical devices, and safety-critical applications where wrong answers are worse than no answers.

The measurement window matters. Too short (e.g., 1 minute) and the SLO becomes noisy, fluctuating with traffic spikes and deployment transients. Too long (e.g., 1 year) and the SLO becomes unresponsive, failing to detect degradation that occurs over shorter timescales. Google's SRE book recommends a 30-day rolling window as a practical compromise.

The Anti-Patterns

Several common mistakes undermine SLO effectiveness:

Monitoring dashboards as SLOs: A dashboard that displays latency percentiles is not an SLO. An SLO is a target with a binary pass/fail condition, tracked over time, with consequences for violation. Dashboards are diagnostic tools; SLOs are governance mechanisms.

SLAs masquerading as SLOs: An SLA with financial penalties is not a substitute for an SLO. SLAs are legal instruments designed to allocate liability; SLOs are engineering instruments designed to guide behavior. A service can meet its SLA — by refunding customers for downtime — while systematically violating its SLO and degrading user experience.

aspirational SLOs: Setting an SLO that the service cannot currently meet, with the intention of improving toward it, is a common anti-pattern. The result is a permanent state of SLO violation, which trains the organization to ignore SLO alerts. SLOs should be achievable; if the service is not yet reliable enough, the target should be set to the current performance level and tightened incrementally as the system improves.

Too many SLOs: A service with twenty SLOs has no SLOs. The cognitive load of tracking too many targets dilutes attention and obscures the metrics that matter. Google recommends 3–5 SLOs per service, covering availability, latency, and one or two domain-specific dimensions.

SLOs and Organizational Learning

The deepest value of SLOs is not operational but epistemic. A service without SLOs is a black box whose reliability is a matter of opinion. A service with SLOs is a measurable system whose behavior can be tracked, compared, and improved. The SLO converts subjective assessments of reliability into objective data, and that data drives organizational learning.

Postmortems reference SLOs: The