Append-only log

Append-only log (also called a write-ahead log, commit log, or journal) is a data structure and systems primitive in which records are only ever appended — never updated, deleted, or overwritten in place. It is the simplest possible data structure: a sequence of immutable records, ordered by time, persisted to durable storage. And it is, by most measures, the most important data structure in distributed systems.

The append-only log is the architectural foundation of modern data infrastructure. It is the write-ahead log in PostgreSQL, MySQL, and SQLite. It is the commit log in Apache Kafka, Apache Pulsar, and Amazon Kinesis. It is the transaction log in Apache BookKeeper and the block log in distributed storage systems. It is the blockchain in Bitcoin and Ethereum. In every case, the same primitive — append, do not mutate — produces the properties that distributed systems need most: durability, ordering, replayability, and consensus.

The Log as Source of Truth

The central architectural principle of the append-only log is that the log is the source of truth, and all other data structures are derived from it. In a database, the write-ahead log records every modification before it is applied to the B-tree or heap file. If the database crashes, it replays the log to reconstruct the consistent state. In Kafka, the log is the primary storage: consumers read from the log, and the log is the only persistent record of events. In event-sourced systems, the application state is not stored directly; it is recomputed on demand by replaying the event log.

This principle — log-first, state-second — inverts the traditional database architecture, where the mutable table is primary and the log is secondary. The inversion is not merely aesthetic. It is a structural change that makes the system more robust. Mutable state is fragile: concurrent updates, partial writes, and crash recovery are sources of complexity and bug. Immutable logs are simple: append a record, sync to disk, acknowledge. The complexity moves from the write path to the read path, and the read path is easier to optimize because it is read-only.

Properties of the Append-Only Log

Durability. Each record is written to persistent storage and acknowledged before the system proceeds. The fsync operation — forcing the operating system to flush buffered writes to disk — is the fundamental bottleneck of log-based systems, and much of the engineering of high-performance logs is devoted to batching, group commit, and asynchronous replication that hides the latency of fsync.

Ordering. The log provides a total order of events. This order is the backbone of distributed consistency: if all nodes agree on the log order, they can reconstruct the same state. Consensus protocols like Raft and Paxos are, at their core, protocols for agreeing on the order of a distributed append-only log.

Replayability. Because the log is immutable, it can be replayed from any point in time. This enables point-in-time recovery, stream processing, and the reconstruction of derived data structures. A consumer that falls behind can catch up by reading the log. A new system can be bootstrapped by replaying the log from the beginning.

Idempotency. Appending a record is naturally idempotent if the record has a unique identifier. The system can detect duplicate appends and ignore them. This makes the log resilient to retries and network failures in a way that mutable updates are not.

From Log to Distributed System

The append-only log is the primitive from which more complex distributed systems are built. LSM-trees are log-structured: they batch writes into sorted runs that are merged in the background, with the log serving as the recovery mechanism. Delta Lake uses a transaction log to provide ACID semantics over cloud object storage. Event sourcing uses the log as the primary store and derives read models from it. The blockchain is an append-only log with a consensus protocol that makes the log tamper-evident.

The common pattern is that the log provides the durable, ordered substrate, and the mutable state is a cache that can be reconstructed. This pattern is not limited to databases. It is the architecture of modern data platforms: Kafka as the log, Flink or Spark as the processors, and databases or object storage as the derived sinks.

The Systems Critique

The append-only log is not without costs. The most obvious is storage: immutable logs grow without bound, and systems must implement retention policies, compaction, and tiered storage to manage the growth. The second is read latency: because the log is optimized for append, not for random read, queries that need random access require a derived index, which introduces consistency lag between the log and the read view.

The deepest cost is conceptual: the log-first architecture forces a functional, deterministic model of computation on systems that are often inherently stateful and non-deterministic. Not every application benefits from event sourcing. Some domains are naturally mutable: a user profile, a shopping cart, a game state. Applying the log-first model to these domains produces complexity without commensurate benefit.

The append-only log is a systems primitive, not a systems panacea. It is the right foundation for systems that need durability, ordering, and replayability. It is the wrong foundation for systems that need low-latency random access, small working sets, or simple CRUD semantics. The discipline of systems engineering is knowing which primitive applies.

The append-only log is the simplest data structure and the most consequential. Every important distributed system of the last decade — from Kafka to Bitcoin to Delta Lake — is built on it. The simplicity is not accidental. It is the source of the power. In a world of complexity, the log says: write it down, do not change it, and let the future read what the past wrote.