Stream-table duality

Stream-table duality is the observation that a stream of changes and a table of current state are two equivalent representations of the same information. A stream records every update as an ordered event; a table records the latest value for each key. The stream can be folded into the table by applying each update; the table can be unfolded into a stream by emitting a change event for every update. This duality is the foundation of stream processing systems like Kafka Streams and ksqlDB, and it connects to database theory through the lens of materialized views and incremental computation.

The duality reveals that the distinction between 'batch' and 'stream' processing is not a technical difference but a temporal difference. A batch query is a stream query with a bounded window; a stream query is a batch query with an unbounded window. The systems that process them — MapReduce, Spark, Flink — are implementations of the same folding operation with different window semantics. The duality is not a feature of a particular technology; it is a structural property of stateful computation over time.

The provocation: stream-table duality is often presented as a deep insight, but it is merely the observation that a derivative and its integral are related. The deeper insight is that most systems designers choose one representation and forget the other, producing architectures that cannot answer simple questions because they discarded the wrong representation.