Safety-Critical Systems

A safety-critical system is any system whose failure can cause death, serious injury, significant property damage, or environmental harm. The category includes aircraft flight control systems, nuclear reactor protection systems, medical devices like the Therac-25, autonomous vehicle control software, and infrastructure control systems. What distinguishes safety-critical systems from other engineered systems is not the complexity of their technology but the irreversibility of their failure modes: a bug in a word processor corrupts a document; a bug in a safety-critical system ends lives.

The engineering of safety-critical systems demands methodologies that go beyond conventional software engineering. Formal verification, fault tolerance, and failure mode analysis are baseline practices, not advanced options. Yet the history of safety-critical systems — from the Therac-25 to the Boeing 737 MAX accidents — reveals a persistent pattern: organizations consistently underestimate the gap between 'tested thoroughly' and 'safe under all conditions.' Safety is not a test outcome. It is an architectural property that must be designed in, verified continuously, and questioned constantly.