Safety Engineering

Safety engineering is the discipline of designing systems that do not fail catastrophically, not by eliminating all possible failures but by ensuring that the failures that do occur are contained, survivable, and recoverable. It is not the same as reliability engineering — a reliable system that fails predictably is not necessarily safe, and a safe system that fails in known, bounded ways may be deliberately less reliable than technically possible.

The field emerged from the study of high-risk technologies — nuclear power, aviation, chemical processing, spaceflight — where single failures could produce mass casualties. But its principles apply to any system where the cost of failure exceeds the cost of prevention: software infrastructure, financial systems, medical devices, and increasingly, machine learning systems whose failures propagate silently through automated decisions.

From Absence to Capacity: The Shift in Safety Thinking

Traditional safety thinking defined safety as the absence of accidents: a safe system is one that has not yet had an accident. This definition is retrospective and passive. It tells you nothing about whether the next hour will be safe.

Modern safety engineering, influenced by the work of Sidney Dekker, Erik Hollnagel, and Nancy Leveson, defines safety as the presence of capacity: a safe system is one that can absorb perturbation, adapt to surprise, and recover from unexpected states. This is the difference between "safety-I" (preventing things from going wrong) and "safety-II" (ensuring things go right). The shift is not semantic. It changes what you measure, what you design for, and what you reward.

In safety-I, success is measured by the absence of incidents. In safety-II, success is measured by the presence of resilience: the system\s

Safety in Complex Adaptive Systems

The safety-II framework becomes genuinely challenging when applied to systems that are not merely complicated but complex — systems in which the behavior of the whole cannot be predicted from the behavior of the parts, and where causal influence propagates through feedback loops rather than linear chains. In complex adaptive systems, the distinction between a 'normal' state and a 'failure' state is not always well-defined. A financial market, a social media recommendation ecosystem, or a climate model exhibits behaviors that are simultaneously functional and pathological depending on the time scale and the observer's framing.

Machine learning systems present a particularly acute version of this problem. Unlike mechanical failures, which are localized and observable, the failures of automated decision systems are often distributed, gradual, and epistemically inaccessible: the system does not 'break' so much as it slowly drifts into a regime where its predictions are systematically biased in ways that no single metric captures. The concept of epistemic safety — the property of a system that it knows the limits of its own knowledge, and can signal when it is operating outside its domain of competence — has been proposed as a complement to traditional safety engineering. A system that is resilient but epistemically blind may recover from perturbations without ever recognizing that it is systematically misrepresenting the world.

This suggests that the safety-I/safety-II distinction may itself be insufficient. What complex adaptive systems require is not merely the capacity to recover from surprise, but the capacity to recognize when the surprise signals a fundamental change in the system's operating environment — a shift that requires not adaptation within the current frame, but a reframing of what the system is trying to do. Safety, in this view, is not a property of the system at all. It is a property of the relationship between the system and the observers who are capable of recognizing its limitations.