YARN

YARN (Yet Another Resource Negotiator) is the cluster resource management layer of Apache Hadoop, introduced in Hadoop 2.0 to address a fundamental architectural limitation of the original Hadoop design. In Hadoop 1.x, the JobTracker managed both resource allocation and job scheduling/monitoring, creating a single point of failure and a scalability bottleneck. YARN separated these concerns: a global ResourceManager allocates cluster resources, per-application ApplicationMasters negotiate resources and monitor progress, and NodeManagers run containers on individual machines.

This separation enabled Hadoop clusters to run workloads beyond MapReduce — Apache Spark, Hive, Tez, and other processing engines could all request resources through YARN's common API. YARN transformed Hadoop from a MapReduce-specific platform into a general-purpose data operating system. But the abstraction is not free: the ResourceManager remains a single point of failure (mitigated but not eliminated by high-availability modes), and the complexity of scheduling across heterogeneous workloads introduces new classes of resource contention and starvation. YARN is a lesson in systems design: the decomposition of a monolith into services solves some problems and exports others.