The engineering problem
Distributed systems fail in combinations that design documents rarely enumerate. Netflix's Simian Army made selected failures routine so teams could discover whether redundancy, failover, and recovery mechanisms worked before an uncontrolled incident supplied the test.
The approach
The memorable names can obscure the engineering discipline: inject a known class of failure, constrain when and where it occurs, observe user and system behavior, and improve automatic recovery. The system turns resilience from an architectural assertion into an operationally testable property.
Tradeoffs
Experiments consume operational attention and can cause real harm when blast radius, abort conditions, observability, or ownership are weak. The safest early experiment is rarely random destruction. It is a narrow hypothesis tested during staffed hours with clear steady-state signals.
What generalizes
Teams should identify their most important recovery assumptions and exercise them continuously: instance loss, dependency latency, stale data, queue backlog, credential failure, or regional isolation. Reliability improves when recovery is practiced rather than documented once.
What is context-dependent
Netflix's availability needs and cloud architecture justified a sophisticated program. Lower-scale or higher-regulation systems may need simulations, game days, or pre-production fault injection before production experiments.
Architecture review questions
- What failure are we claiming the system survives?
- Which signal defines steady state from the user's perspective?
- What limits the blast radius and stops the experiment?
- Can the system recover without a particular expert online?
The Simian Army description is based on Netflix's publication; the modern experiment framing is my synthesis.