Day 110: Runbooks and incident command
Runbooks: encoding what you'd otherwise have to remember at 3am
A runbook is a written, step-by-step guide for responding to a specific known alert — "when this fires, check X, and if Y, do Z." Good runbooks reduce time-to-resolution and reduce the odds an exhausted on-call engineer improvises the wrong fix under pressure.
Incident command
For serious incidents, having one person as Incident Commander — coordinating, communicating, delegating — while others actively investigate, prevents the chaos of everyone doing everything (and nothing) at once. This role doesn't have to be the most senior engineer or the one fixing the bug; its job is coordination, not technical heroics.
Why separating "commander" from "fixer" matters
A single person trying to both deeply debug the problem AND field status update requests AND coordinate other responders is a recipe for both a slower fix and worse communication. Splitting these roles explicitly is a small process change with an outsized effect on incident duration.
Key terms
- Runbook
- A written, step-by-step response procedure for a specific known alert or failure mode.
- Incident Commander
- The person coordinating an incident response, distinct from whoever is actively debugging.
Why does having a dedicated Incident Commander role improve incident response, even if they're not the one fixing the bug?
