Past year
Reframing incident response from heroics to hygiene
Engineering Manager
Inherited a team where the on-call engineer was a single hero and the runbooks were three years stale. Rewrote the incident-response model — distributed knowledge, mandatory blameless retros, and a runbook-as-code culture.
- Mean-time-to-mitigate cut by ~40%
- On-call pages distributed across the team — no single hero
- Runbooks now treated as code: PR-reviewed, version-controlled, tested
What I walked into
One engineer was answering 70% of pages. He was excellent — and burning out quietly. The rest of the team had implicitly opted out of operational excellence because it was his job. Runbooks were a wiki graveyard.
The fix was cultural before it was technical
- Pages now route to whoever is on shift. The hero engineer became the trainer, not the responder.
- Every page produces a tiny artefact: either an updated runbook or a flagged-as-broken note. Boring; the boredom is the point.
- Retros are blameless and 30 minutes max. We focus on what made the system fragile, not who was holding it when it broke.
- Runbooks moved into the same repo as the code. PRs against runbooks are reviewed like PRs against code. Stale runbooks fail CI.
A surprising side-effect
The team started catching fragility before the alert fired. Once you treat operational hygiene as a shared craft rather than a cost, people start enjoying the work. Pages went from a punishment to a small, manageable part of the rhythm.
What I would teach a first-time manager
The “10x engineer” you’re protecting is often the symptom of a system you have not yet fixed.