Writing runbooks your on-call engineer can follow at 3am
Most runbooks are written by the person who understands the system best, which is exactly the problem. They are written from the perspective of someone who already knows what is wrong, for a reader who does not.
Start with the alert, not the architecture
The reader arrived from a page alert, not from a browse of your documentation. The first line should confirm they are in the right place: name the alert exactly as it appears in the alerting tool, and state in one sentence what it means when it fires.
Architecture context is useful, but it belongs below the immediate steps. Anyone who needs the background will scroll for it; anyone who needs to stop the bleeding will not.
Write commands that can be copied without editing
Placeholders are where runbooks fail at three in the morning. A command containing an angle-bracketed variable is a command that will be run incorrectly at least once. Where a value is required, say precisely where to find it and show the command that retrieves it.
Include the expected output as well as the command. A tired reader needs to know whether what they are looking at is normal, and comparing against a printed example is much faster than reasoning about it.
Say what to do when the step does not work
Every recovery step should end with two branches: what to do if it worked, and what to do if it did not. Without the second branch the reader is stranded at the exact moment the runbook was supposed to help.
Include the escalation path with names of rotations rather than individuals, and state the threshold explicitly. Fifteen minutes without recovery is a clear instruction; use your judgement is not.
Prove it by following it
A runbook is verified the way any procedure is verified: someone who did not write it follows it, exactly as written, in a game day or against staging. Every place they hesitate is a defect in the document, not in the reader.
Record the date of that exercise on the page. It tells the next reader that the steps were true at a known point in time, which is far more reassuring than a page with no such evidence at all.
Written by the DocuRail Team.
Get the next one
One piece a month on documentation practice.
