Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!
We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!
Learn from Guru Rajesh Kumar and double your salary in just one year.
This is a fictional story about a small operations team, told to illustrate what a next day review looks like in practice. The team, the company and the incident are invented, but every detail will feel familiar to anyone who has shipped a change late on a busy afternoon.
The situation
The platform group at an online shop of moderate size had spent the afternoon rewriting part of their incident runbook. The old version was out of date: it referred to a database host that had been retired and described a restart sequence that no longer matched reality. The rewrite was clean, thorough and, in the opinion of the engineer who wrote it, finished.
It was almost six in the evening. The usual practice on this team, adopted after a painful incident the previous spring, was that any change to production runbooks waited until the next morning for a solo re-read. The engineer sighed, saved the draft as a proposed change and went home.
The morning read
Next day, before opening chat, the engineer opened the draft on a different screen, in the review view rather than the editor. The first paragraph read well. The second paragraph seemed fine. Then, on the restart sequence, something felt off.
The runbook said to restart the cache service first and the application servers second. Reading it fresh, the engineer realised that in the current architecture the order was the opposite: restarting the cache first would drop sessions while the application servers were still serving traffic, causing a burst of errors for customers. The mistake was invisible the previous evening because the engineer had been thinking about the cache as the thing that needed fixing, and the order simply followed that train of thought.
Worse, a second error sat nearby. A command in the troubleshooting section referred to a flag that had been renamed months ago. Pasted at three in the morning by a tired on-call person, it would have failed with an unhelpful message.
What the team did next
The engineer fixed both mistakes and then did something the team had begun doing routinely: asked what other part of the page might hide the same kind of problem. A scan of the rest of the document turned up one ambiguous instruction, “restart the service”, with no mention of which one. It was rewritten to name the service and the host group.
Finally, the engineer added a line at the bottom of the change description recording what the morning read had found, so the team could see the value of the habit and adjust their checklist accordingly. A short entry was added to the team’s list of recurring slips: order of operations, renamed flags, unnamed services.
The test that proved it
Two weeks later, an incident hit the cache layer in the middle of the night. The on-call engineer, who had not been involved in the rewrite, followed the runbook to the letter. Application servers first, cache second, exactly as the corrected document said. Sessions survived, the error rate barely moved, and the incident closed quickly. In the retrospective, the on-call engineer mentioned that every step in the runbook had been unambiguous, which is not something people often say about documentation.
What the review did not do
Some limits deserve a plain statement. The morning read did not replace testing, peer review or the practice of rehearsing a restart in a staging environment. It was one layer among several, and its value lay in catching a different class of error: those that come from the author’s own certainty. The team still ran their normal checks afterwards, and a colleague still looked at the finished page before it was marked as current.
The team also resisted the temptation to treat the story as proof that the habit is magic. Another engineer, reading the same draft the previous evening, might have spotted the order problem too. The point was not that the author was uniquely careless, but that the odds improve whenever someone returns to a piece of work with distance, and the cheapest person to provide that distance is the author, a day later.
Lessons worth taking away
- Make the overnight gap a team rule. It is easier to follow when nobody has to justify it.
- Change the view. Reading in a different tool or layout helped the engineer see the page rather than remember it.
- Look for families of errors. One bug usually has cousins nearby.
- Record what you find. A running list turns individual saves into improved checklists.
Training programmes that teach operational resilience often emphasise this kind of habit. The article about teaching failure so that training builds resilience for incidents makes the case that practising small, safe failures prepares teams for larger ones, and a read on the following day is among the simplest practices to begin with. Another useful pairing is observability, since good signals tell you quickly whether a change behaved as intended; the full curriculum for monitoring and observability courses sets out what such signals look like.
A note from outside operations
The same idea turns up wherever people publish information others rely on. The independent educational directory IVFHospitalsNow presents itself as a place to learn about fertility care and is explicit that it is not a clinic. A reader can tell from the first lines what the resource is for, and that clarity is the product of careful re-reading, much like the corrected runbook.
Epilogue
The engineer in this story is not exceptional. What made the difference was a rule that put a night between writing and release. Without it, the original runbook would have gone live, and the wrong restart order would have waited, patiently, for the worst possible hour. With it, the mistake died quietly on a Wednesday morning, and the only trace left was a better document and a slightly longer checklist.