The incident that taught me the most about quality didn't happen in a test lab. It happened at 2am, in production, staring at a dashboard while a regression suite that had gone green four hours earlier sat there smugly, having tested absolutely nothing that mattered.
I had spent the best part of three years building that suite: nine thousand assertions, a coverage badge nudging 94%. It caught every bug we had already thought of. It caught none of the one that took the payments service down, a timeout cascading through a queue nobody had drawn on the architecture diagram in two years.
Charity Majors makes the point better than the 2am version of me managed: you cannot pre-test your way to confidence in a complex distributed system. Testing tells you the system behaves as expected under conditions you imagined. Production will always imagine conditions you did not.
It is like passing your MOT on Tuesday and having the exhaust drop off on the M25 on Thursday. The test was real. It just was not testing the thing that broke.
What changed after that incident was not more tests:
- Structured logging and traces that let us ask new questions of a live system, not just re-ask old ones.
- Feature flags, so a bad change was a toggle, not an incident review.
- Canary releases that let 2% of traffic find the problem before 100% of customers did.
- A hard rule: every postmortem produces one new signal, not one new test case.
The suite still runs. Nobody is precious about it any more. It catches the bugs we already know to be afraid of, and observability catches the ones we do not.
Nine thousand green checkmarks felt like safety. They were closer to a very expensive nightlight.
into your process?
A QA Review finds the gaps in 1–2 weeks. From £1,200.