"We've got 3,000 automated tests." What's it costing you to keep them honest?
Having audited more automation suites than I care to remember, I keep finding the same thing: a dashboard shown only to stakeholders is green, and a quarantine folder nobody has opened since a sprint two years ago ships with a straight face as "coverage."
The automation pitch always skips this: once written, a test is a maintenance liability with a running invoice. Martin Fowler's test pyramid makes the point. The goal was never "automate everything". It was to choose the cheapest, fastest tests that still tell you the truth, because every test you keep is a future cost, not a trophy.
Nobody puts "ongoing suite upkeep" in the board-level business case, much as nobody mentions the cambelt when selling you the car. The number they lead with is the one that ages best in the room.
What tells you whether automation is paying off:
- Count maintenance hours per sprint against the suite. If triaging flaky failures takes longer than writing new tests, you've bought a second job, not a safety net.
- Delete tests as deliberately as you write them. A test nobody trusts is a liability in a hi-vis vest.
- Measure ROI as defects caught before production per hour of upkeep, not by test count.
- Push logic down the pyramid. A UI test doing a unit test's job is the most expensive way to fail slowly.
A big automation number impresses the board for exactly one quarter. After that someone has to maintain it, and by then it is called technical debt rather than quality investment.
into your process?
A QA Review finds the gaps in 1–2 weeks. From £1,200.