"Our tests are flaky. We're shopping for a better test runner."
You are not looking at a runner problem. You are looking at a mirror.
When I am handed a "flaky" suite, the flakiness almost never lives in the tooling. It lives in the system under test, and the team has quietly agreed not to look at it. A test that passes on Tuesday and fails on Wednesday is not broken. It is often the most honest thing in the building.
Flakiness is non-determinism, and you built it in: shared state nobody resets, a sleep() standing in for a real wait, tests that depend on running order, the wall clock, or a network call to something held together with hope. Martin Fowler has written at length about non-deterministic tests, and my own view is that they are worse than no test, because it trains the whole team to shrug at a red build.
Swapping the runner to fix that is like rotating the tyres because the engine knocks. New kit, same noise, lighter wallet.
What fixes it:
- Treat each flaky test as a bug report about your design, not a nuisance to retry until green.
- Kill shared mutable state between tests: fresh fixtures, nothing left over from the last run.
- Replace every sleep() with an explicit wait on the real condition.
- Make each test own its data, so they stop fighting each other over the same row.
- Quarantine the offender loudly, with an owner and a ticket, rather than adding to the "muted temporarily since 2023" graveyard.
A "retry up to 3 times" flag launders a known defect into a green tick and calls it a pipeline.
Flaky tests are not lying to you. They are the one part of the system still telling the truth.
into your process?
A QA Review finds the gaps in 1–2 weeks. From £1,200.