When a green suite still misses the defect class that hurt you

A Melbourne payments team once showed us a dashboard of steadily rising automation counts. Builds were green. Confidence was not. The defects that had escaped were pricing rule edge cases at the API boundary—exactly the layer their UI suite treated as someone else’s problem.

Start from the last three escapes

List the last three production defects that mattered commercially or regulatorily. For each, ask which automated check could have caught it, at which layer, and whether that check exists. If the answer is “an end-to-end path might have,” you are probably automating theatre.

Re-map criteria, not tools

Release criteria already written in programme plans are often clearer than the suite. Align checks to those statements. Retire scripts that only prove the framework still runs.

What we change in workshops

We spend less time debating frameworks and more time naming owners for flaky suites that block the train. Ownership disputes surface faster than tool debates—and they explain more of the risk.