How a maintained SAP regression suite changes the economics of releases

The question

Most automation business cases are built the same way: estimate the manual effort of a regression cycle, estimate the automated effort, multiply the difference by the number of cycles per year, and present a saving. The arithmetic is fine. The trouble is that it describes a world where the suite keeps working, and does not price the thing that determines whether it does.

What we see in programmes

The cost of building an automated scenario is visible, budgeted and finite. The cost of keeping it working is invisible, unbudgeted and recurring. Because only one of those appears in the business case, the second one gets absorbed by whoever has capacity — which in practice means it stops happening once the implementation team disperses.

What follows is predictable. The suite decays unevenly. Some scenarios still work; nobody is quite sure which. A run produces failures, most of them environmental. Triage takes two days. Somebody observes that testing manually would have been faster this cycle, which is true, and the suite is skipped. Skipping it once makes it worse, because now it is further behind.

Notice that nothing here is a technology failure. The scripts were fine. What was missing was a maintenance model, and the business case never asked for one because the business case was about build cost.

A better way to think about it

The unit of value is not the script. It is the execution.

A scenario built and run once has cost money and returned a test result that a person could have produced. A scenario built once and run twelve times has returned twelve results, at a marginal cost approaching zero, in windows where a person could not have produced them at all. Everything about automation economics follows from that ratio, and the ratio is controlled by maintenance.

This reframes the business case around a different question: not “what will we save?” but “what becomes possible?”

What becomes possible is a specific and quite valuable thing: the ability to re-test the critical business processes inside a window short enough to be useful. That changes the options available to a programme. A release can be absorbed on schedule rather than deferred. A patch can be validated before it goes on. A question from the business — “is procure-to-pay still fine after that change?” — can be answered this week instead of next month.

A hundred scenarios that run every release are worth more than a thousand nobody trusts by year two.

What good looks like

Maintenance costed into the original case. A recurring line, owned by a named person, sized honestly. This makes the case look worse on paper and vastly more likely to be true.

A build rate matched to the maintenance capacity. If you can service sixty scenarios, building four hundred does not give you four hundred. It gives you an unknown number of working scenarios inside four hundred, which is worse than sixty.

Executions tracked, not just scenarios built. The health metric is how many scenarios ran in the last cycle and how many passed first time. “Scenarios automated” is a vanity number after month three.

Triage effort planned per cycle. Every run produces failures that are not defects. Budget the hours. Programmes that do not, discover the cost by having the cycle overrun.

Honest claims. Self-healing capabilities genuinely reduce maintenance for a class of change — typically identifier and layout drift in the interface. They do not maintain a scenario whose underlying business process has changed, because that is a semantic change, not a technical one. Treating the first as if it were the second is how suites get built without maintenance owners.

Questions leadership should ask

What did we spend on maintenance last year, and who did it? If nobody can answer, the suite is probably in decay.

How many automated scenarios ran in the last cycle, out of how many built?

How many hours went into triaging failures that turned out not to be defects?

If we stopped building new scenarios for six months and only maintained what we have, would the suite be more or less useful? The answer is often “more”, and it is a useful thing to notice.