Five quality signals every SAP steering committee should see

The question

A steering committee meets for an hour, once a fortnight, and has to form a view on something enormous. The reporting it receives has usually been designed to demonstrate progress. What it needs is something slightly different: material that supports a decision, including the decision to do nothing.

The difference shows up in a simple test. Can anyone in the room disagree with the numbers? If the pack cannot be argued with, it is not informing a decision.

What we see in programmes

The standard quality slide has a completion percentage, a pass rate, a defect count by severity and a RAG rating. Each of those has a specific weakness.

The completion percentage hides its denominator. The pass rate rewards the behaviour of getting things to pass. The defect count by severity uses a scale that means something to engineers and almost nothing to a CFO. The RAG rating compresses everything that was interesting into one of three colours, usually by someone who will be asked to explain a red.

None of this is dishonest. It is what happens when reporting evolves to answer “are we on track?” rather than “what should we do?”

A better way to think about it

Five signals, all of which can be produced from what a well-run programme already has.

1. Critical-process coverage. Of the business processes classified as critical, how many are covered by tested scenarios, how many partially, how many not at all. Expressed as a count against a named list, not a percentage. The value of this signal is that the list can be challenged — a process owner can say “that one should be critical and isn’t”, which is exactly the conversation you want in week six rather than week twenty.

2. Execution across four states. Passed, failed, blocked, not started. Blocked is the important one and it is routinely hidden. A blocked scenario is not slow progress; it is a dependency somebody else owns, and surfacing it in a steering pack is often the fastest way to unblock it.

3. Open defects expressed as business impact. Not “fourteen high severity” but “three defects affecting invoice output, one affecting month-end close, workaround tested for two of them”. This is a translation exercise and it takes effort, which is why it is usually skipped. It is also the only version of the defect report that a non-technical executive can act on.

4. Untested and blocked exposure. The parts of scope that will not be covered by the current plan, stated plainly, with the reason. Descoped by decision, blocked by data, blocked by environment, blocked by a third party. This is the signal most likely to be uncomfortable and most likely to be useful.

5. Evidence readiness. Could the programme produce the evidence behind any given result on request, and how quickly? This does not need a number. It needs an honest answer, and the honest answer is often revealing.

The useful question is not “how much have we completed?” but “what exposure remains behind the percentage?”

What good looks like

A quality view that fits on one page, uses named lists rather than only ratios, distinguishes the four execution states, translates defects into process language, and states what is not covered as clearly as what is.

It should also be boring in a specific way: the same shape every fortnight, so that movement between reports carries information. A pack that is redesigned each cycle to present the best available story cannot be tracked.

Where test management is centralised — SAP Cloud ALM being the common case — most of this can come from the system rather than from a slide-building exercise. That matters less for the effort saved and more because a number that comes from the system is harder to gently improve on the way to the meeting.

Questions leadership should ask

Show me the list behind the percentage.

How many scenarios are blocked, and on whom?

Which open defects would affect a business process on day one, and do the workarounds work?

What is in scope that we now know we will not test?

Pick a result at random — how long to show me the evidence?