The Compliance Bottleneck Nobody Measures

Software teams measure how fast they ship. Almost nobody measures how fast they can tell what a change does to certification. This is a proposal for the metric, and the discipline, that closes the gap.

Executive summary

In software engineering, delivery performance is measured carefully. Teams track how long a change takes to reach production, how often they deploy, how often a change fails, and how quickly they recover. These measures changed how software organisations are run, because what gets measured gets managed.

In regulated hardware, the equivalent question is usually unmeasured and often unasked: how long does it take, after an engineering change, to know its certification impact, and how reliable is that answer. This article names that quantity Certification Velocity, distinguishes it from engineering velocity, and argues that the gap between the two is where regulated programmes quietly lose time and money.

It then borrows a second idea. Just as unmanaged technical debt accumulates invisibly until it is paid at a bad time, unevaluated changes accumulate as Certification Debt, the growing distance between the configuration that was tested and the configuration that ships. The article proposes a small set of measures for both, explains why naming and measuring the thing changes behaviour, and is explicit that final compliance judgement stays with qualified people and accredited bodies.

An asymmetry worth noticing

Software engineering spent the last fifteen years learning to measure delivery. The research programme behind the State of DevOps reports, summarised in the book Accelerate by Nicole Forsgren, Jez Humble and Gene Kim, identified a small set of measures of software delivery performance and showed they correlated with organisational outcomes. The four that became widely used are deployment frequency, lead time for changes, change failure rate, and time to restore service, now maintained as the DORA metrics.

The detail matters less than the cultural effect. Once these measures existed, teams could see their own delivery performance, compare it, and improve it deliberately. Delivery stopped being a matter of folklore and became a managed quantity.

Now ask the equivalent question in a regulated hardware organisation. After an engineering change, how long does it take to know its certification impact, and how confident can you be in the answer. In most organisations there is no measure of this at all. There is no number on a dashboard, no trend line, no target. The work happens, the question gets answered eventually, and the time and uncertainty it consumed are absorbed without ever being named.

This is the asymmetry. The activity that often sits on the critical path to market, understanding the certification impact of change, is the one activity that is not measured. What is not measured is not managed, and what is not managed tends to surprise you.

Defining the term

Let us name it precisely, because a vague metric is worse than none.

Certification Velocity is the speed and reliability with which an organisation can determine the certification impact of an engineering change, and produce the evidence and decision that support it.

The definition has two parts on purpose.

The first is speed. How long does it take, from the moment a change is proposed, to reach a defensible answer about what that change does to certification: which rules apply, which are affected, what evidence is required, and what the decision is.

The second is reliability. How trustworthy and reproducible is that answer. Two organisations might both produce an answer in a day, but one produces it as a recorded determination anchored to specific clauses and evidence, and the other produces it as an experienced engineer's recollection that cannot be reconstructed or checked. Those are not the same velocity, because the second one is borrowing certainty it has not actually established.

Certification Velocity is therefore not just "how fast" but "how fast, and how soundly." A fast answer that cannot be defended is not high velocity. It is debt, which is the next idea.

Engineering velocity versus certification velocity

Engineering velocity, the rate at which a team can design, change and iterate a product, has risen sharply. Modular design, simulation, rapid prototyping and better tooling have all compressed the time to make a change.

Certification velocity has not kept pace. The determination of what a change means for compliance is still, in most organisations, a manual act of reading standards and prior evidence and applying expert judgement, with no reusable record that makes the next determination cheaper.

When one capability speeds up and a dependent one does not, the bottleneck moves to the slower one. This is a general property of systems, and it applies here directly. As teams get better at changing the design, the constraint on the programme shifts towards the activity that tells them what the change costs in compliance terms.

The visible result is a familiar pattern. The design is iterated quickly and confidently, right up until the point where the certification consequences have to be reckoned with, usually near a test campaign, at which point the programme slows abruptly and sometimes reverses. The slowdown is read as a certification problem. It is more accurately a velocity mismatch: the design moved faster than the understanding of the design's compliance impact could follow.

As teams get better at changing the design, the constraint shifts to the activity that tells them what the change costs.

The unit of work: the certification impact question

To measure certification velocity, you need a unit of work to measure it against. That unit is what we can call the certification impact question.

Stated fully, the question is: given this change, in these markets, for this device class, which rules apply, which of them are affected by the change, what evidence is needed to support the affected ones, and what is the resulting decision.

This is a precise, answerable question. It has a beginning, a change proposed, and an end, a defensible decision recorded against specific clauses and evidence. The time between those two points, and the soundness of the answer at the end, is exactly what Certification Velocity measures.

Framing the work as a repeated, well-defined question is the move that makes measurement possible. Most organisations do not frame it this way. The work is treated as an open-ended investigation that happens when needed, which is precisely why its duration and reliability are never captured. Naming the unit is the precondition for measuring the rate.

Certification Debt

Here is the second borrowed idea, and the more important one.

The "technical debt" metaphor was introduced by Ward Cunningham, who used the idea of borrowing to describe shipping code that is not quite right in order to learn faster, with the understanding that the borrowing has to be repaid through rework or it accrues interest. Martin Fowler later set out a useful distinction in his Technical Debt Quadrant, separating debt that is deliberate from debt that is inadvertent, and debt taken on prudently from debt taken on recklessly. The point of the metaphor was never that debt is bad. It was that debt is a real liability that should be taken on deliberately and managed, not accumulated by accident.

The same structure applies to certification, and it deserves its own name: Certification Debt.

Certification Debt is the accumulated distance between the configuration that was evaluated against the standards and the configuration that actually ships. Every change whose certification impact is not properly evaluated, or is evaluated informally without a recorded, evidence-linked rationale, is a borrowing against future certainty. The "balance" on the loan is the drift between the tested configuration and the real one. The "interest" is paid at the test lab, or worse in the field, usually late and with a schedule slip attached.

Fowler's quadrant maps onto this cleanly, and the mapping is useful.

Deliberate and prudent. A team knowingly ships a change without a full certification re-evaluation, having judged the risk acceptable, and records that decision and its reasoning. This is sometimes the right call under time pressure. The debt is visible and chosen.

Deliberate and reckless. A team knows a change probably has certification impact and ships it anyway without evaluation or record, hoping it will be fine. The debt is chosen but unmanaged.

Inadvertent and prudent. A team did its best to evaluate a change with the knowledge it had, and later learns the standard had an implication it did not catch. The debt was not chosen, but the process was sound. This is the debt that better infrastructure reduces.

Inadvertent and reckless. A team did not consider that a change might have certification impact at all, because the question was never asked. This is the most common and most dangerous quadrant in regulated hardware, and it is exactly the category the component-change article describes: the "pin-compatible" swap that nobody thought to evaluate.

The value of the metaphor is the same as it was for software. Once you can name Certification Debt, you can decide to take it on deliberately, keep it visible, and pay it down on purpose, rather than discovering an unknown balance at the test lab.

How to measure Certification Velocity

A metric that cannot be operationalised is just rhetoric. Here are practical measures an organisation could adopt. They are proposed measures, and the right targets for any of them depend heavily on sector and product. No benchmark values are asserted here.

Certification impact lead time. The elapsed time from a change being proposed to a recorded certification-impact decision anchored to specific clauses and evidence. This is the direct analogue of lead time for changes in software delivery. It is the headline measure of speed.

Evaluated-change ratio. The proportion of engineering changes that have a recorded, evidence-linked certification-impact decision, as opposed to changes that shipped with no such record. This measures how much of your change flow is generating Certification Debt.

Configuration drift. The number of changes applied to a shipping configuration since the last configuration that was actually evaluated against the standards. This is a direct read of the outstanding Certification Debt balance. A large and growing drift is a warning regardless of how confident the team feels.

Re-evaluation rework rate. The proportion of certification determinations that have to be reopened or redone, for example because a change was missed or an interpretation was wrong. This is the analogue of change failure rate, and it measures the reliability half of Certification Velocity.

Determination reproducibility. Whether a given certification-impact decision can be reconstructed by someone other than its author, from the recorded clauses and evidence, without relying on the original person's memory. This is harder to express as a single number, but it is the difference between sound velocity and borrowed certainty.

None of these requires new science. They require treating the certification impact question as a defined, repeated unit of work, and recording its inputs, its duration and its outcome. That recording is the instrument.

Why measuring it changes behaviour

The argument for measuring Certification Velocity is the same argument that worked for technical debt and for software delivery performance. Naming and measuring a quantity moves it from folklore to management.

Three things change once the measures exist.

The work becomes visible. A determination that took three weeks and could not be reproduced is no longer invisible. It is a data point, and a slow, unreliable one. Patterns that were absorbed silently become legible.

The debt becomes a decision. Configuration drift on a dashboard turns "we have probably accumulated some risk" into a number that someone owns. Taking on Certification Debt becomes a deliberate, recorded choice, which is exactly what Fowler argued debt should always be.

The bottleneck becomes improvable. You cannot deliberately improve certification velocity if you have never measured it. Once it is measured, the slow steps are identifiable, and the case for investing in the infrastructure that speeds them up can be made on evidence rather than on anecdote.

The deeper point is that regulated hardware organisations already manage what they measure, and they measure delivery, cost and schedule. Certification velocity is currently outside that frame, which is why it behaves like an uncontrolled variable. Bringing it inside the frame is the entire proposal.

How Crado approaches this

Crado builds a deterministic chain from an engineering change through to applicable clauses, required evidence and a verifiable decision. Because that chain is recorded and anchored to specific clauses and evidence, the certification impact question becomes a defined unit of work with an output that can be reproduced and audited.

AI is used only to read and structure documents such as standards and test reports. The decision itself comes from deterministic verification against recorded rules, which is what makes the answer reliable as well as fast, the two halves of Certification Velocity.

Crado does not certify products, does not guarantee compliance, and does not replace engineers, accredited laboratories or certification bodies. It is intended to raise certification velocity and keep Certification Debt visible, so that the trade-offs are made on purpose.

Conclusion

Software engineering learned that the activity on the critical path should be measured, and that an accumulating liability should be named and managed rather than absorbed by accident. Regulated hardware has not yet applied either lesson to certification.

Certification Velocity is the missing instrument: how fast, and how soundly, you can tell what a change does to compliance. Certification Debt is the liability it lets you see. Neither is exotic. Both are what you get when you treat the certification impact question as a defined unit of work and write the answer down. You cannot manage what you cannot see, and at present, in most organisations, this is invisible.

1. DORA (DevOps Research and Assessment) — DORA
2. Martin Fowler, Technical Debt — Martin Fowler
3. Martin Fowler, Technical Debt Quadrant — Martin Fowler
4. Ward Cunningham explains the debt metaphor — C2 Wiki