Run It Again: Reproducibility as an Examination Standard

Replay, not approximation

There are two things an institution can do when asked to reproduce a past decision, and they are not versions of each other. Replay takes the decision as it was made: the same inputs, the same model version, the same instructions and rules, the same configuration — and re-executes it, producing a record that can be compared against the original step by step. Approximation takes the decision as a question: it puts the same case to the current system and reports that the answer came back similar. Approximation is not worthless — it tells you what today’s system would do — but it concerns a different decision than the one under review. The distinction is one every examiner already knows: replay is evidence; approximation is testimony.

The distinction gets blurred because approximation is what most deployed systems can actually do. The evidence was retrieved live and never sealed; the hosted model has been updated under a stable product name; the prompts were edited in place; the sampling settings were never recorded. Each of those is an ordinary engineering choice. Together they make the honest answer to “run it again” an apology.

Four things to pin

Replay is not a heroic capability. It is a bookkeeping discipline over four artifacts, each of which either was captured at decision time or was not:

1. The inputs. The sealed evidence set exactly as the system saw it: the documents, fields, and retrieved context presented at decision time. If retrieval re-runs live during the replay, the inputs are not pinned — the system is deciding a subtly different case.
2. The model version. The specific weights that executed each reasoning step. A product name is not a version; hosted models change under stable names, and “the same model” must mean the same artifact, not the same brand.
3. The instruction and rule-corpus version. The prompts, policy text, and rule corpus that governed the decision, as of its date. These change more often than models do, and a replay under today’s rules tests today’s rules.
4. The decoding configuration. The sampling settings under which the output was generated — deterministic settings, where reproducibility is the goal. Temperature and seed are configuration, and configuration is governable.

Notice what is not on the list: the original output. Replay does not consult the answer it is trying to reproduce. It reconstructs the conditions and lets the record speak.

Why “similar” is not the same

The tempting shortcut is to compare narratives: run the case again, put the two write-ups side by side, and score their similarity. The shortcut fails for a structural reason — decisions do not live in prose, they live in steps. Two fluent narratives can agree in conclusion and tone while differing in the one step that mattered: one cited the enrollment record, the other assumed it; one tested each element of the rule, the other summarized past them. Paraphrase similarity grades the writing. The examination is about the reasoning. A record that exists as structured steps can be compared the only way that means anything: step by step, where “matching” means each step reaches the same determination from the same cited evidence — not that the sentences rhyme.

Nondeterminism is a finding

Engineering culture treats nondeterminism as ambient weather — a property of the tools, noted and shrugged at. Supervision should not. If a decision cannot be re-run under pinned conditions, that is a property of the decision process the institution chose to operate, and it belongs in the findings alongside every other control gap. The point is not that all AI must be deterministic everywhere; it is that the institution must know which parts of its process are nondeterministic, must have chosen deterministic settings where decisions are consequential, and must be able to distinguish divergence caused by sampling noise from divergence caused by an unrecorded change. An unexplained divergence is an unexplained decision. Structured, sealed, step-level records reduce and interrupt the failure modes that produce unexplained divergence — they do not abolish nondeterminism, they make it visible and assignable.

The protocol, operationally

Operationally, a re-run protocol answers three questions. Who can invoke it: parties independent of the build — validation, internal audit, compliance — without engineering assistance being the bottleneck, because a replay that requires the builders is a demonstration, not a control. What artifact it produces: a replay record, itself sealed, comparing original and re-run step by step and ending in one of three verdicts — match, explained divergence, or unexplained divergence. What happens on divergence: explained divergence documents its cause against the change history; unexplained divergence opens an exception with an owner and a disposition, like any other incident. An examination team can ask for all three — and can reasonably ask when the protocol was last actually run.