What to Ask the Vendor: An Examiner’s Question Bank
A dozen questions for anyone evaluating an AI decision system — examiners reviewing an institution’s vendors, or institutions who would rather ask them before the examiner does. The questions share one design principle: none of them can be answered by a demo. Demos select for fluency, and fluency is the one property this technology never lacks. Each question probes a structural property instead — inspectability, reproducibility, boundedness, challengeability — and comes with what a good answer looks like. A vendor who answers all twelve on the spot is describing architecture. A vendor who needs to “follow up on that” is describing a demo.
Why questions beat demos
An evaluation meeting for an AI decision system has an asymmetry problem: the vendor controls the demonstration, and generative systems demonstrate beautifully. The failure modes that matter — the unfaithful rationale, the silently changed model, the decision that cannot be reconstructed a year later — are precisely the ones a demonstration cannot exhibit. So the useful evaluation ignores what the system says and interrogates what the system keeps: records, versions, bounds, and the machinery that produces them. That is also, not coincidentally, what an examination will interrogate — a theme we developed in The Examiner Is the User.
The question bank
- “Show me one complete decision record — now, for a decision I pick.” Good: a self-contained record with inputs, steps, evidence citations, and outcome, produced in minutes. Bad: a polished sample prepared in advance, or a promise to export something later.
- “Is the reasoning stepwise or narrative?” Good: discrete steps, each with its own inputs, output, and checks — individually inspectable. Bad: one long generated explanation, however articulate; a narrative is a single step wearing a costume.
- “What blocks?” When a verification fails mid-decision, does anything actually stop — and where is a blocked decision visible? Good: named checkpoints with recorded outcomes and a queue of holds. Bad: “the model is instructed to be careful.”
- “Re-run this decision from last year.” Good: sealed inputs plus pinned model, instruction, and rule versions reproduce the same record, and divergence is investigable. Bad: “you’d get a very similar answer today” — similarity is not reproducibility.
- “Which versions governed this decision?” Good: every decision is stamped with the model, prompt, and rule-corpus versions that produced it, so populations affected by a change can be enumerated. Bad: version history exists only as release notes.
- “What does it do when it doesn’t know?” Good: abstention is a designed outcome — novel cases, infeasible constraints, and insufficient evidence route to humans as such. Bad: the system always produces an answer; confidence is a tone of voice.
- “What was evaluated — answers or steps?” Good: step-level evaluation evidence with the protocol available for your validators to challenge. Bad: a single end-to-end accuracy figure on an unnamed test set.
- “What does the human reviewer actually see and sign?” Good: the reviewer attests over the full record, their questions and overrides become part of it, and attention is verifiable. Bad: an approve button next to a summary.
- “Who can change the system’s behavior, and how do we find out?” Good: contractual change notification with dates, so the institution can map every behavioral change to the decisions it touched. Bad: continuous silent improvement.
- “What is the stated-limitations register?” Good: a maintained document of what the system is known not to do well, reflected in the decision records themselves. Bad: limitations live in the sales engineer’s head.
- “If we disagree with a decision, what does the challenge operate on?” Good: a specific step in a specific record can be contested, re-run, and corrected, and the correction propagates to affected decisions. Bad: challenges are a support ticket.
- “What leaves with us at exit?” Good: complete decision records in a documented, usable format — the institution’s obligations outlive the contract. Bad: records readable only inside the vendor’s platform.
Reading the answers
No single question is disqualifying, and honest vendors will have uneven answers — these properties are engineering investments, and some are on roadmaps. What the bank measures is the shape of the product. Systems built as governed reasoning answer questions one through twelve from architecture: records, versions, gates, and bounds exist because the decision process is made of them. Systems built as model wrappers answer from aspiration, because the properties would have to be bolted onto a process that is, at its core, one long generation. The distinction is the one we drew in Active vs. Passive Governance — and no demonstration, however impressive, can substitute for it.
For institutions, one suggestion: run the bank internally before any examination does. The questions your own vendors cannot answer are the questions you cannot answer about your own decisions — and they are answerable, with notice, on your schedule rather than an examiner’s.