Why the Numbers Should Never Come From the Model
A language model does not compute numbers; it predicts what numbers would look like in context. For prose, that distinction is pedantic. For back-tests, fee calculations, deadline arithmetic, and fairness statistics, it is disqualifying — because the failure mode is a plausible number, and plausible wrong numbers pass review. The engineering answer is the deterministic tool-node: computations executed in code inside the reasoning graph, consuming zero LLM tokens, byte-reproducible from the same inputs. The design property that makes it work is authority: the model may cite tool outputs, but may not recompute them, approximate them, or override them. Getting that boundary right is most of what separates a decision system from a very confident chatbot.
Prediction is not computation
When an LLM outputs “the variance is 4.7%,” nothing computed a variance. The model produced the tokens that were most probable given the context — a process that frequently lands on the right value, especially for arithmetic that resembles its training data, and drifts unpredictably as the numbers get longer, the operations compose, or the inputs get unusual. The accuracy curve is the treacherous part: a system that is right 98% of the time trains its reviewers to stop checking, and the 2% arrives with identical confidence and formatting. There is no misspelling, no hesitation, no tell. A hallucinated citation at least can be looked up. A hallucinated intermediate value inside a long analysis is invisible unless someone re-derives the whole chain — which is exactly the labor the system was deployed to eliminate.
Consequential domains are where this bites hardest, because they are dense with computations that must be exact, not approximately right: the annualized loss rate in a model back-test, the 45th calendar day on a Reg E claim with a weekend landing, the Adverse Impact Ratio that determines whether a fair-lending review escalates, the interest recalculation behind a remediation payment. Every one of these numbers ends up in an artifact a regulator, auditor, or opposing counsel can re-run. “Directionally correct” is not a category they recognize.
The deterministic tool-node
The fix is old and the discipline is new. Old: don’t use a language model as a calculator; call a calculator. Every agent framework has some version of tool use. New: making the tool call a governed, first-class node in the reasoning graph rather than an optional accessory the model may or may not invoke. A deterministic tool-node in an IRG graph has four properties. It computes in code — a reviewed, versioned function, not a sampled generation. It consumes zero LLM tokens — there is no stochastic step anywhere in the computation path. It is byte-reproducible — the same inputs produce the same output bytes, forever, which means an examiner can re-run the block and get your numbers. And its execution is structural — the graph topology routes the computation through the node; the model cannot skip it, because the downstream nodes’ inputs are typed to the tool’s output.
In our model-risk domains, this is how math verification and back-testing work — and it is why the fair-lending statistics in an outcomes review are computed rather than asserted. The pattern generalizes to anything with a closed-form answer: date arithmetic against regulatory clocks, threshold tests, fee and interest math, portfolio aggregations, reconciliation checks. A useful heuristic for where tool-nodes belong: any number that would appear in the artifact you hand an examiner should be produced by code.
Authority is the property that matters
Zero tokens and reproducibility are easy to admire. The property that does the governance work is less obvious: the tool’s output is authoritative. The model may cite it. The model may reason about it — interpret the variance, weigh the flag, draft the finding. The model may not recompute it, restate it approximately, or contradict it. If the tool says the AIR is 0.76 and the model’s draft says “approximately 0.8,” that draft fails a gate, mechanically.
Why so strict? Because the failure mode of tool-using LLMs is not refusing to call tools — it is laundering tool outputs through paraphrase. The model calls the calculator, gets 0.76, and writes “roughly four-fifths, suggesting the disparity is marginal” — and in one fluent clause, an exact escalation trigger became a judgment call that never fires. Nothing in an ordinary agent framework catches this, because the tool was called; the log shows compliance. What went wrong is semantic, and only a binding between the tool’s output and the claims downstream can catch it. That binding is what confidence-reasoning discipline looks like at the level of arithmetic: stated values must trace to computed values, exactly, or the trace fails. The division of labor becomes clean and auditable — code does what code does perfectly (compute), the model does what models do well (interpret, contextualize, argue), and the graph enforces the boundary so neither impersonates the other.
The objection, answered
“Frontier models barely make arithmetic errors anymore.” Mostly true, and beside the point. A 0.5% error rate on a million annual determinations is five thousand wrong numbers with no way to know which five thousand — so verifying any number requires re-deriving every number, and the audit cost you thought you eliminated returns in full. Determinism isn’t about the average error rate; it is about making verification cheap. A byte-reproducible block can be spot-checked by re-execution in milliseconds. A sampled number can only be checked by an independent expert doing the math again. One of these scales to examination. The other is why validation teams still re-derive AI outputs by hand — and why AI stays at the perimeter of the workflows where the numbers matter most.
Let the model reason. Let the code compute. Build the graph so neither can do the other’s job — because the number an examiner re-runs must be one your system computed, not one it predicted.