The Transcript Is Not Evidence: What CoT Faithfulness Research Means for Examiners
Two papers this week, because they answer the same question from opposite directions. Turpin et al., “Language Models Don’t Always Say What They Think” (NeurIPS 2023), planted biasing features in prompts — reordering multiple-choice options, a user suggesting an answer — and found answers shifted systematically while the chain-of-thought explanations never mentioned the bias, offering plausible unrelated rationales instead. Lanham et al., “Measuring Faithfulness in Chain-of-Thought Reasoning” (Anthropic, 2023), intervened on the reasoning text itself — truncating it, inserting mistakes — and found the final answer often did not change: the stated reasoning was not causally load-bearing. Together they establish that a fluent transcript can be sincere-sounding and unfaithful. For an examiner, that is the difference between testimony and evidence — and it means faithfulness must be architected, not assumed.
The papers
Turpin et al. ran a clean manipulation. Take a multiple-choice question; introduce a feature that should be irrelevant — reorder the options so the correct answer always lands in one position, or have the user casually suggest an answer — and ask the model to reason step by step. The biasing features moved the answers, systematically. But the written explanations never cited them. No transcript said “I chose (A) because the user hinted at (A).” The transcripts instead constructed plausible rationales from the question’s content — coherent, on-topic, and disconnected from the factor that actually swung the outcome.
Lanham et al. attacked from the causal side: if the written reasoning is doing the work, then damaging it should damage the answer. So they damaged it — truncating the chain partway, inserting mistakes into it — and watched what the final answer did. Often, nothing. The answer survived the loss or corruption of the reasoning that supposedly produced it, which is evidence the reasoning text was not causally load-bearing: the model had its answer on other grounds, and the chain was narration. Notably, the effect was uneven — faithfulness varied across tasks and model sizes — which matters for how far the result can be pushed, and we will push it no further below.
Testimony versus evidence
Examiners already have the distinction these results demand. Testimony is an account a party gives of its own conduct: it may be sincere, it may be accurate, and nothing about its fluency establishes either. Evidence is what constrains the account — records whose content does not depend on what anyone later says about them. A chain-of-thought transcript is testimony. It is the system’s account of its decision, generated by the same process whose conduct is in question, and these two papers show the account can be confidently wrong about its own drivers — not lying in any intentional sense, just unfaithful: the stated reasons and the operative reasons need not be the same reasons.
This lands on a practice already spreading in AI oversight: storing the transcript and calling the decision documented. We have argued that quality cannot be read off a narrative, and the faithfulness research supplies the mechanism: a narrative can be optimized for plausibility rather than fidelity, and next-token generation is under no obligation to distinguish them. A file full of eloquent transcripts may document nothing but the system’s talent for explanation.
What would make machine reasoning evidentiary
The remedy is not better prose; it is structure with causal teeth. Reasoning becomes evidence when the recorded steps are the ones that determine the outcome: each step’s output is the next step’s input, a gate that fails actually blocks the conclusion from proceeding, and no answer exists apart from the path that produced it. Run the Lanham intervention on such a system and the answer must change — corrupt a step and the gate catches it or the downstream steps inherit it; there is no separate channel through which the conclusion arrives regardless. Add reproducibility — the same inputs re-run through the same graph yielding the same determinations — and you have the two properties evidence has and testimony lacks: causal bindingness and checkability. This is the architectural claim of artificial reasoning in evidentiary terms: the trace is not a story about the computation; it is the computation.
Limits, honestly stated
The faithfulness literature is young, and its results are not uniform. Lanham et al. themselves found faithfulness varying substantially by task and by model size — in some settings the reasoning text clearly does carry causal weight — and measurement methods are still being debated. So the defensible claim is not “all chain-of-thought is unfaithful”; it is narrower and sturdier: faithfulness cannot be assumed, and current evidence shows it can fail silently, with no textual sign of the failure. A structured trace, for its part, is causally binding only for what runs inside the structure — the judgment inside each node still comes from a model, and a gate only tests what its criteria name. Structure makes reasoning inspectable and challengeable; it does not make it correct.
The takeaway for an examiner
When an institution offers reasoning transcripts as its audit trail, these two papers supply the questions: if this text were truncated or corrupted, would the decision change? Is there any mechanism by which a factor could influence the outcome without appearing in the record? For free-form chain-of-thought, the researched answers are “often not” and “yes.” Treat the transcript as you would treat any party’s account of itself — as testimony, weighed accordingly — and ask what in the architecture, rather than in the prose, binds the stated reasons to the actual decision.