Can AI Check Its Own Work? What the Self-Correction Research Shows
This week’s paper: Huang et al., “Large Language Models Cannot Self-Correct Reasoning Yet” (Google DeepMind, ICLR 2024). The question it isolates: when a model critiques and revises its own reasoning with no external feedback — what the authors call intrinsic self-correction — does the reasoning get better? The answer: no. Accuracy did not improve, and often degraded, with models talking themselves out of correct answers. Earlier optimistic results, the authors show, had quietly leaned on external signals — including oracle knowledge of the correct answer. The supervisory translation writes itself: self-review is not effective challenge, and “the model double-checks itself” is not a control.
The paper
By 2023, a comforting story had taken hold: even if a model’s first answer is flawed, you can prompt it to review its own work — “check your reasoning; fix any mistakes” — and the second draft will be better. Several papers had reported exactly that. Huang et al. asked the sharper question underneath: better because of what? They separated the setting where the model receives genuine external feedback from the setting where it receives none — where the only critic available is the model itself, working from the same knowledge that produced the first answer. They named the second setting intrinsic self-correction and tested it on reasoning benchmarks.
Intrinsic self-correction did not help. Accuracy failed to improve and in many cases went down: invited to reconsider, models revised correct answers into incorrect ones. And the earlier optimism dissolved under inspection — the encouraging results had depended on external signals, in some cases the strongest one imaginable: oracle knowledge of the correct answer, used to decide when to stop revising. Tell a system when it is wrong and it will look impressively capable of fixing itself. Withhold that signal — which is the deployed condition, since a system that already knew the answer would not need to reason — and the capability evaporates.
Why the failure is structural, not incidental
The finding should not surprise anyone who has thought about where a model’s judgment comes from. The critic and the author are the same weights. Whatever priors, gaps, and biases produced the first answer are fully present for the review of it — the reviewer inherits the author’s blind spots because the reviewer is the author. A flaw the model could recognize, it mostly would not have committed; the flaws it does commit are precisely the ones its own review is least equipped to see. We made the same argument from the architecture side in Artificial Intelligence Is Not Artificial Reasoning: a justification, reread by its own author, confirms itself. This paper is the empirical version of that scene.
Why regulators will find this familiar
Supervisory doctrine arrived at this conclusion about humans decades ago. The second line of defense exists because first-line review of first-line work is not review; effective challenge, as the model-risk guidance frames it, requires challengers with the independence and standing to actually change the outcome — a function we examined in Effective Challenge as a Function. No examiner would accept “the analyst rereads her own memo” as a control, however diligent the analyst. The paper’s contribution is evidence that the same independence requirement holds for machine reasoning: a model rereading its own output is the analyst rereading her own memo, at scale and with better prose.
The defensible design follows directly. Verification must come from outside the reasoning being verified: separate verifier steps with their own criteria, gates that can actually stop a conclusion from proceeding, challengers that receive the evidence and the claim rather than the author’s framing of both. In an IRG, that is not an aspiration but the topology — the critique node is a different component from the node it critiques, and a gate’s verdict is not the author’s to overrule. None of this eliminates error; it reduces the odds that an error reviews itself and passes.
Limits, honestly stated
Three cautions. First, the title says yet, and the authors mean it — the paper measures models of its moment, not a ceiling on the technique; later systems trained specifically to revise may shift the picture, and the finding should be read as “do not assume,” not “never possible.” Second, the negative result is about intrinsic correction on reasoning tasks; the same paper is not evidence against feedback-driven revision, which is a different and better-supported design — indeed it is the design we advocate. Third, external verification is not a solved problem either: a separate verifier is still a component with its own error rate, to be validated like any other. Independence improves the odds of catching a flaw; it does not certify the outcome. Our claim stays where it always is — these mechanisms reduce and interrupt failure modes; they do not eliminate them.
The takeaway for a compliance reader
When a vendor’s architecture slide shows a “self-verification” loop, this paper hands you the question to ask: what external signal does the loop receive — and would it still work without one? If the check is the same model rereading its own output, the research says the check adds little and may subtract. Ask instead where the independent verification lives: which step, which criteria, what happens when it fails. Independence of review is not a nicety imported from human governance. On current evidence, it is a functional requirement of machine reasoning too.