The Compliance Officer’s LLM Primer
This is the opening lesson of a curriculum: the mental model a compliance professional needs for large language models, with zero math. Picture an assessor. Their training is frozen into professional habits — nothing in it can be looked up, quoted, or deleted. The pile on their desk is the only thing they read at answer time. They answer by writing the most plausible next word — which is why fluency is not knowledge, and fabrication is a structural property of the mechanism rather than a defect to be patched. Fine-tuning adjusts the habits; retrieval restocks the pile. What follows: model outputs are drafts from a fast, well-read, unaccountable colleague — usable only inside an architecture that verifies them before anything depends on them.
One assessor, two forces
You do not need linear algebra to govern a language model. You need one accurate mental model; the most durable we know is an assessor at a desk. Everything the model does at answer time is the product of two things: the habits the assessor arrived with, and the pile of work in front of them. We have written about these as two kinds of momentum; this lesson is the plain-language version. Hold onto the assessor — every term below is one of their properties.
The weights: habits, not a filing cabinet
Training distills an enormous corpus into billions of numeric parameters — the weights — and the critical intuition is what they are not: a database. No sentence from training is stored anywhere retrievable; what remains is residue — the statistical shape of everything read, the way a decade of case files leaves an assessor with instincts rather than transcripts. Ask them where a habit came from and they cannot cite one; the source survives only as a tendency. This is parametric momentum: the pull of accumulated professional habit, competent and fast, formed entirely before your case arrived.
Two governance consequences fall out. Nothing specific can be looked up in the weights — a model “knowing” your policy manual means only that text like it shaped some habits, not that the manual is on file. And nothing specific can be deleted from them — there is no record to redact, which is why data-deletion questions about trained models are genuinely hard.
The context window: the pile on the desk
At answer time, the model reads one thing: the context window — the prompt, the documents supplied with it, the conversation so far, and its own words as they accumulate. This is the pile on the assessor’s desk — the only thing actually read when your question is answered. Whatever is not in the pile is answered from habit. The pile has a fixed size, it empties between sessions — the assessor has no memory of yesterday — and it exerts its own pull: what is already on the desk, including what the assessor has written today, shapes what gets written next. That pull is contextual momentum — why an early framing or mistake tends to carry forward rather than get revisited.
Generation: one plausible word at a time
The answer gets written one token at a time, each chosen because it is the most plausible continuation of everything so far — given the habits and the pile. That is the entire mechanism. There is no separate step where claims are checked against a store of facts — there is no store of facts, only what text like this usually looks like. Fluency is not evidence of knowledge; fluency is the thing the mechanism directly optimizes. And fabrication is not a defect awaiting a patch: when the habits contain no answer, the mechanism still produces the most plausible continuation, and the most plausible continuation of a question is a confident answer. A citation-shaped string where a citation would go. We have looked at why this worsens in specialized domains; the primer-level point is simpler: the model that fabricates is working as designed. The mitigation must live outside the mechanism.
Fine-tuning and retrieval, demystified
Two terms arrive in every vendor deck. Fine-tuning means additional training on a narrower corpus — further adjustment of the habits. It usefully changes tone, format, and tendency. What it does not reliably do is install facts: the new material becomes residue like all the rest — same blur, same inability to cite, same fabrication when the habit runs out. A fine-tuned model is the same assessor after a training seminar — better instincts, still no filing cabinet.
Retrieval — RAG, in the jargon — takes the opposite route: it puts documents in the pile. A search step fetches passages judged relevant and places them in the context window, where the model actually reads them. This is the stronger pattern for factual grounding, and its weakness is where you would expect: the answer is only as good as what got retrieved. Fetch the wrong passage, a stale version, or nothing, and the assessor answers from habit — fluently, without flagging the substitution.
What this implies for reliance
Put the pieces together and the reliance posture writes itself. A language model is a fast, well-read colleague who works from habit and from whatever is on the desk, cannot cite the source of their instincts, fills gaps with plausible invention, and signs nothing. From such a colleague you accept drafts — and institutions increasingly treat model outputs that way: usable only inside an architecture that verifies each claim against the record before anything depends on it, and that stops rather than guesses when verification fails. What that architecture looks like — steps, gates, challenge, abstention — is where the rest of our writing picks up.
This was lesson one: the mechanism. The rest of the curriculum builds on it — what counts as a model for inventory purposes, what diligence to demand of vendors, what a complete decision record contains. Subscribe below and the next lesson comes to you.