lesson 2 of 8 · judgment gate — prompt, rag, or fine-tunelesson 2 of 8 · judgment gate — prompt, rag, or fine-tunelesson 2 of 8 · judgment gate — prompt, rag, or fine-tunelesson 2 of 8 · judgment gate — prompt, rag, or fine-tunelesson 2 of 8 · judgment gate — prompt, rag, or fine-tunelesson 2 of 8 · judgment gate — prompt, rag, or fine-tunelesson 2 of 8 · judgment gate — prompt, rag, or fine-tunelesson 2 of 8 · judgment gate — prompt, rag, or fine-tunelesson 2 of 8 · judgment gate — prompt, rag, or fine-tunelesson 2 of 8 · judgment gate — prompt, rag, or fine-tunelesson 2 of 8 · judgment gate — prompt, rag, or fine-tune
Three failure modes, three levers
Before anyone opens a trainer, name the failure. Most "we should fine-tune" threads are really one of these three:
- Bad brief. The prompt never said the output must be one of
flake | product-bug | needs-human, or it said it in a paragraph the model treats as flavor. Fix: rewrite the brief. Add a schema. Add two worked examples. Measure again. - Missing documents. The label depends on a runbook, an allowlist, or yesterday's incident writeup that is not in the prompt. The model is guessing because the file is missing. Fix: retrieve the file. Fine-tune will freeze last quarter's guess into weights.
- Format or label drift. The brief is tight. The docs are present or irrelevant. The model still wanders — extra prose around the label, swapped synonyms,
needs_humanvsneeds-human, a fourth label it invented. Fix: a reviewed example set and a specialist adapter, after you can show the prompt-only baseline on a held-out set.
The gate is evidence, not vibes. You need a short golden set you already scored under the current prompt. If prompt-only already hits the acceptance tests, you do not fine-tune. If RAG is the only thing that would have supplied a changing fact, you do not fine-tune. If the remaining miss is consistency on a stable job, you may.
Wrong CLEARED (the chapter pass) is fine-tuning a knowledge gap. That run can look "smarter" in a demo and then lie with confidence the first week a policy changes.