Triage the bad prediction — reproduce with the logged input, then route by class — step 1 of 7

Reproduce. Classify. Route. In that order.

A bad prediction lands in the inbox: "your AI said my invoice total was $1,240 — it's $940." Lesson 01 taught you to open the trace. Lesson 02 gave you the four classes. This lesson is the runbook that connects them: what you actually do, in what order, before anyone touches a prompt.

Step 1: reproduce — with the logged input, not a paraphrase

The report you receive is a paraphrase. Support summarizes what the user said. The user summarizes what they typed. Feed either summary back into the pipeline and you're debugging a different input than the one that failed. LLM pipelines are brutally sensitive to exact wording — the paraphrase often works fine, you declare the bug unreproducible, and it bites the next customer a week later.

The only input that counts is the one in the trace. Pull logged_input, re-run the pipeline on it, and confirm you see the same wrong output. Can't reproduce? Triage is blocked. Fix your logging before you fix anything else, because right now you're chasing a ghost.

Step 2: classify — the four classes, one more angle

Same taxonomy as lesson 02, no new buckets. But notice what the four classes really are — each one names the kind of problem:

1. Retrieval — a data problem. The wrong context went in.
2. Prompt — an expectation problem. You asked for something other than what you meant.
3. Hallucination — a model problem. Good input, fabricated output.
4. Parse — a code problem. Correct output, mangled on the way out.

Data, expectation, model, code. Run classify_failure(trace) from lesson 02 and get one word back.

Notice which category is missing: "the model is dumb." That phrase names no layer, no owner, and no fix. It's what triage sounds like when nobody read the trace — a shrug wearing the costume of a diagnosis. Three of the four classes don't involve the model misbehaving at all.

Step 3: route — every class has an owner

Classification is only useful because each class routes somewhere different: retrieval bugs go to whoever owns the pipeline that feeds the model, prompt bugs go to whoever owns the spec, hallucinations go to grounding and guardrails, parse bugs go to the post-processing code. Route a parse bug to the prompt owner and you get a "more careful" prompt edit that fixes nothing and breaks two adjacent cases.

Run the editor. The whole runbook is a gate, a lookup, and a routing line — small enough to hold in your head on a bad Tuesday.