The four breakage classes — sort any LLM failure before you touch the prompt — step 8 of 9
Write classify_failure(trace) that returns one of
"retrieval", "prompt", "hallucination", or "parse".
Apply the priority order from the chapter:
- If
trace["retrieved_chunks"]is non-empty ANDtrace["retrieved_chunks_match_query"]isFalse→"retrieval". - Else if
trace["raw_output"] != trace["output_after_postprocess"]→"parse". - Else if
trace["retrieved_chunks"]is empty (the list[]) →"hallucination". - Else →
"prompt".
Three traces run for you, one per branch. Expected output:
retrieval
parse
hallucination
⌘↵ runs the editor.read, then continue.
Write classify_failure(trace) that returns one of
"retrieval", "prompt", "hallucination", or "parse".
Apply the priority order from the chapter:
- If
trace["retrieved_chunks"]is non-empty ANDtrace["retrieved_chunks_match_query"]isFalse→"retrieval". - Else if
trace["raw_output"] != trace["output_after_postprocess"]→"parse". - Else if
trace["retrieved_chunks"]is empty (the list[]) →"hallucination". - Else →
"prompt".
Three traces run for you, one per branch. Expected output:
retrieval
parse
hallucination
this step needs the editor
on desktop today; in the app (coming soon). save your spot and we'll bring you back here when you're ready.