The four breakage classes — sort any LLM failure before you touch the prompt — step 9 of 9
One last thing before we move on. Same surface as a write step — but the lesson doesn't complete until this passes.
Final drill. You got three user-reported failures from the support inbox this morning. Each has a trace attached. Classify them so the right team gets the right ticket — retrieval team, prompt team, ML team (hallucination), or backend team (parse).
Write classify_failure(trace) using the same priority order
from the previous step:
- retrieved_chunks non-empty + retrieved_chunks_match_query False
→
"retrieval" - raw_output != output_after_postprocess →
"parse" - retrieved_chunks empty →
"hallucination" - else →
"prompt"
Then print one line per trace in this format:
ticket_<id>: <class>
Expected output:
ticket_8821: retrieval
ticket_8822: prompt
ticket_8823: parse
One last thing before we move on. Same surface as a write step — but the lesson doesn't complete until this passes.
Final drill. You got three user-reported failures from the support inbox this morning. Each has a trace attached. Classify them so the right team gets the right ticket — retrieval team, prompt team, ML team (hallucination), or backend team (parse).
Write classify_failure(trace) using the same priority order
from the previous step:
- retrieved_chunks non-empty + retrieved_chunks_match_query False
→
"retrieval" - raw_output != output_after_postprocess →
"parse" - retrieved_chunks empty →
"hallucination" - else →
"prompt"
Then print one line per trace in this format:
ticket_<id>: <class>
Expected output:
ticket_8821: retrieval
ticket_8822: prompt
ticket_8823: parse
this step needs the editor
on desktop today; in the app (coming soon). save your spot and we'll bring you back here when you're ready.