promptdojo_

The four breakage classes — sort any LLM failure before you touch the prompt — step 6 of 9

The classify_failure(trace) below is the lazy default: it returns "hallucination" for every failure. That's the worst habit on a debugging team — "the model hallucinated" is what you say when you don't want to read the trace.

Fix the function to check the trace fields in priority order:

  1. If retrieved_chunks is non-empty AND retrieved_chunks_match_query is False, return "retrieval".
  2. Else if raw_output != output_after_postprocess, return "parse".
  3. Else if retrieved_chunks is empty, return "hallucination".
  4. Else return "prompt".

The trace in the editor is a class-1 retrieval bug (retrieval returned the wrong account's ticket). The fixed function should return "retrieval".

Expected output:

class: retrieval
The break is on line 10 — but read the whole snippet first.

full-screen editor opens — close anytime to keep reading.