promptdojo_

Retrieval metrics and vector-DB shape — recall@k, MRR, and why the filter runs before the search — step 1 of 7

ch 22 · context and retrieval1/7

"It feels relevant" is not a metric

You tuned the threshold, deduped, stabilized the sort. Then you typed five queries into your own retriever, nodded at the results, and called it good. That's not evaluation. That's vibes with extra steps — and vibes don't tell you whether last Tuesday's re-chunking made retrieval better or quietly broke it.

The fix costs an afternoon: an eval set. Twenty or thirty real queries, each paired with the chunk ids a human says are the right answer. That's the whole artifact — a list of (query, gold ids) pairs. Build it from the question shapes your users actually ask (the fork lesson's rubric applies here too: mostly direct lookups → your eval set should be mostly direct lookups). Then three numbers fall out:

  • recall@k — of the chunks a human marked relevant, what fraction showed up in your top-k? Missing gold chunks means the model never sees the answer. This is the number that predicts hallucination.
  • precision@k — of the k chunks you sent, what fraction was relevant? Low precision means you're paying tokens to feed the model distractions.
  • MRR (mean reciprocal rank) — how high did the first correct chunk rank, averaged as 1/rank? Rank 1 scores 1.0, rank 3 scores 0.33. It matters because the model attends hardest to the first chunk in the prompt.

Run the editor. One retrieved chunk is gold, one gold chunk is missing, one retrieved chunk is filler — and the two metrics disagree about how bad that is. They usually do. Recall and precision pull in opposite directions as you turn the k knob, which is exactly why you need both.

The other half: what a vector DB row actually is

Strip the marketing and a vector database stores rows of exactly three things:

{"id": "policy/p2", "vec": [0.12, -0.4, ...], "meta": {"tenant": "acme", "year": 2026}}

An id, a vector, and a payload (metadata). That's it. The product you pay for adds fast approximate search over millions of vectors — but the shape is a list of dicts, which is why this lesson can model one honestly in fifteen lines of stdlib Python.

The payload is where the second decision of this lesson lives: filter-then-search or search-then-filter? When a query says "acme's refund policy," you can filter rows to tenant == "acme" first and rank only those — or rank everything and filter the top-k afterward. One of these silently returns nothing when the global top-k happens to be full of other tenants' documents. You'll break and fix that exact bug in a few steps.