promptdojo_

Retrieval metrics and vector-DB shape — recall@k, MRR, and why the filter runs before the search — step 7 of 7

Checkpoint

One last thing before we move on. Same surface as a write step — but the lesson doesn't complete until this passes.

Final drill: the whole lesson in one harness. The rows below are honest vector-DB rows — id, vector, metadata payload — and the eval set is three labeled queries. Build both halves:

  1. search(q_vec, rows, tenant, k) — filter rows to the tenant FIRST, rank the survivors by dot(q_vec, row_vec) descending, and return the top-k ids.
  2. evaluate(eval_set, rows, k) — run search for every case, then return a (mean_recall, mrr) tuple: recall is hits divided by the size of that case's relevant set, averaged over cases; MRR averages 1/rank of the first hit (0 for a case whose top-k has no hit).

One detail worth noticing in the data: for the first query, globex/refunds out-scores every acme row. Filter late and it steals rank 1 for the wrong tenant — filter first and your metrics measure what acme actually sees.

Expected output:

recall@2: 0.89
mrr@2:    0.83

full-screen editor opens — close anytime to keep reading.