lesson 5 of 7 · add evals and traces — measure the agent, don't trust itadd evals and traces — measure the agent, don't trust itlesson 5 of 7 · add evals and traces — measure the agent, don't trust itadd evals and traces — measure the agent, don't trust itlesson 5 of 7 · add evals and traces — measure the agent, don't trust itadd evals and traces — measure the agent, don't trust itlesson 5 of 7 · add evals and traces — measure the agent, don't trust itadd evals and traces — measure the agent, don't trust itlesson 5 of 7 · add evals and traces — measure the agent, don't trust itadd evals and traces — measure the agent, don't trust itlesson 5 of 7 · add evals and traces — measure the agent, don't trust itadd evals and traces — measure the agent, don't trust itlesson 5 of 7 · add evals and traces — measure the agent, don't trust itadd evals and traces — measure the agent, don't trust itlesson 5 of 7 · add evals and traces — measure the agent, don't trust itadd evals and traces — measure the agent, don't trust itlesson 5 of 7 · add evals and traces — measure the agent, don't trust itadd evals and traces — measure the agent, don't trust itlesson 5 of 7 · add evals and traces — measure the agent, don't trust itadd evals and traces — measure the agent, don't trust itlesson 5 of 7 · add evals and traces — measure the agent, don't trust itadd evals and traces — measure the agent, don't trust it
Cursor wrote this eval: it calls the agent, gets the answer, then asserts the answer equals what the agent returned. The eval ALWAYS passes — it's checking the agent against itself. Tautological.
Fix the eval to assert against an INDEPENDENT ground truth string ("Paris") that came from a fact-check, not from the agent.
Expected output:
pass: True
The break is on line 9 — but read the whole snippet first.