Evaluator-optimizer — write a draft, let a judge critique it, revise — step 8 of 9
Build the full evaluator-optimizer. Write write_with_judge(task, max_iters) that:
- Maintains
feedback = None(initial draft has no prior feedback). - Loops up to
max_iterstimes. Each iteration:- Calls
gen(task, feedback=feedback)to produce a draft. - Calls
judge(draft)to get a verdict dict. - If
verdict["status"] == "PASS": return a dict{"draft": <draft>, "iters": <iteration count, 1-based>, "passed": True}. - Otherwise update
feedback = verdict["feedback"]and continue.
- Calls
- If
max_itersis exhausted without a PASS, return{"draft": <last draft>, "iters": max_iters, "passed": False}.
The script will run two cases (one that PASSes on iter 2, one that exhausts the cap). Expected output:
passed=True iters=2 draft=Cats are small carnivorous mammals.
passed=False iters=2 draft=v2
⌘↵ runs the editor.read, then continue.
Build the full evaluator-optimizer. Write write_with_judge(task, max_iters) that:
- Maintains
feedback = None(initial draft has no prior feedback). - Loops up to
max_iterstimes. Each iteration:- Calls
gen(task, feedback=feedback)to produce a draft. - Calls
judge(draft)to get a verdict dict. - If
verdict["status"] == "PASS": return a dict{"draft": <draft>, "iters": <iteration count, 1-based>, "passed": True}. - Otherwise update
feedback = verdict["feedback"]and continue.
- Calls
- If
max_itersis exhausted without a PASS, return{"draft": <last draft>, "iters": max_iters, "passed": False}.
The script will run two cases (one that PASSes on iter 2, one that exhausts the cap). Expected output:
passed=True iters=2 draft=Cats are small carnivorous mammals.
passed=False iters=2 draft=v2
this step needs the editor
on desktop today; in the app (coming soon). save your spot and we'll bring you back here when you're ready.