one sitting, one agent feature, end to end: a tool schema that makes the dangerous call a validation error, a loop that meters its own spend, golden cases wired before the router works, a trace you debug from instead of vibes, and the ratchet that turns tonight's incident into tomorrow's eval. practice at production tempo.

The demo was the easy part

Every developer who's built an agent feature has lived this sequence: the demo runs clean on Tuesday, ships on Wednesday, and on Thursday it reruns the same red job four times, burns real money, opens a p3 issue for a prod outage, and nobody can say why because the only record is a chat transcript. The model didn't get dumber overnight. You shipped a loop with no meter, tools with no contract, and zero test cases that encode what "correct" even means for this feature.

This chapter is a studio, not a lecture series. You ship one agent feature — a CI triage agent that reads failing jobs and decides between rerun, open an issue, or page a human — end to end, in one sitting, in the order working agent engineers actually build: tool schema first, loop with budget caps second, golden evals wired before the router logic is finished, trace on every step, ratchet entry on every failure. The same five artifacts you'd produce shipping this at work, minus the standup.

Viv Trivedy's line is the frame for all of it: "Agent = Model + Harness. If you're not the model, you're the harness." You are not the model. Everything in this chapter is your half of that equation.

Why it matters

The leverage sits on your side of the API call. You saw the receipt in chapter 30: HumanLayer citing the Terminal Bench 2.0 leaderboard, where the same Opus 4.6 ranked #33 inside Claude Code and #5 in a different harness (±4 positions per HumanLayer's stated margin). Identical model, 28-place swing, and everything that moved was scaffolding code — the exact kind you write in this studio.
Agent bugs don't reproduce; agent traces do. You can't git bisect a nondeterministic loop. The developers who debug agents fast are the ones whose loop wrote down every step, cost, and verdict as it ran. Everyone else is rereading a transcript and guessing.
Failures are signals, not flukes. Addy Osmani's rule for agent configs, verbatim from Agent Harness Engineering: "every line in a good AGENTS.md should be traceable back to a specific thing that went wrong." Most teams retry past agent mistakes like they're flaky tests. The ones whose harness improves every quarter treat each mistake like an incident report — it earns a rule, an eval case, and an exit condition.

What AI specifically gets wrong

It calls the widest tool available. A tool named run that accepts anything will eventually run anything. That's not a model alignment problem you can prompt away — it's a schema problem you can fix in ten lines. Narrow names, required fields, enum values, a confirm flag on the calls that cost money or touch prod.
It doesn't know when to stop. A loop retries a red build, the build stays red, the loop retries again. Without a step cap and a cost cap inside the loop definition, the stop condition is your credit card limit. The cap isn't defensive paranoia; it's part of what "loop" means.
It passes the demo and fails the queue. The three cases you tried by hand all worked. The 4th real case — the npm registry 503 that looks nothing like a timeout — routes wrong on day one. Golden cases from real past incidents catch this before a teammate does, but only if the eval runner exists before you declare the feature done.
It regresses silently. You fix the 503 case, and three weeks later a prompt tweak un-fixes it. Nothing notices, because nothing was watching. The eval set that only grows — every incident appends a case, no case is ever deleted to make the run green — is the only regression test an agent feature gets.

What you'll be able to do at the end

Define a tool surface as data — required fields, allowed values, confirm flags — and write the validator that turns a destructive miscall into reject: needs confirm instead of a 2 a.m. page.
Run an agent loop that meters itself: step cap, cost cap, named stop reasons, a trace line per step, so "why did it stop" is a lookup instead of an investigation.
Wire golden-case evals from real past incidents before the routing logic is finished, then code until the run says 4/4 — ship — red-to-green, the same rhythm as TDD, aimed at model behavior instead of your own.
Keep a ratchet: every failure becomes a five-field log entry (date, failure, rule, artifact, remove_when) plus a new golden case, so the gate is stronger after every incident and no rule lives forever without an exit condition.

Every drill runs in plain Python, stdlib only, deterministic. The model's decisions are scripted so the harness logic — your half — is the thing under test. Which is the whole point: when the feature misbehaves in production, the fix is almost never on the model's side of the wire.

agent studio

The demo was the easy part

Why it matters

What AI specifically gets wrong

What you'll be able to do at the end

lessons in this chapter