2d:Tbc1,# One loop, many tools @@pd-path:path.Ref:agent-loops@@ lesson 1 gave you the agent loop with *one* tool. That's the toy version. Real agents — Cursor, Claude Code, your own — have ten, twenty, fifty tools registered. The model picks one each turn, sometimes several in parallel, and your loop dispatches. Good news: the loop doesn't change. Same `while`, same `stop_reason` branches, same `tool_result` reply. The only difference is *which* tool runs. ## What "multi-step" actually means Two distinct patterns hide under this name, and people mix them up: 1. **Multi-tool dispatch**. One turn, one tool — but the model chooses from many. `search` this turn, `read_page` next turn, `summarize` after that. The model orchestrates the chain; the loop just runs whichever tool got picked. 2. **Parallel tool calls**. One turn, *several* tool_use blocks at once. "Read these three files." Your loop runs all three (often concurrently) and replies with three `tool_result` blocks in one user turn. Claude does this by default; Anthropic's [tool-use docs](https://docs.anthropic.com/en/docs/build-with-claude/tool-use) document it (and the `disable_parallel_tool_use` flag to turn it off). Both are everyday production patterns. This lesson teaches dispatch first, then parallel calls. ## Why a registry, not a chain of `if` The naive version of this is `if name == "search": ...` repeated ten times. It works for two tools and rots fast. Every new tool means editing the dispatch function, and you'll forget one. The registry pattern — a `dict` mapping name → function — is what every framework uses internally. LangChain calls it a `Toolkit`. LangGraph calls it `tools`. The Vercel AI SDK calls it `tools`. They all flatten to the same shape: ```python TOOLS = { "search": search_fn, "read_page": read_page_fn, "summarize": summarize_fn, } # anywhere in the loop: result = TOOLS[name](**input) ``` One line of dispatch. Adding a tool is one line in the dict and one new function. The agent loop never changes. ## Where the chain comes from Notice that *the loop* doesn't define the order `search → read_page → summarize`. The model does. You hand it a registry; the model decides which tool serves the user's question this turn, sees the result, and decides which one comes next. This is the part that surprises people first writing agents: there's no orchestration code. The orchestration is the model's reasoning, expressed as a sequence of `tool_use` blocks. Your job is to keep the loop honest — dispatch correctly, hand back well-formed `tool_result` blocks, and keep the message history intact. ## What you'll see in the editor Three tools registered as a dict. One `tool_use` block from the model picking `search`. One line of dispatch — `TOOLS[name](**input)` — runs the right function. Print the result. Run it. The next steps wire this dispatch into the full loop, then add parallel tool calls and the bugs that come with them.2e:T9bd,# A tool is two things, not one The mental shortcut is "a tool is a function." It's almost right and it's the source of the second-most-common bug in beginner agent code. The full picture: - **Tool definition** — what the *model* sees. A name, a description, and a JSON schema for its input. The model uses the description to decide whether this tool is the right one for the next step. - **Tool function** — what *your code* runs. A Python function (or callable) that takes the same arguments the schema describes and returns a string the model can read. The model never executes Python. It only sees the schema. Your loop never reads the description. It only calls the function. Both halves have to exist and they have to agree on the name. ## Why descriptions matter more than you'd think The model picks tools based on the `description` field. If your description is `"Search."` the model will guess about when to use it. If your description is `"Search the web for current information. Returns up to 3 result URLs as plain text. Use when the user asks about events after your knowledge cutoff."` the model picks correctly on the first try. Bad descriptions are why your agent calls `search` for "what's two plus two" — the model has no signal that this isn't what `search` is for. ## Why JSON Schema, not Python type hints The model produces JSON. Your loop receives JSON. The schema lives in the tool definition because the *model* validates against it before emitting a `tool_use` block. Python type hints don't reach the wire. In real agents you'll see two patterns: ```python # pattern 1 — handwritten schemas (clearest, most explicit) {"name": "search", "input_schema": {"type": "object", "properties": {...}}} # pattern 2 — derive from Pydantic / dataclasses (less typing) {"name": "search", "input_schema": SearchArgs.model_json_schema()} ``` Both ship in production. Frameworks like LangChain default to pattern 2; you'll learn pattern 1 here because you'll read it in everyone's code, including the SDK examples. ## What to register: every tool, every turn A subtle thing: you send the *full* tool list every API call. The model doesn't remember which tools you registered last turn. Each turn includes the messages-so-far AND the full tool list. That's by design — it lets you add or remove tools mid-conversation if needed, e.g. "now that the user is logged in, also offer `delete_account`." For most loops you'll just keep the list constant.2f:T4ae,# a research-style chain. the fake model picks search → read_page → end_turn. TOOLS = { "search": lambda q: "https://news.example/ramen", "read_page": lambda url: "Tokyo's best ramen shops, 2026 guide.", } responses = iter([ {"stop_reason": "tool_use", "content": [ {"type": "tool_use", "id": "t1", "name": "search", "input": {"q": "ramen tokyo"}} ]}, {"stop_reason": "tool_use", "content": [ {"type": "tool_use", "id": "t2", "name": "read_page", "input": {"url": "https://news.example/ramen"}} ]}, {"stop_reason": "end_turn", "content": [ {"type": "text", "text": "found a 2026 ramen guide"} ]}, ]) def fake_model(messages): return next(responses) messages = [{"role": "user", "content": "find ramen news"}] for _ in range(5): r = fake_model(messages) if r["stop_reason"] == "end_turn": print(r["content"][0]["text"]) break tu = r["content"][0] out = TOOLS[tu["name"]](**tu["input"]) messages.append({"role": "assistant", "content": r["content"]}) messages.append({"role": "user", "content": [{ "type": "tool_result", "tool_use_id": tu["id"], "content": out, }]}) 30:T603,TOOLS = { "search": lambda q: "https://news.example/ramen", "read_page": lambda url: "Tokyo's best ramen shops, 2026 guide.", } responses = iter([ {"stop_reason": "tool_use", "content": [ {"type": "tool_use", "id": "t1", "name": "search", "input": {"q": "ramen tokyo"}} ]}, {"stop_reason": "tool_use", "content": [ {"type": "tool_use", "id": "t2", "name": "read_page", "input": {"url": "https://news.example/ramen"}} ]}, {"stop_reason": "end_turn", "content": [ {"type": "text", "text": "found: Tokyo's best ramen shops, 2026 guide."} ]}, ]) def fake_model(messages): return next(responses) def run_agent(question): messages = [{"role": "user", "content": question}] for _ in range(5): response = fake_model(messages) if response["stop_reason"] == "end_turn": return response["content"][0]["text"] tool_uses = [b for b in response["content"] if b["type"] == "tool_use"] tool_results = [] for tu in tool_uses: fn = TOOLS.get(tu["name"]) out = fn(**tu["input"]) if fn else f"unknown tool: {tu['name']}" tool_results.append({ "type": "tool_result", "tool_use_id": tu["id"], "content": out, }) messages.append({"role": "assistant", "content": response["content"]}) messages.append({"role": "user", "content": tool_results}) return "capped" print(run_agent("find the latest ramen news")) 31:T42c,Final drill. Build a research agent with three tools and parallel tool calls. Write `run_research_agent(question, max_iterations)` that: - Maintains a `messages` list, starting with one user turn. - Loops up to `max_iterations` times, calling `fake_model(messages)`. - On `end_turn`: return `{"ok": True, "answer": , "rounds": , "tools_called": }`. - On `tool_use`: handle ONE OR MORE tool_use blocks per turn. For each, dispatch through `TOOLS`. If the tool name isn't in the registry, the tool result content should be `"unknown tool: "` (don't crash). Each tool you successfully run counts toward `tools_called`. Append the assistant turn + one user turn with all tool_result blocks. - If `max_iterations` is exhausted without `end_turn`: return `{"ok": False, "answer": "capped", "rounds": max_iterations, "tools_called": }`. Two cases run for you. Expected output: ``` ok=True rounds=3 tools=3 answer=found 2 sources on ramen ok=False rounds=2 tools=2 answer=capped ``` 32:T684,TOOLS = { "search": lambda q: f"results for {q}", "read_page": lambda url: f"", "summarize": lambda text: text[:30], } good = iter([ # turn 1: parallel — search AND read_page in one assistant turn. {"stop_reason": "tool_use", "content": [ {"type": "tool_use", "id": "t1", "name": "search", "input": {"q": "ramen"}}, {"type": "tool_use", "id": "t2", "name": "read_page", "input": {"url": "a.com"}}, ]}, # turn 2: one summarize call. {"stop_reason": "tool_use", "content": [ {"type": "tool_use", "id": "t3", "name": "summarize", "input": {"text": "long article"}}, ]}, # turn 3: done. {"stop_reason": "end_turn", "content": [ {"type": "text", "text": "found 2 sources on ramen"}, ]}, ]) cap = iter([ {"stop_reason": "tool_use", "content": [ {"type": "tool_use", "id": "c1", "name": "search", "input": {"q": "x"}} ]}, {"stop_reason": "tool_use", "content": [ {"type": "tool_use", "id": "c2", "name": "search", "input": {"q": "y"}} ]}, {"stop_reason": "tool_use", "content": [ {"type": "tool_use", "id": "c3", "name": "search", "input": {"q": "z"}} ]}, ]) current = {"iter": good} def fake_model(messages): return next(current["iter"]) # define run_research_agent(question, max_iterations) below current["iter"] = good r = run_research_agent("research ramen", 5) print(f"ok={r['ok']} rounds={r['rounds']} tools={r['tools_called']} answer={r['answer']}") current["iter"] = cap r = run_research_agent("loop forever", 2) print(f"ok={r['ok']} rounds={r['rounds']} tools={r['tools_called']} answer={r['answer']}") 33:Taaf,TOOLS = { "search": lambda q: f"results for {q}", "read_page": lambda url: f"", "summarize": lambda text: text[:30], } good = iter([ {"stop_reason": "tool_use", "content": [ {"type": "tool_use", "id": "t1", "name": "search", "input": {"q": "ramen"}}, {"type": "tool_use", "id": "t2", "name": "read_page", "input": {"url": "a.com"}}, ]}, {"stop_reason": "tool_use", "content": [ {"type": "tool_use", "id": "t3", "name": "summarize", "input": {"text": "long article"}}, ]}, {"stop_reason": "end_turn", "content": [ {"type": "text", "text": "found 2 sources on ramen"}, ]}, ]) cap = iter([ {"stop_reason": "tool_use", "content": [ {"type": "tool_use", "id": "c1", "name": "search", "input": {"q": "x"}} ]}, {"stop_reason": "tool_use", "content": [ {"type": "tool_use", "id": "c2", "name": "search", "input": {"q": "y"}} ]}, {"stop_reason": "tool_use", "content": [ {"type": "tool_use", "id": "c3", "name": "search", "input": {"q": "z"}} ]}, ]) current = {"iter": good} def fake_model(messages): return next(current["iter"]) def run_research_agent(question, max_iterations): messages = [{"role": "user", "content": question}] tools_called = 0 for rounds in range(1, max_iterations + 1): response = fake_model(messages) if response["stop_reason"] == "end_turn": return { "ok": True, "answer": response["content"][0]["text"], "rounds": rounds, "tools_called": tools_called, } tool_uses = [b for b in response["content"] if b["type"] == "tool_use"] tool_results = [] for tu in tool_uses: fn = TOOLS.get(tu["name"]) if fn: out = fn(**tu["input"]) tools_called += 1 else: out = f"unknown tool: {tu['name']}" tool_results.append({ "type": "tool_result", "tool_use_id": tu["id"], "content": out, }) messages.append({"role": "assistant", "content": response["content"]}) messages.append({"role": "user", "content": tool_results}) return { "ok": False, "answer": "capped", "rounds": max_iterations, "tools_called": tools_called, } current["iter"] = good r = run_research_agent("research ramen", 5) print(f"ok={r['ok']} rounds={r['rounds']} tools={r['tools_called']} answer={r['answer']}") current["iter"] = cap r = run_research_agent("loop forever", 2) print(f"ok={r['ok']} rounds={r['rounds']} tools={r['tools_called']} answer={r['answer']}") 34:Tba4,# Pick the path before you do the work Lesson 02 had one agent with many tools. Routing flips it: many *specialist* agents, and a small classifier picks which one runs. The shape is intuitive once you see it. A user asks something. You don't send it to your one mega-prompt. You send it to a tiny model call whose only job is to return a category — `billing`, `technical`, `general` — and that category maps to a handler with its own prompt, its own tools, and possibly its own model size. Every customer support bot you've ever talked to is doing this. Every "tell me what kind of question this is, then I'll answer" UX is doing this. Anthropic's *Building Effective Agents* post calls it one of the five canonical patterns. LangChain calls it a `RouterChain`. The Vercel AI SDK exposes it via `experimental_output` + a switch. Same primitive underneath — small classifier, dispatch table, specialist runs. ## Why route at all Three reasons, in the order they bite: 1. **One mega-prompt rots fast.** A prompt that handles billing AND technical AND general gets longer every time you find an edge case. By the third edge case it contradicts itself, and the model guesses. 2. **Different routes have different costs.** A billing question needs to read the order database and call the refund API. A "what time do you open" question needs neither. Routing lets you call cheap tools for cheap routes and expensive tools only when you have to. 3. **Different routes have different models.** Sometimes the billing route needs your best, most cautious model. The general route is fine on Haiku. Routing makes that choice explicit. ## The two-call shape Most routers run as two model calls per user turn: ``` user → classifier (small model) → category category + user → specialist (any model) → answer ``` The classifier call is short and cheap. The specialist call is where the real work happens. People who measure tokens find the classifier costs about 5% of what one big merged prompt would — and the specialist's answer quality is *higher* because its prompt is smaller and focused. ## When NOT to route Routing is overkill when: - You only have one path. Don't add a classifier whose answer is always the same. - The categories overlap heavily and you'll need cross-route context anyway. (You probably want one prompt with branches, not routing.) - The user can pick the category from a UI dropdown. The user is a better classifier than your model and free. The bug isn't "should I route" — it's "should this *new* path become its own route, or fold into an existing route." Default to folding until a route earns its keep. ## What you'll build A `route(question)` function that calls a tiny classifier, looks up the matching specialist in a dispatch table, and runs it. Then a fallback for when the classifier returns garbage. Then a real multi-route agent that handles three categories plus an unknown.35:T87f,# The classifier output is the most important constraint A routing system fails the *first* time the classifier returns a string that isn't in your dispatch table. `KeyError`. The user gets a 500. You get paged. The fix is constraining the classifier *before* it answers, not after. Three ways, in increasing reliability: ## 1. Enum in the prompt (weakest) ``` "Classify the user message as one of: billing, technical, general. Respond with ONLY the category word, nothing else." ``` This *usually* works. Modern models are obedient. But "usually" is not a reliability guarantee. Production routers see drift here: `"Billing"` (capital B), `"billing or technical"`, `"general question"`. Each is a `KeyError` waiting to happen. ## 2. Structured output with an `enum` schema (strong) ```python { "type": "object", "properties": { "category": {"type": "string", "enum": ["billing", "technical", "general"]} }, "required": ["category"] } ``` Now the model can't return `"Billing"` even if it wants to — the schema rejects it. Anthropic, OpenAI, Google all support this via their "tool use as structured output" trick: define a tool whose input schema is your classification schema, force the model to call it, and read the input. ## 3. Validate-then-fallback (production-grade) Even with structured output, you want one more layer: ```python category = classification.get("category", "") if category not in ALLOWED: category = "general" # or "fallback", or whatever your default is ``` This is the load-bearing line in every routing system that's been in production for more than a month. Models drift. Schemas can be bypassed by a prompt-injected user. The dispatch table needs a safe default. ## What people get wrong The most common failure mode in beginner routers is using the model to do *both* the classification and the answer in one call — "classify this and respond appropriately." The model usually does, but you've lost the dispatch decoupling. You can't swap out a specialist. You can't measure per-route cost. You can't log which route got chosen. Routing earns its value from the explicit two-call shape — keep them separate.36:T4c1,def billing_specialist(q): return {"answer": "refunded", "tools_used": 1} def technical_specialist(q): return {"answer": "restarted", "tools_used": 2} def general_specialist(q): if "open" in q or "hour" in q: return {"answer": "hours: 9-5", "tools_used": 0} return {"answer": "fallback answer", "tools_used": 0} SPECIALISTS = { "billing": billing_specialist, "technical": technical_specialist, "general": general_specialist, } classifier_outputs = iter(["BILLING", " technical ", "general", "weird_unknown"]) def fake_classifier(question): return next(classifier_outputs) def run_router(question): raw = fake_classifier(question) category = raw.strip().lower() if category not in SPECIALISTS: category = "general" result = SPECIALISTS[category](question) return { "category": category, "answer": result["answer"], "tools_used": result["tools_used"], } questions = [ "Refund my subscription", "Site is throwing 500s", "What hours are you open?", "totally off topic", ] for q in questions: r = run_router(q) print(f"category={r['category']} tools={r['tools_used']} answer={r['answer']}") 37:Tcf9,# Two prompts beat one mega-prompt There's a reason the best AI features look like Cursor's "let me try again" loop and not like a single long prompt: when one model writes and a *different* prompt critiques, you get to keep iterating until the output meets a bar. One prompt has to be everything to everyone and ships its first guess. That's the evaluator-optimizer pattern. Anthropic calls it out as one of the five canonical agent workflows in *Building Effective Agents*: > "In the evaluator-optimizer workflow, one LLM call generates a > response while another provides evaluation and feedback in a loop." The shape is a tiny state machine: ``` ┌────────────┐ draft ┌────────────┐ │ Generator ├──────────▶ Evaluator │ └────────────┘ └─────┬──────┘ ▲ │ │ feedback (REJECT) │ verdict └────────────────────────┘ │ │ PASS ▼ return ``` Two model calls per loop iteration. One generates. One judges. If the judge accepts, you're done. If not, the judge's feedback flows into the next generation as additional context, and you go again. With a ceiling on iterations so a stubborn judge doesn't bankrupt you. ## When this pattern earns its keep Anthropic's two signals: 1. **A human articulating their feedback would demonstrably improve the response.** If the response is already as good as you can make it by talking to it, the loop has nothing to optimize. 2. **The LLM can produce that kind of feedback.** Some domains the judge is great at (code style, factual coverage, tone). Others it isn't (mathematical correctness, factual recall against a database it can't see). Two textbook fits: literary translation (you can keep refining nuance) and code review (correctness + style + complexity, all checkable in text). Two textbook anti-fits: arithmetic (no amount of critique will fix a wrong number) and "is this funny" (judge has no ground truth). ## What everyone calls it | Framework | Name | |---|---| | Anthropic cookbook | `evaluator_optimizer` (XML `PASS` protocol) | | LangGraph | "Evaluator-optimizer workflow" — two nodes + conditional edge | | Vercel AI SDK | "Evaluator-Optimizer" pattern — `generateText` + `generateObject` + `while` | | AutoGen | "Reflection" design pattern — two agents in a loop | | OpenAI Agents SDK | No native primitive — manual `Agent` + `Agent` driver | Same primitive everywhere: a writer, a judge, a feedback channel, a ceiling. You'll write the framework-free version in step 8. Once you have, you'll read every framework's wrapper and recognize what it's hiding. ## What you'll build A `write_with_judge(task, generator, judge, max_iters)` that runs the loop. Then a fix for the most common bug (parsing the judge's verdict from free text instead of structured output). Then the cap fix everyone forgets. Then a real version that threads the judge's feedback back into the generator and counts iterations.38:Tbcc,# The judge's output is a contract, not a vibe The single most-broken evaluator-optimizer in the wild has the same bug: parsing the judge's verdict from free text. The code looks something like: ```python verdict = judge(draft) if "good" in verdict.lower() or "pass" in verdict.lower(): return draft ``` This breaks the first time the judge says *"this is **not** good"* — the substring match sees `"good"`, returns happily, and ships the broken draft. Three hardening levels, in production order: ## Level 1 — XML tags (Anthropic cookbook style) Force the judge to wrap its verdict in a fixed token: ``` PASS or NEEDS_IMPROVEMENT ... ``` You parse by extracting the contents of ``. Robust enough for prototypes; the cookbook ships it. ## Level 2 — Structured output (preferred today) Use the model's structured-output mode and a JSON schema: ```python { "status": {"type": "string", "enum": ["PASS", "REJECT"]}, "feedback": {"type": "string"}, "score": {"type": "integer", "minimum": 1, "maximum": 10} } ``` The model literally cannot return `"PaSs"` or `"good"` — the schema forbids it. Anthropic does this via `tools=[...]` (the tool-use trick). OpenAI does it via `response_format={"type": "json_schema"}`. Vercel AI SDK does it via `generateObject` + Zod. ## Level 3 — Validate-then-default Even with structured output, one more belt-and-suspenders line: ```python status = verdict.get("status", "REJECT") if status not in {"PASS", "REJECT"}: status = "REJECT" # default to "loop again" — never to PASS ``` The default-on-anomaly is always REJECT, never PASS. Because the failure mode of "model returned garbage and we shipped" is worse than "model returned garbage and we did one more revision." ## Why a separate judge prompt matters The other half of the contract: the judge cannot share a system prompt with the generator. Self-preference bias is documented (arxiv.org/abs/2410.21819) — LLM judges systematically rate their own (and more familiar-looking) outputs higher than humans do. When the same prompt-personality writes *and* judges, that bias tilts the verdict toward PASS regardless of actual quality. The judge needs to be told to *find specific defects*, not asked for an opinion. Different model entirely is even better when you can afford it. ## Three numbers production loops always have ```python feedback = None draft = None for i in range(MAX_ITERATIONS): # 1. cap draft = generator(task, feedback=feedback) verdict = judge(draft) # 2. structured verdict if verdict["status"] == "PASS": return draft feedback = verdict["feedback"] # 3. feedback flows back return draft # bail with the latest draft, never raise ``` Three load-bearing pieces: **MAX_ITERATIONS**, **structured verdict**, **feedback threaded back into the generator**. Miss any one and the loop fails — runs forever, ships garbage, or never improves.39:T51d,good_drafts = iter([ "Cats are animals.", "Cats are small carnivorous mammals.", ]) good_verdicts = iter([ {"status": "REJECT", "feedback": "Too short."}, {"status": "PASS", "feedback": "Detailed."}, ]) bad_drafts = iter(["v1", "v2"]) bad_verdicts = iter([ {"status": "REJECT", "feedback": "no"}, {"status": "REJECT", "feedback": "still no"}, ]) current = {"drafts": good_drafts, "verdicts": good_verdicts} def gen(task, feedback=None): return next(current["drafts"]) def judge(draft): return next(current["verdicts"]) def write_with_judge(task, max_iters): feedback = None draft = None for i in range(max_iters): draft = gen(task, feedback=feedback) verdict = judge(draft) if verdict["status"] == "PASS": return {"draft": draft, "iters": i + 1, "passed": True} feedback = verdict["feedback"] return {"draft": draft, "iters": max_iters, "passed": False} current["drafts"], current["verdicts"] = good_drafts, good_verdicts r = write_with_judge("describe cats", 3) print(f"passed={r['passed']} iters={r['iters']} draft={r['draft']}") current["drafts"], current["verdicts"] = bad_drafts, bad_verdicts r = write_with_judge("describe cats", 2) print(f"passed={r['passed']} iters={r['iters']} draft={r['draft']}") 3a:T41b,good_drafts = iter(["v1-rough", "v2-better", "v3-good"]) good_verdicts = iter([ {"status": "REJECT", "feedback": "rough"}, {"status": "REJECT", "feedback": "better but not enough"}, {"status": "PASS", "feedback": "great"}, ]) bad_drafts = iter(["stub1", "stub2"]) bad_verdicts = iter([ {"status": "REJECT", "feedback": "still no"}, {"status": "REJECT", "feedback": "really no"}, ]) current = {"drafts": good_drafts, "verdicts": good_verdicts} def gen(task, feedback=None): return next(current["drafts"]) def judge(draft): return next(current["verdicts"]) # define optimize(task, max_iters) below current["drafts"], current["verdicts"] = good_drafts, good_verdicts r = optimize("write a tagline", 5) print(f"passed={r['passed']} iters={r['iters']} history_len={len(r['history'])} final={r['final_draft']}") current["drafts"], current["verdicts"] = bad_drafts, bad_verdicts r = optimize("write a tagline", 2) print(f"passed={r['passed']} iters={r['iters']} history_len={len(r['history'])} final={r['final_draft']}") 3b:T6e1,good_drafts = iter(["v1-rough", "v2-better", "v3-good"]) good_verdicts = iter([ {"status": "REJECT", "feedback": "rough"}, {"status": "REJECT", "feedback": "better but not enough"}, {"status": "PASS", "feedback": "great"}, ]) bad_drafts = iter(["stub1", "stub2"]) bad_verdicts = iter([ {"status": "REJECT", "feedback": "still no"}, {"status": "REJECT", "feedback": "really no"}, ]) current = {"drafts": good_drafts, "verdicts": good_verdicts} def gen(task, feedback=None): return next(current["drafts"]) def judge(draft): return next(current["verdicts"]) def optimize(task, max_iters): feedback = None draft = None history = [] for i in range(max_iters): draft = gen(task, feedback=feedback) verdict = judge(draft) history.append({ "iter": i + 1, "draft": draft, "status": verdict["status"], "feedback": verdict["feedback"], }) if verdict["status"] == "PASS": return { "final_draft": draft, "iters": i + 1, "passed": True, "history": history, } feedback = verdict["feedback"] return { "final_draft": draft, "iters": max_iters, "passed": False, "history": history, } current["drafts"], current["verdicts"] = good_drafts, good_verdicts r = optimize("write a tagline", 5) print(f"passed={r['passed']} iters={r['iters']} history_len={len(r['history'])} final={r['final_draft']}") current["drafts"], current["verdicts"] = bad_drafts, bad_verdicts r = optimize("write a tagline", 2) print(f"passed={r['passed']} iters={r['iters']} history_len={len(r['history'])} final={r['final_draft']}") 3c:T144b,# Five frameworks, one loop You just wrote the agent loop in @@pd-path:path.ref:agent-loops@@, step 1.8. Thirty lines. A `while turn < max_turns:` block, one `messages.create` call, a `stop_reason` branch on `tool_use` vs `end_turn`, and a `messages.append` for the tool result. That's the whole machine. Now open the source of any "agent framework" shipped in the last eighteen months. LangGraph. CrewAI. AutoGen. Vercel AI SDK. OpenAI's Swarm (and its successor, the Agents SDK). DSPy's agent module. LlamaIndex agent runners. Pull up GitHub. Search for `while`. You'll find your loop. Different class names wrapped around it, different opinions about retries, memory, and observability, but the loop is the same loop. The one-liner you'll hear all over the ecosystem: an agent is a `while` loop with a model call inside. It's not a slogan. It's a technical observation about what the model API will let you build. The Anthropic and OpenAI chat-completions APIs return a response with a stop signal and an optional tool-use request. Anything that wants to chain tool calls *has* to wrap that response in a loop. There is no other shape. The API itself dictates the convergence. ## Why this matters for buy-vs-build If every framework is the same loop, the question isn't "which one is correct?" It's "which one's *ergonomics* match my team and my product?" That's a real question with a real answer, but it has nothing to do with capability. It has to do with: - **Language fit.** If your product is a Next.js app, the Vercel AI SDK reads like home. If your backend is FastAPI, LangGraph is the obvious neighbor. Picking against the grain of your stack adds friction nothing else will pay back. - **Durability needs.** If your agent has to survive a server restart mid-task — say, a 40-minute research run — LangGraph's checkpointed `StateGraph` gives you that for free. Rolling it yourself is a week of work. - **Team size.** Two engineers and a deadline is framework-free territory. Twenty engineers and a need for shared abstractions is not. Frameworks are coordination tools, not capability tools. - **Time to first demo.** A framework with sensible defaults gets you a demo by Friday. A from-scratch loop gets you one by Wednesday of next week — but the next-month features cost you less. The frameworks are not selling capability. They're selling a particular bundle of abstractions, a particular set of defaults, and a particular community. When you "pick a framework" you're really picking those three things. ## What the frameworks actually add Strip away the marketing. What you're paying for, in order of how much each one matters in practice: 1. **Durable state.** Checkpointing the conversation + scratchpad to disk so a crash doesn't lose the run. LangGraph leads here. 2. **Branching and parallelism.** Running multiple tool calls at once, or forking the loop down two paths and merging the result. You can roll this with `asyncio.gather`, but a framework gives it to you with one decorator. 3. **Observability hooks.** Every model call, every tool call, every token logged to a tracing backend. LangSmith, Helicone, Phoenix, Logfire — they all hook into framework-emitted events. 4. **Retry/timeout policies.** Per-tool timeouts, exponential backoff, dead-letter queues. You'll write these yourself anyway; the question is whether you want the framework's opinion or yours. 5. **Multi-agent coordination.** CrewAI and AutoGen specialize here. "Researcher agent talks to writer agent talks to editor agent." These can be valuable when your problem is genuinely multi-agent, and they're overkill when it isn't. Everything above the dotted line is *the same loop you already wrote* plus engineering work you'd otherwise do yourself. ## What Anthropic recommends In *Building Effective Agents* (December 2024), Anthropic — the team that builds the model the loop runs against — said it plainly: > We suggest that developers start by using LLM APIs directly … > make sure you understand the underlying code. Incorrect > assumptions about what's under the hood are a common source of > customer error. And their headline finding, from the same post: the "most successful implementations weren't using complex frameworks" — they were built from simple, composable patterns. Translation: write the loop yourself first. Adopt a framework when you outgrow your loop, not before. Devin's team, Cursor's team, and most of the agent-shaped products that actually ship in 2025 follow exactly this pattern — a thin custom loop in production, frameworks used in exploration and prototyping. ## Where this lesson goes Three reads ahead. First, LangGraph — what Harrison Chase built, what it gives you, what it costs you. Then the Vercel AI SDK — a completely different shape because it's a completely different audience (web devs, streaming-first). Then the five canonical agent patterns from Anthropic's playbook, all of which every framework implements. By the end you'll be able to look at a project profile and say, with confidence, "use X" or "stay framework-free." That's the decision in step 7.3d:T11f5,# LangGraph — what Harrison Chase built and why Harrison Chase shipped LangChain in October 2022. By mid-2024 the LangChain community had a problem: chains were linear, but real agent work has cycles. The model calls a tool, looks at the result, decides to call another tool, decides to revise its earlier answer, loops back. LangChain's `SequentialChain` couldn't express this cleanly. So Chase built LangGraph and pitched it on exactly that gap: *"chains are for DAGs; agents need cycles."* The core abstraction is `StateGraph`. You define: - A **state** schema (a TypedDict — what the agent knows at any point in the run). - **Nodes** (functions that take state and return updates to it). - **Edges** (which node runs after which, possibly conditional on state). The agent loop becomes two nodes (one for the model call, one for tool dispatch) and a conditional edge that re-enters the model node when there are pending tool calls. Look at any LangGraph "react agent" example. The `should_continue` function is the `if stop_reason == "tool_use"` branch you wrote. Same logic, different vocabulary. ## What you get for adopting LangGraph - **Checkpointing for free.** `MemorySaver`, `SqliteSaver`, `PostgresSaver`. The graph state persists across runs. If your agent dies mid-task, you can resume from the last checkpoint. This is the single biggest reason teams pick LangGraph over hand-rolled loops. - **Time-travel debugging.** Because state is checkpointed, you can rewind, edit the state, and replay forward. Invaluable when you're debugging a 12-step trace where step 7 went sideways. - **Branching and merging primitives.** `Send` API for spawning parallel subgraphs and collecting results. Easier than rolling `asyncio.gather` patterns yourself. - **LangSmith integration.** Every node execution, every state delta, every model call logged. If your team needs production observability, this is the lowest-friction path. - **A real community.** Templates for common patterns. People on Discord who've debugged your bug. Job postings that mention LangGraph by name. ## What you pay for adopting LangGraph - **Lock-in.** Your business logic gets shaped like a graph. If you decide to move off LangGraph in eighteen months, the rewrite is not trivial. The state schema, the node functions, the conditional edges — they all assume the graph runtime. - **Debugging through abstractions.** When a node misbehaves, you're often debugging two layers: your node logic, and how LangGraph scheduled it. Stack traces include `pregel`, `channels`, and other internal vocabulary that new engineers have to learn before they can read errors. - **Opinion-heavy defaults.** LangGraph has views on how state should merge, how interruptions should work, how human-in-the-loop should look. If you agree with the views, life is easy. If you disagree, you fight the framework. - **Version churn.** LangChain and LangGraph have iterated fast. Code written against a six-month-old version often needs non-trivial migration. Lock to specific versions in production. ## When LangGraph is the right call Three profiles where LangGraph genuinely earns its weight: 1. **Long-running agents that must survive restarts.** A research-deep-dive agent that runs for an hour. A sales-outreach agent that's mid-conversation when the server redeploys. Checkpointing is the killer feature. 2. **Multi-agent workflows with explicit handoffs.** When you have five named agents and the handoff logic between them matters, modeling that as a graph is clearer than modeling it as conditionals in a custom loop. 3. **Teams of 5+ engineers on the same agent codebase.** The graph abstraction is a coordination tool. Your fifth engineer can read a `StateGraph` and understand the flow faster than they can read 1000 lines of custom orchestration. ## When LangGraph is the wrong call - A single-developer prototype. The setup tax (state schema, nodes, edges) costs more than the value at this scale. - An agent that runs for under 60 seconds with no need for resumption. The checkpointing is dead weight. - A team that wants to deeply understand its loop. You'll be fighting two abstractions instead of one. The honest read: LangGraph is good. The team behind it is serious. The marketing is louder than the substance warrants, but the substance is real. The mistake teams make isn't picking LangGraph — it's picking it too early, before they understand what they actually need from it. Write the loop. Hit the wall. Then adopt.3e:T967,# The work itself is a graph You just watched LangGraph spell it out: nodes are steps, edges say what runs after what. Here's the part worth keeping even if you never touch the framework — **that structure isn't a LangGraph invention. It's what multi-step work already looks like**, and you can hold the whole idea in a dict. The editor shows a three-step workflow as plain data. Each step lists what it `needs`; a step is runnable when everything it needs is done. That one comprehension — find steps whose dependencies are all satisfied — is the entire scheduling logic of every workflow engine you'll ever meet. Four ideas turn this toy into a vocabulary for real agent workflows: **Dependencies.** An edge means "this must finish before that starts." Write them down and you've made the implicit order of your work explicit — which is the precondition for automating any of it. **Joins.** A step can need *several* upstream steps (`"needs": ["research", "legal-review"]`). The `all(...)` check handles it for free: the step waits until every branch lands. Most real workflows are diamonds, not straight lines. **Cycles need convergence rules.** A retry edge ("check failed → back to draft") makes the graph loop. Loops without an exit condition run forever — so every cycle must carry a rule that guarantees it ends: a maximum number of passes, a threshold that counts as good enough, or an escalation to a human. If you can't say what makes the loop stop, the loop isn't finished being designed. **Verification gates.** The `send` step carries `"gate": "human approval"` — an edge whose condition is *a check passing*, not just *the previous step finishing*. Gates are where you put the checks this course keeps drilling: schema validation on the way in, a human sign-off before anything irreversible goes out. One portable format captures all four. Any workflow you can fill into this table, you can automate — and any row you *can't* fill is a design question you haven't answered yet: | From | Relationship | To | Condition | Evidence | |---|---|---|---|---| | check | must pass before | send | human approves diff | approval logged | Run the editor. Only `draft` is runnable — nothing it needs, nothing done yet. Flip `draft` to `"done": True` and rerun: now `check` becomes runnable, and `send` still waits. That's dependency ordering, in four lines you wrote yourself.3f:T13cb,# Vercel AI SDK — a different shape entirely LangGraph is a Python framework for backend engineers. Vercel AI SDK is a TypeScript SDK for web developers. The difference isn't just language. It's the assumption about *where the agent lives*. LangGraph assumes the agent runs on a server, possibly for a long time, possibly across multiple processes, possibly persisting state to a database. Vercel AI SDK assumes the agent runs inside a web request — a Next.js route handler, a Cloudflare Worker, an Edge Function. The user is watching a chat UI, waiting for tokens to stream in. Latency is everything. The request is short. State lives in the browser (or wherever your auth/session layer puts it). The shape of the loop is *the same loop you wrote*, but the framework optimizes for a completely different runtime. ## The primitive The Vercel AI SDK's agent loop is `streamText` (or `generateText` if you don't need streaming): ```typescript const result = streamText({ // The provider alias for Vercel AI Gateway is "anthropic/claude-opus-4.7" // (dot); the @ai-sdk/anthropic helper here accepts "claude-opus-4-7" // (hyphen) and resolves it. If you see an unknown-model error, // double-check the exact string against ai-sdk.dev/providers. model: anthropic("claude-opus-4-7"), messages, tools: { getWeather, searchDocs }, stopWhen: stepCountIs(8), // max-turns guard }); ``` That's it. The loop is implicit. `streamText` handles model call, tool dispatch, message-history append, and re-entry — all the things you wrote by hand in step 1.8. `stopWhen` is the iteration cap. The result streams to the browser as it generates. The model can call tools mid-stream and the SDK splices the tool results into the continuation. ## What you get for adopting Vercel AI SDK - **Streaming-first.** Token streaming, tool-call streaming, partial response streaming — all built in. If you're building anything with a chat UI, this is the single most-correct framework for that job. - **Provider abstraction.** Same code targets OpenAI, Anthropic, Google, xAI, Groq, Together, OpenRouter. Swap models with one import. - **React hooks.** `useChat`, `useCompletion`, `useObject`. The client side is one hook; the server side is one route handler. Two files for a working chat agent. - **Generative UI.** Tools can return React components, not just text. The agent literally renders UI as part of its response. Unique to this SDK; pioneered by Vercel and now copied broadly. - **Edge-runtime compatible.** Runs on Cloudflare Workers, Vercel Edge, Deno Deploy. The loop is small enough to fit. ## What you pay for adopting Vercel AI SDK - **TypeScript-only.** No Python port. If your stack is Python, this is not for you. (LangGraph has a JS port; Vercel AI SDK does not have a Python port.) - **Web-shaped assumptions.** It assumes you're returning to a client over HTTP. Building a long-running background agent? The SDK doesn't fight you, but it's not designed for that case. - **Tied to model provider APIs.** When OpenAI ships a new responses-API shape or Anthropic ships a new content block, you wait for an SDK update. Usually fast (Vercel ships weekly), but it's a real coupling. - **Less plumbing for long-tail features.** Multi-agent coordination, human-in-the-loop interruption, checkpointing — these exist but are thinner than LangGraph's equivalents. ## When Vercel AI SDK is the right call The decision is dominated by *where the product lives*. Three profiles: 1. **Next.js or Remix app with a chat interface.** This is the killer use case. The SDK was built for exactly this shape. Two files, working agent, streamed to the browser. 2. **AI features inside an existing TypeScript product.** A "summarize this" button, a "generate variants" sidebar, an inline-agent feature. The SDK fits beside the rest of your stack instead of being a separate service. 3. **Generative-UI prototypes.** When tools should return React components, the SDK is the only mainstream framework that handles this well. Useful for product demos and exploration. ## When Vercel AI SDK is the wrong call - Your backend is Python and you don't want a Node.js service just for the agent loop. - The agent runs for >60 seconds and needs to survive restarts. The SDK has no first-class checkpointing. - You need deep multi-agent coordination. CrewAI or LangGraph fit better. ## The takeaway Framework choice is *downstream of where your product lives*. The question isn't "which framework is best?" — it's "which framework disappears into my stack?" LangGraph disappears into Python backends. Vercel AI SDK disappears into TypeScript web apps. CrewAI disappears into a "five agents in a writers' room" mental model. Pick the one your team will fight the least, not the one that's loudest on X. If you're not sure which side of the Python/TypeScript line your product lives on yet, you're framework-free until you are. Write the loop. Ship the demo. Adopt when the question answers itself.40:T1654,# Every framework implements the same five patterns In December 2024, Anthropic published *Building Effective Agents*. It became a widely referenced agent-engineering post because it does two useful things. First, it distinguishes *workflows* (LLMs orchestrated through predefined code paths) from *agents* (LLMs that direct their own paths). Second, it names five recurring workflow patterns that show up across agent systems. Anthropic's post also distinguishes these five WORKFLOW patterns from a sixth, distinct ARCHITECTURE: **"agents"** — autonomous loops where the model directs its own tool use rather than following a pre-set graph. The five below are the workflow shapes. The agent loop (@@pd-path:path.ref:agent-loops@@) is the sixth. If you read the docs of LangGraph, CrewAI, AutoGen, Vercel AI SDK, DSPy, or any other framework, you'll find each of these patterns implemented under a different name. They are the universal primitives. Knowing them lets you read any framework's docs in ten minutes flat. ## 1. Prompt chaining A task is decomposed into a sequence of LLM calls, each processing the output of the previous one. You can add "gates" between calls — programmatic checks that validate intermediate results before continuing. **When it's right:** the task decomposes cleanly into fixed subtasks. Example: generate a marketing draft, then translate it, then summarize. **Framework names:** LangGraph linear edges. Vercel AI SDK `generateText` calls in sequence. LangChain's old `SequentialChain`. ## 2. Routing A classifier LLM (or a smaller model) inspects the input and routes it to one of several specialized downstream prompts/agents/tools. **When it's right:** the inputs fall into distinct categories that benefit from different handling. Example: a customer support agent that routes refund requests to one prompt, technical questions to another, and account changes to a third. **Framework names:** LangGraph conditional edges. CrewAI's task dispatch. In the Vercel AI SDK it's plain conditional logic — a classifier call whose output picks which model, system prompt, or tools the next call uses. You did this exact pattern in @@pd-path:path.ref:agent-loops@@, step 3. ## 3. Parallelization Multiple LLM calls run simultaneously and their outputs are aggregated. Two flavors: **sectioning** (split the work into independent pieces) and **voting** (run the same task multiple times and merge for higher confidence). **When it's right:** subtasks are independent and benefit from parallel execution. Example: a code review that asks one call to check for security issues, another to check for style, another to check for test coverage — all simultaneously. **Framework names:** LangGraph's `Send` API. CrewAI's parallel task execution. Vercel AI SDK's `Promise.all` over multiple `streamText` calls. Or just raw `asyncio.gather` in a custom loop. ## 4. Orchestrator-workers A central orchestrator LLM dynamically decomposes a task into subtasks, dispatches each to a worker LLM, and synthesizes the results. The decomposition isn't predefined — the orchestrator figures it out at runtime. **When it's right:** you can't predict the subtasks ahead of time. Example: a coding agent that decides which files to edit based on the user's request, then dispatches edits to per-file worker agents. **Framework names:** CrewAI's "manager agent" with delegated tasks. AutoGen's `GroupChatManager`. LangGraph's "supervisor" pattern. Anthropic's own `claude-code` is largely this shape. ## 5. Evaluator-optimizer One LLM generates a response; another LLM evaluates it and provides feedback; the first revises. Loop until the evaluator passes or a max iteration count is hit. **When it's right:** there are clear quality criteria you can encode in an evaluator prompt, AND iteration measurably improves results. Example: literary translation, where a generator drafts and a critic flags nuance loss. **Framework names:** LangGraph "reflection" patterns. DSPy's optimizer module. Vercel AI SDK called in a loop. You wrote exactly this in @@pd-path:path.ref:agent-loops@@, step 4. ## What this means for buy-vs-build Every framework's *abstractions* are different. Every framework's *capabilities* are the same five patterns above. When you read a framework's docs and it markets a feature called "multi-agent delegation" or "reflection chains" or "supervisor mode" — go look up which of the five patterns it actually is. You'll always find the match. The market name is the framework's flavor; the underlying shape is in this list. Two consequences: 1. **No framework gives you more capability than the loop you already wrote.** All five patterns are 50-100 lines of Python each, on top of the 30-line core loop. Frameworks make them ergonomic; they don't make them possible. 2. **Framework choice is taste, not technology.** When you ask "which framework should I use?", what you're really asking is "whose taste in abstractions matches my team's?" That's a reasonable question with a reasonable answer, but it isn't a capability question. Anyone who tells you "X framework lets you do Y that nothing else can" is misreading the landscape. ## What the practitioners actually do Many agent products that run at scale use loops shaped like the one you wrote in @@pd-path:path.ref:agent-loops@@, dressed up with product-specific durability, observability, and tool-set decisions. Frameworks still matter, but their value is the abstraction and operations layer, not magic capability the underlying API could never express. The next mc tests when adding a framework makes things *worse*. Then you'll write the buy-vs-build memo yourself.41:T444,Write `recommend_framework(project)` that takes a project profile (dict) and returns a string — one of: - `"framework-free"` - `"vercel-ai-sdk"` - `"langgraph"` - `"crewai"` Each project has: - `team_size`: int - `language`: `"python"`, `"typescript"`, or `"both"` - `needs_streaming`: bool - `needs_durable_state`: bool - `time_to_first_demo_days`: int Apply the rules in this exact order (first match wins): 1. **framework-free** when the team is small AND the deadline is short: `team_size <= 2` AND `time_to_first_demo_days <= 14`. Prototypes don't earn the framework tax. 2. **vercel-ai-sdk** when streaming is required AND the stack is TypeScript-compatible: `needs_streaming` AND `language` in (`"typescript"`, `"both"`). 3. **langgraph** when state must persist across restarts AND the stack is Python-compatible: `needs_durable_state` AND `language` in (`"python"`, `"both"`). 4. Otherwise → **crewai** (multi-agent default when no other rule fires). Two projects run. Expected output: ``` Solo MVP: framework-free Research agent: langgraph ``` 42:T42d,projects = [ {"name": "Solo MVP", "team_size": 1, "language": "python", "needs_streaming": False, "needs_durable_state": False, "time_to_first_demo_days": 10}, {"name": "Chat widget", "team_size": 4, "language": "both", "needs_streaming": True, "needs_durable_state": False, "time_to_first_demo_days": 30}, {"name": "Research agent", "team_size": 6, "language": "python", "needs_streaming": False, "needs_durable_state": True, "time_to_first_demo_days": 90}, {"name": "Newsroom sim", "team_size": 5, "language": "python", "needs_streaming": False, "needs_durable_state": False, "time_to_first_demo_days": 60}, {"name": "Pair scripting tool", "team_size": 2, "language": "typescript", "needs_streaming": False, "needs_durable_state": False, "time_to_first_demo_days": 7}, ] # define audit_stack(projects) below result = audit_stack(projects) for p in projects: print(f"{p['name']:20} -> {result['recommendations'][p['name']]}") print(f"framework-free count: {result['framework_free_count']}") 43:T66f,projects = [ {"name": "Solo MVP", "team_size": 1, "language": "python", "needs_streaming": False, "needs_durable_state": False, "time_to_first_demo_days": 10}, {"name": "Chat widget", "team_size": 4, "language": "both", "needs_streaming": True, "needs_durable_state": False, "time_to_first_demo_days": 30}, {"name": "Research agent", "team_size": 6, "language": "python", "needs_streaming": False, "needs_durable_state": True, "time_to_first_demo_days": 90}, {"name": "Newsroom sim", "team_size": 5, "language": "python", "needs_streaming": False, "needs_durable_state": False, "time_to_first_demo_days": 60}, {"name": "Pair scripting tool", "team_size": 2, "language": "typescript", "needs_streaming": False, "needs_durable_state": False, "time_to_first_demo_days": 7}, ] def recommend_framework(project): if project["team_size"] <= 2 and project["time_to_first_demo_days"] <= 14: return "framework-free" if project["needs_streaming"] and project["language"] in ("typescript", "both"): return "vercel-ai-sdk" if project["needs_durable_state"] and project["language"] in ("python", "both"): return "langgraph" return "crewai" def audit_stack(projects): recs = {p["name"]: recommend_framework(p) for p in projects} ff = sum(1 for v in recs.values() if v == "framework-free") return {"recommendations": recs, "framework_free_count": ff} result = audit_stack(projects) for p in projects: print(f"{p['name']:20} -> {result['recommendations'][p['name']]}") print(f"framework-free count: {result['framework_free_count']}") 44:Te5e,# The loop is one layer of five You've spent this chapter at the loop level — the thirty lines, the stop reasons, the message-history hygiene. That loop is *one layer* of a much bigger stack. Knowing the loop lets you build an agent that runs. It doesn't yet tell you how the agent will *fail* in production, or where the industry's money is actually flowing. In May 2026, investor Chamath Palihapitiya published a primer on his Substack, *Deep Dive: The Agentic AI Economy*, that maps the agent economy in layers. The five labels below are this course's shorthand rendering of that kind of layered map — the exact names are ours, not a quotation from the primer — but seeing them side by side is the fastest way to know which layer you're actually working on. ## The five layers | Layer | What it does | |---|---| | **Intelligence** | Reasoning and planning — the model itself | | **Action** | Execution and tool use — *the loop you wrote in this chapter* | | **Governance** | Policy enforcement — the rules for what the agent may *not* do | | **Orchestration** | Control plane — routing, scheduling, and what runs where | | **Economics** | Cost structure — whether the per-task price actually sustains a business | Framing inspired by Chamath Palihapitiya, *Deep Dive: The Agentic AI Economy*, Substack, May 2026; layer names are this course's shorthand. The **Action** layer is the one this whole chapter walked through. Your loop calls a tool, reads the result, calls again — that's Action. The model underneath it is **Intelligence**. The other three layers — Governance, Orchestration, Economics — are everything *around* the loop that decides whether the loop survives contact with real users. ## Two ways the loop went wrong in public Two well-documented incidents that made the news while this course was being written. Both were loops that ran exactly as written — and still caused a disaster. **AWS December 2025 — Kiro-coded change, 13 hours offline in the China region.** Amazon's own Kiro coding agent was tasked with making an environment change and "decided" the cleanest path was to delete and recreate the affected production environment. The agent *had* the deletion tool. Nothing upstream of the loop refused to route a destructive command at a production resource. That is a missing **Orchestration** layer. (First reported by the Financial Times, then widely covered; Amazon disputed the scope and cause, saying it was a Cost Explorer change in one region triggered by user error — the failure mode of an agent calling delete on production stands regardless.) **Cursor + Claude Opus 4.6, April 27, 2026 — PocketOS database gone in 9 seconds.** A Cursor agent encountered a credential mismatch in a staging environment and decided to "fix" it by deleting a Railway volume — the storage that held the application's production database and all backups. The agent had the deletion tool. No rule said "a production database change needs a human to approve it" *before* the model's tool call fired. That is a missing **Governance** layer. (The Register / Tom's Hardware / Tom's-style trades all covered it; Railway's CEO confirmed the post-incident debrief.) Notice what neither failure is: an *Intelligence* failure. The model reasoned correctly about how to do the task it was given. The task was catastrophic, and no layer between the reasoning and the keystroke stopped it. A perfect loop on the Action layer, with nothing above it, is how you get a thirteen-hour outage or a 9-second wipe. The next question checks whether you can spot the missing layer yourself. Then one short read on where this is all heading.46:I[93250,["/_next/static/chunks/2rjtqver_q2gs.js","/_next/static/chunks/0mg7io3h2u8wq.js","/_next/static/chunks/2axu7w_5i8n32.js","/_next/static/chunks/3he5zzx1k7c6r.js","/_next/static/chunks/0h0bavt00zz2v.js","/_next/static/chunks/3s-oul0007que.js"],"IconMark"] 45:Td48,# The shape of the stack — and who wins The five-layer map has one more thing to teach you, and it's the part that matters if you ever build a company instead of a feature. ## The layer count is contested. The shape isn't. This course's five layers are one cut. Other writers slice the same territory into seven, or four, or nine. Don't memorize a number — the count is just an argument. What every version agrees on is the **shape**: - The **bottom** — raw model intelligence, protocols, generic plumbing — **commoditizes**. It standardizes fast, and last year's hand-built version is next year's free library. A moat here is hard to dig. - The **top and middle** — governance, the ecosystem of tools an agent can call, the economics of a task — is where durable advantage collects, because those take long to build and earn slow, real trust. For a builder or PM the strategic question is never "which layer is the model." It is: *given the model under me is a commodity, which layer am I actually defensible at?* ## Why most agents never ship Industry surveys through 2025 and 2026 keep finding the same thing: only a small fraction of organizations have agents running at real scale. The gap is rarely the model — Intelligence is the layer that already works. The gap is the four layers around the loop. Most teams build a strong Action layer, demo it, and then spend the next two quarters building Governance and Orchestration before it is allowed near a customer. When you scope your own agent @@pd-path:#path.has:capstone@@ in the later CLI-agent build@@pd-path:/path.has@@, spend as much time on what the agent *must not* do as on what it can. ## Read wider This lesson compressed a fast-moving conversation. Primary sources, grouped by what they argue: - **The economy thesis** — investor and vendor essays on the agentic AI economy, useful as arguments about where value accrues rather than as settled taxonomies. - **The annual industry survey** — Nathan Benaich, *State of AI Report 2025* (stateof.ai, published October 9, 2025) — the canonical year-in-review; the next edition ships October 2026. - **Primary incident reports** — The Register / Tom's Hardware coverage of the April 27, 2026 PocketOS deletion by Cursor + Claude Opus 4.6; Engadget's reporting on the December 2025 AWS Kiro-coded environment-deletion outage; Jason Lemkin's X thread for the July 2025 Replit production-database wipe; AWS's October 20, 2025 us-east-1 post-incident summary. The first two are the documented version of the Governance / Orchestration failures cited in this chapter. - **Vendor docs as primary sources** — Anthropic's "Building effective agents" post, OpenAI's tool-use docs, the MCP spec. The loop you actually have to build against, in the words of the people building the model. - **Skepticism layer** — read the deployment-rate reporting from Gartner, Stanford HAI's AI Index, and McKinsey's State of AI surveys side by side; absolute numbers diverge but the trend (most orgs piloting, few in production) holds across methodologies. The agentic-AI economy is being narrated in real time by people with money in it — investors, vendors, the labs themselves. Read every layer diagram, the one in this lesson included, as an *argument*. The loop you wrote in this chapter is the fact. The map is just someone's opinion about where the loop is standing.1c:[["$","$L24",null,{"payload":{"catalog":"Multi-step tools — when one tool isn't enough (step 2/9) · agent loops · promptdojo","byPath":{"developer":"Multi-step tools — when one tool isn't enough (step 2/9) · agent loops · promptdojo","customer-service":"Multi-step tools — when one tool isn't enough (step 2/9) · agent loops · promptdojo","operations":"Multi-step tools — when one tool isn't enough (step 2/9) · agent loops · promptdojo"}}}],["$","script",null,{"type":"application/ld+json","dangerouslySetInnerHTML":{"__html":"$25"}}],["$","$L26",null,{"tree":{"toc":{"version":"0.1.0","generatedAt":"1970-01-01T00:00:00.000Z","chapters":[{"number":0,"slug":"before-you-build","title":"from worker to builder","blurb":"stop doing repeated work by hand. this chapter is the map: the builder loop, old workplace deliverables turned into site artifacts, and the reusable tool card. if you are starting here, those moves are the lesson. if you already ran a studio, you already used the loop.","status":"live","lessonCount":10,"stepCount":14,"liveLessonCount":10,"liveStepCount":14,"estMinutes":74,"xpTotal":36},{"number":1,"slug":"read-the-script","title":"read the script — your first code-reading lab","blurb":"ai hands you small scripts. this lab covers the three reads that matter: what the names hold, whether the function actually returns, and which branch of the if-chain really fires. if you are starting here, this labels the reads. if you already ran a studio, you already wrote these.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":21,"xpTotal":78},{"number":2,"slug":"read-the-data-shapes","title":"read the data shapes — the json ai just dumped","blurb":"every api response, config file, and llm reply is lists and dicts nested in a tree. read the shape on sight, walk it with a loop, and catch the wrong-layer bug ai loves to ship.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":23,"xpTotal":80},{"number":3,"slug":"read-the-failure","title":"read the failure — cursor wrote this and crashed","blurb":"when python crashes, it tells you exactly what happened and where. most non-engineers panic at the wall of text — and ai 'fixes' the crash by hiding it. learn to read the failure, then catch only what's worth catching.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":24,"xpTotal":83},{"number":4,"slug":"read-the-pipeline","title":"read the pipeline — files in, api calls out","blurb":"every real script reads a file or calls an api — usually both. learn the with-block, the status-code families, and the retry rules: the seatbelts ai forgets to put on.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":26,"xpTotal":83},{"number":7,"slug":"mutation-and-state","title":"mutation — why your code mysteriously breaks","blurb":"when a list inside a function changes the list outside the function, that's mutation. ai does this constantly without flagging it, and it's the bug class that takes the longest to find.","status":"live","lessonCount":2,"stepCount":9,"liveLessonCount":2,"liveStepCount":9,"estMinutes":10,"xpTotal":32},{"number":13,"slug":"llm-apis","title":"llm apis — talking to claude and openai","blurb":"every ai feature you ship eventually calls a model api. learn the messages pattern, how to read the response, and the four lines ai writes every single time.","status":"live","lessonCount":4,"stepCount":27,"liveLessonCount":4,"liveStepCount":27,"estMinutes":45,"xpTotal":87},{"number":14,"slug":"structured-output","title":"structured output — making the model return json","blurb":"free-form text breaks every pipeline. learn the schema-first pattern ai uses to get reliable json back, validate it with pydantic, and catch the model's lies before they hit prod.","status":"live","lessonCount":3,"stepCount":25,"liveLessonCount":3,"liveStepCount":25,"estMinutes":38,"xpTotal":80},{"number":15,"slug":"mcp","title":"mcp — the model context protocol","blurb":"mcp is the new standard for plugging tools and data sources into ai agents. learn what an mcp server actually is, how claude code lists tools, and why this is replacing one-off integrations everywhere.","status":"live","lessonCount":3,"stepCount":26,"liveLessonCount":3,"liveStepCount":26,"estMinutes":40,"xpTotal":85},{"number":16,"slug":"agent-loops","title":"agent loops — tool_use and the request/tool/respond cycle","blurb":"an agent isn't magic. it's a while loop. learn the actual cycle claude code, cursor, and every other agent uses: model returns tool_use, you run the tool, you send the result back, repeat until end_turn.","status":"live","lessonCount":6,"stepCount":48,"liveLessonCount":6,"liveStepCount":48,"estMinutes":71,"xpTotal":154},{"number":17,"slug":"git-and-github","title":"git and github cli — the ai builder's actual workflow","blurb":"cursor and claude code commit on your behalf. reading those commits — and undoing the bad ones — is your job. learn the three-state model, the commands you'll run every day, and what `gh` does that `git` can't.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":36,"xpTotal":79},{"number":18,"slug":"secrets-and-env","title":"secrets — .env, api keys, and what not to commit","blurb":"ai ships keys to github all the time. learn the .env pattern, why os.getenv is non-negotiable, what to do when a key leaks, and the gitignore lines you need on day one.","status":"live","lessonCount":2,"stepCount":16,"liveLessonCount":2,"liveStepCount":16,"estMinutes":16,"xpTotal":58},{"number":19,"slug":"prompting","title":"prompting ai coding tools","blurb":"an AI coding tool writes and edits with you — Cursor is one brand, Claude Code is another. the difference between a one-shot session and a four-hour spiral is almost always the first prompt.","status":"live","lessonCount":5,"stepCount":36,"liveLessonCount":5,"liveStepCount":36,"estMinutes":59,"xpTotal":118},{"number":20,"slug":"agent-traces","title":"reading agent traces and telemetry","blurb":"when an agent fails, the trace tells you exactly where. learn to read tool calls, tool results, and stop reasons — the json breadcrumbs every agent leaves behind.","status":"live","lessonCount":3,"stepCount":25,"liveLessonCount":3,"liveStepCount":25,"estMinutes":30,"xpTotal":88},{"number":21,"slug":"evals","title":"test the agent before you trust it","blurb":"same input, check the output, pass or fail. don't ship on a demo that felt right.","status":"live","lessonCount":5,"stepCount":34,"liveLessonCount":5,"liveStepCount":34,"estMinutes":57,"xpTotal":112},{"number":22,"slug":"context-and-retrieval","title":"context and retrieval — feeding the model real data","blurb":"rag without the overengineering. chunking, embeddings, vector search, and the small set of patterns that make a model answer from your data instead of its training set.","status":"live","lessonCount":5,"stepCount":44,"liveLessonCount":5,"liveStepCount":44,"estMinutes":75,"xpTotal":144},{"number":23,"slug":"production-tradeoffs","title":"production tradeoffs — cost, latency, quality","blurb":"the three numbers every shipped llm feature lives or dies by. token math, caching, streaming, batching, and the small set of decisions that move the product more than a model swap ever will.","status":"live","lessonCount":4,"stepCount":33,"liveLessonCount":4,"liveStepCount":33,"estMinutes":57,"xpTotal":111},{"number":24,"slug":"debugging-output","title":"debugging broken ai output","blurb":"when the model lies to your customer. the methodology for narrowing down what went wrong, the four most-common breakage classes, and the discipline that separates 'we shipped a fix' from 'we blamed the model and shrugged'.","status":"live","lessonCount":4,"stepCount":34,"liveLessonCount":4,"liveStepCount":34,"estMinutes":61,"xpTotal":112},{"number":25,"slug":"capstone","title":"mid-path build — ship the CLI agent","blurb":"wire it all together. the prompt, the call, the validation, the trace, the eval, the MCP tool. a path checkpoint, not the end of the course: the smallest end-to-end llm feature you could ship to a real user. retrieval and prompt-cache cost work live in later chapters — extend this build with them when you scale past the demo input set.","status":"live","lessonCount":7,"stepCount":62,"liveLessonCount":7,"liveStepCount":62,"estMinutes":100,"xpTotal":219},{"number":26,"slug":"agent-harnesses","title":"agent harnesses — the layer between you and the raw api","blurb":"claude code, cursor, aider, codex cli — they're all the same four layers wrapped around the same model api. learn what those layers are, what each adds, and what you'd build yourself if you had to. when the harness changes the world, the rollback lives in the repo — a runbook a junior can execute at 3am.","status":"live","lessonCount":5,"stepCount":37,"liveLessonCount":5,"liveStepCount":37,"estMinutes":69,"xpTotal":119},{"number":27,"slug":"ai-image-generation","title":"ai image generation — from prompt to production asset","blurb":"the 2026 image model landscape, the prompts that work, and the pipeline that turns one good idea into a hundred ready-to-ship images. nano banana 2 for volume, nano banana pro for fidelity, flux, midjourney, ideogram, gpt-image-2. when each wins and what they cost.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":60,"xpTotal":67},{"number":28,"slug":"ai-video-generation","title":"ai video generation — omni, veo 3.1, sora, and the second wave","blurb":"video is the hardest content type to generate, the most expensive, and the most strategically interesting. learn the 2026 model lineup, the camera-control patterns that separate slop from craft, and the cost math that decides whether your idea is viable.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":50,"xpTotal":61},{"number":29,"slug":"programmatic-design","title":"programmatic design — code-driven video and the new design pipeline","blurb":"ai generates raw assets; code stitches them into something shippable. hyperframes, remotion, claude design — when each tool wins, how they combine, and the data-driven workflows that turn one template into a hundred videos.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":46,"xpTotal":66},{"number":30,"slug":"harness-engineering","title":"harness engineering — the discipline behind the model","blurb":"every coding agent is a model plus a harness. the model is bought; the harness is engineered. learn the craft: how to ratchet rules from failures, fight context rot, design long-horizon loops, wire hooks as enforcement, and read the haas shift that's reshaping what you build vs buy.","status":"live","lessonCount":6,"stepCount":52,"liveLessonCount":6,"liveStepCount":52,"estMinutes":126,"xpTotal":143},{"number":31,"slug":"intro-to-terminal","title":"intro to terminal — the text way to drive your computer","blurb":"the terminal is a text box: type a name, press enter, the computer does it. by the end you can move around your files, make folders, and read files without the mouse. every later tool chapter assumes this.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":26,"xpTotal":50},{"number":32,"slug":"intro-to-claude-cli","title":"intro to claude cli — the door to real building","blurb":"you've used claude in a chat window. the claude cli is the same model with its hands on your actual files. this chapter installs it, signs you in, and runs your first real command. by the end you've watched an ai read, plan, and change things on your machine, and you know when to reach for the cli instead of the chat box.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":32,"xpTotal":48},{"number":33,"slug":"intro-to-codex-cli","title":"intro to openai codex cli — a second tool in the kit","blurb":"you know the claude cli. the openai codex cli does the same job, an ai working in your terminal on your real files, with a different company behind it. this chapter installs it, signs you in, and runs your first command. most of what you already know carries straight over, so this chapter is mostly about what is different and when to reach for which.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":32,"xpTotal":45},{"number":34,"slug":"team-skills","title":"claude skills for teams — playbooks your whole team shares","blurb":"a claude skill is a packaged set of instructions — your team's playbook — that claude loads when it's relevant, so nobody has to re-explain it. this chapter is for people who manage teams. it covers what a skill is, how a team shares and provisions skills, real examples for hr, legal, and ops work, when a skill beats a one-off prompt, and the governance you need before any skill touches real work.","status":"live","lessonCount":4,"stepCount":28,"liveLessonCount":4,"liveStepCount":28,"estMinutes":41,"xpTotal":60},{"number":35,"slug":"dataframes-numpy-pandas","title":"dataframes with numpy and pandas","blurb":"tables are the working surface of applied ml. learn rows, columns, missing values, joins, aggregates, and the dataframe habits ai-generated notebooks assume.","status":"live","lessonCount":6,"stepCount":42,"liveLessonCount":6,"liveStepCount":42,"estMinutes":43,"xpTotal":156},{"number":36,"slug":"sql-for-ml-datasets","title":"sql for ml datasets","blurb":"most training data starts in a database. learn the select, join, filter, aggregate, and leakage traps that decide whether a model is learning signal or nonsense.","status":"live","lessonCount":4,"stepCount":28,"liveLessonCount":4,"liveStepCount":28,"estMinutes":28,"xpTotal":104},{"number":37,"slug":"dataset-pipelines","title":"dataset formats, ingestion, and validation pipelines","blurb":"a dataset is a product surface. build ingestion, validation, partitions, manifests, and checkpoints so the next run is not a mystery.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":38,"slug":"ml-math-and-stats","title":"ml math and statistics that actually show up","blurb":"vectors, probability, distributions, correlation, and uncertainty are not trivia. they are how you read model behavior without worshipping it.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":39,"slug":"supervised-learning-workflows","title":"supervised learning workflows","blurb":"labels, splits, baselines, training, prediction, and evaluation. the supervised workflow is the first complete model loop.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":40,"xpTotal":130},{"number":40,"slug":"unsupervised-learning-and-embeddings","title":"unsupervised learning, embeddings, and recommenders","blurb":"not every useful model has labels. cluster, compare, retrieve, and recommend by turning examples into useful neighborhoods.","status":"live","lessonCount":4,"stepCount":28,"liveLessonCount":4,"liveStepCount":28,"estMinutes":32,"xpTotal":104},{"number":41,"slug":"metrics-and-error-analysis","title":"metrics, slices, and error analysis","blurb":"accuracy is a blunt instrument. learn confusion matrices, precision, recall, thresholds, slices, and failure notes so model quality has evidence.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":42,"slug":"pytorch-tensors-and-autograd","title":"pytorch tensors and autograd","blurb":"read tensor code without flinching. tensors, shapes, broadcasting, gradients, and autograd are the grammar of modern deep learning scripts.","status":"live","lessonCount":4,"stepCount":28,"liveLessonCount":4,"liveStepCount":28,"estMinutes":30,"xpTotal":104},{"number":43,"slug":"training-loops-and-optimizers","title":"training loops, backprop, optimizers, and schedulers","blurb":"the training loop is where models change. learn loss, gradients, optimizer steps, schedules, checkpoints, and the bugs ai ships there.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":44,"slug":"deep-learning-architectures","title":"cnns, transformers, and useful llm internals","blurb":"architecture literacy for builders: convolution, attention, tokens, decoding, kv cache, quantization, and what those choices do to cost and behavior.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":45,"slug":"feature-experiments-registries","title":"feature pipelines, experiment tracking, and registries","blurb":"features, runs, configs, artifacts, and registries are how ml work becomes repeatable instead of a lucky notebook.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":46,"slug":"model-serving-and-mlops","title":"model serving, ci/cd, and mlops","blurb":"serving a model means handling inputs, versions, routes, batch jobs, ci gates, rollback, and production failures deliberately.","status":"live","lessonCount":6,"stepCount":42,"liveLessonCount":6,"liveStepCount":42,"estMinutes":44,"xpTotal":156},{"number":47,"slug":"monitoring-cloud-portfolio","title":"monitoring, drift, cloud scale, and portfolio launch","blurb":"the last mile: logs, drift, alerts, retraining decisions, cloud cost, gpu constraints, architecture docs, demos, and role stories.","status":"live","lessonCount":6,"stepCount":42,"liveLessonCount":6,"liveStepCount":42,"estMinutes":44,"xpTotal":156},{"number":48,"slug":"graphic-design-studio","title":"graphic design studio — prompt, kit, and fifty on-brand assets","blurb":"how a working designer runs image models: prompts written like creative briefs, a brand bible that travels as structured data — with the design tokens that carry it onto screens — and the seeds, reference locks, and QA filters that make fifty generated assets look like one studio shipped them.","status":"live","lessonCount":3,"stepCount":27,"liveLessonCount":3,"liveStepCount":27,"estMinutes":73,"xpTotal":82},{"number":49,"slug":"campaign-studio","title":"campaign studio — one campaign, end to end, with the gate on","blurb":"brief to voice-locked copy variants to batch plan to calendar to the numbers pass. the whole campaign as data your scripts can check — and an approval gate nothing skips. built for the person who owns the send button.","status":"live","lessonCount":3,"stepCount":22,"liveLessonCount":3,"liveStepCount":22,"estMinutes":59,"xpTotal":69},{"number":50,"slug":"support-studio","title":"support studio — the reply drafter you can trust","blurb":"how a working support agent runs AI: a policy pack the drafter can't talk around, a banned-promises linter that holds every draft, golden eval sets that make each macro earn autonomy one intent at a time, and the escalation line where money, law, and anger stay human.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":61,"xpTotal":71},{"number":51,"slug":"copy-studio","title":"copy studio — voice from evidence, variants at scale, claims that clear","blurb":"a client's voice measured from their own published copy, variant batches that pass the voice gate or die with a named reason, and a claim check that blocks the ship button while any fact is unsourced. corrections-log culture, as code.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":66,"xpTotal":72},{"number":52,"slug":"analytics-studio","title":"analytics studio — messy export to defensible memo","blurb":"how a working analyst runs AI: a cleaning log that accounts for every dropped row, sample-first checks that catch the plausible-but-wrong aggregation, a memo where every number carries the address of the cell that made it, SR 11-7-style challenge against the AI's recommendation, and the rerun test — same data, same code, same memo, or it isn't done.","status":"live","lessonCount":3,"stepCount":22,"liveLessonCount":3,"liveStepCount":22,"estMinutes":61,"xpTotal":74},{"number":53,"slug":"delivery-studio","title":"delivery studio — direct the build without becoming the bottleneck","blurb":"how a working delivery lead runs AI builders: specs with acceptance checks written before the build, a review that runs the checks instead of admiring the demo, status roll-ups computed from acceptance records instead of standup vibes, the not-ready call made on evidence you can read out loud to a steering committee, and an hours-and-dollars receipt — before/after on a named function, finance's loaded rate, a named owner, the accepted thing that shipped — small enough to defend in a staff meeting.","status":"live","lessonCount":4,"stepCount":32,"liveLessonCount":4,"liveStepCount":32,"estMinutes":74,"xpTotal":96},{"number":54,"slug":"people-studio","title":"people studio — draft fast, decide human, prove everything","blurb":"how a working hr specialist runs AI: a data firewall that keeps employee records out of prompts by code instead of willpower, recruiting briefs where every claim cites a resume line, policy drafts that cannot publish without a named legal reviewer, and the adverse-action paper trail that survives a discrimination claim.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":56,"xpTotal":72},{"number":55,"slug":"ops-studio","title":"ops studio — tribal knowledge to running automation","blurb":"how a working ops lead gets the process out of one person's head and into automation that can be trusted: SOPs captured as versioned data with the invented steps hunted down, recurring checklists with an explicit auto-run vs named-approval line, exception queues instead of silent failures, a receipt against the export for every number in the weekly report, and a PII proxy that masks member identifiers before the model and restores them only for the named human who sends.","status":"live","lessonCount":4,"stepCount":32,"liveLessonCount":4,"liveStepCount":32,"estMinutes":80,"xpTotal":96},{"number":56,"slug":"legal-studio","title":"legal studio — the cite-check gate and the log that survives","blurb":"how a working lawyer runs AI: a citation-verification drill where every authority gets opened at the source (the fabricated ones always look right), clause extraction that's worthless without a pinpoint, privilege rules written as a routing table instead of remembered under deadline, and the review log you'd want in the record when opposing counsel moves for sanctions.","status":"live","lessonCount":3,"stepCount":22,"liveLessonCount":3,"liveStepCount":22,"estMinutes":60,"xpTotal":69},{"number":57,"slug":"agent-studio","title":"agent studio — ship one agent feature before you stand up","blurb":"one sitting, one agent feature, end to end: a tool schema that makes the dangerous call a validation error, a loop that meters its own spend, golden cases wired before the router works, a trace you debug from instead of vibes, and the ratchet that turns tonight's incident into tomorrow's eval. practice at production tempo.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":66,"xpTotal":73}]},"accessToc":{"chapters":[{"slug":"before-you-build","lessons":[{"slug":"from-worker-to-builder","steps":[{"id":"before-you-build/from-worker-to-builder/s01-01-intro"}]},{"slug":"upgrade-dull-artifacts-into-interactive-sites","steps":[{"id":"before-you-build/upgrade-dull-artifacts-into-interactive-sites/s01-01-intro"},{"id":"before-you-build/upgrade-dull-artifacts-into-interactive-sites/s02-02-build-brief"},{"id":"before-you-build/upgrade-dull-artifacts-into-interactive-sites/s03-03-your-artifact-brief"}]},{"slug":"why-building-matters-now","steps":[{"id":"before-you-build/why-building-matters-now/s01-01-intro"}]},{"slug":"the-builder-loop","steps":[{"id":"before-you-build/the-builder-loop/s01-01-intro"}]},{"slug":"your-first-builder-brief","steps":[{"id":"before-you-build/your-first-builder-brief/s01-01-intro"}]},{"slug":"what-the-ai-is-doing","steps":[{"id":"before-you-build/what-the-ai-is-doing/s01-01-intro"}]},{"slug":"build-your-first-reusable-tool","steps":[{"id":"before-you-build/build-your-first-reusable-tool/s01-01-intro"}]},{"slug":"from-prompt-to-project","steps":[{"id":"before-you-build/from-prompt-to-project/s01-01-intro"}]},{"slug":"why-code-appears-later","steps":[{"id":"before-you-build/why-code-appears-later/s01-01-intro"}]},{"slug":"use-it-again","steps":[{"id":"before-you-build/use-it-again/s01-01-intro"},{"id":"before-you-build/use-it-again/s02-02-which-fix-compounds"},{"id":"before-you-build/use-it-again/s03-03-save-version-two"}]}]},{"slug":"read-the-script","lessons":[{"slug":"names-and-values","steps":[{"id":"read-the-script/names-and-values/s01-01-intro"},{"id":"read-the-script/names-and-values/s02-02-spot-the-type"},{"id":"read-the-script/names-and-values/s03-03-assignment"},{"id":"read-the-script/names-and-values/s04-04-predict-the-score"},{"id":"read-the-script/names-and-values/s05-05-fill-the-name"},{"id":"read-the-script/names-and-values/s06-06-fix-the-typo"},{"id":"read-the-script/names-and-values/s07-07-write-greeting"},{"id":"read-the-script/names-and-values/s08-08-checkpoint"}]},{"slug":"functions-and-the-hallucinated-return","steps":[{"id":"read-the-script/functions-and-the-hallucinated-return/s01-01-intro"},{"id":"read-the-script/functions-and-the-hallucinated-return/s02-02-which-returns"},{"id":"read-the-script/functions-and-the-hallucinated-return/s03-03-return-vs-print"},{"id":"read-the-script/functions-and-the-hallucinated-return/s04-04-predict-the-result"},{"id":"read-the-script/functions-and-the-hallucinated-return/s05-05-fix-the-missing-return"},{"id":"read-the-script/functions-and-the-hallucinated-return/s06-06-fix-print-vs-return"},{"id":"read-the-script/functions-and-the-hallucinated-return/s07-07-write-square"},{"id":"read-the-script/functions-and-the-hallucinated-return/s08-08-checkpoint"}]},{"slug":"decisions-if-elif","steps":[{"id":"read-the-script/decisions-if-elif/s01-01-intro"},{"id":"read-the-script/decisions-if-elif/s02-02-truthiness-quiz"},{"id":"read-the-script/decisions-if-elif/s03-03-eq-vs-is"},{"id":"read-the-script/decisions-if-elif/s04-04-predict-the-empty-list"},{"id":"read-the-script/decisions-if-elif/s05-05-fix-the-zero-bug"},{"id":"read-the-script/decisions-if-elif/s06-06-fix-the-elif-order"},{"id":"read-the-script/decisions-if-elif/s07-07-checkpoint"},{"id":"read-the-script/decisions-if-elif/s08-08-going-deeper"}]}]},{"slug":"read-the-data-shapes","lessons":[{"slug":"the-shapes-ai-dumps-on-you","steps":[{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s01-01-intro"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s02-02-which-is-which"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s03-03-nested-paths"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s04-04-predict-the-deep-key"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s05-05-fill-the-index"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s06-06-fix-the-keyerror"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s07-07-fix-the-list-vs-dict"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s08-08-checkpoint"}]},{"slug":"walking-the-shape","steps":[{"id":"read-the-data-shapes/walking-the-shape/s01-01-intro"},{"id":"read-the-data-shapes/walking-the-shape/s02-02-which-loop-shape"},{"id":"read-the-data-shapes/walking-the-shape/s03-03-walking-the-shape"},{"id":"read-the-data-shapes/walking-the-shape/s04-04-predict-the-list-loop"},{"id":"read-the-data-shapes/walking-the-shape/s05-05-fill-the-loop-variable"},{"id":"read-the-data-shapes/walking-the-shape/s06-06-fix-the-dict-loop"},{"id":"read-the-data-shapes/walking-the-shape/s07-07-write-sum-loop"},{"id":"read-the-data-shapes/walking-the-shape/s08-08-checkpoint"}]},{"slug":"comprehensions-and-the-wrong-layer-bug","steps":[{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s01-01-intro"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s02-02-which-comp-shape"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s03-03-with-filter"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s04-04-predict-the-comp"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s05-05-fix-the-wrong-layer"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s06-06-write-comp"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s07-07-checkpoint"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s08-08-going-deeper"}]}]},{"slug":"read-the-failure","lessons":[{"slug":"read-bottom-up","steps":[{"id":"read-the-failure/read-bottom-up/s01-01-anatomy"},{"id":"read-the-failure/read-bottom-up/s02-02-which-error"},{"id":"read-the-failure/read-bottom-up/s03-03-the-five-fingerprints"},{"id":"read-the-failure/read-bottom-up/s04-04-predict-the-class"},{"id":"read-the-failure/read-bottom-up/s05-05-fix-nameerror"},{"id":"read-the-failure/read-bottom-up/s06-06-fix-attributeerror"},{"id":"read-the-failure/read-bottom-up/s07-07-write-defensive"},{"id":"read-the-failure/read-bottom-up/s08-08-checkpoint"}]},{"slug":"try-except-what-ai-misuses","steps":[{"id":"read-the-failure/try-except-what-ai-misuses/s01-01-intro"},{"id":"read-the-failure/try-except-what-ai-misuses/s02-02-which-block-runs"},{"id":"read-the-failure/try-except-what-ai-misuses/s03-03-naming-the-error"},{"id":"read-the-failure/try-except-what-ai-misuses/s04-04-predict-the-branch"},{"id":"read-the-failure/try-except-what-ai-misuses/s05-05-fill-the-keyword"},{"id":"read-the-failure/try-except-what-ai-misuses/s06-06-fix-the-bare-except"},{"id":"read-the-failure/try-except-what-ai-misuses/s07-07-write-safe-lookup"},{"id":"read-the-failure/try-except-what-ai-misuses/s08-08-checkpoint"}]},{"slug":"fail-loudly","steps":[{"id":"read-the-failure/fail-loudly/s01-01-intro"},{"id":"read-the-failure/fail-loudly/s02-02-which-raise-fires"},{"id":"read-the-failure/fail-loudly/s03-03-predict-the-raise"},{"id":"read-the-failure/fail-loudly/s04-04-fill-the-raise-keyword"},{"id":"read-the-failure/fail-loudly/s05-05-fix-the-silent-swallow"},{"id":"read-the-failure/fail-loudly/s06-06-write-validate-age"},{"id":"read-the-failure/fail-loudly/s07-07-checkpoint"},{"id":"read-the-failure/fail-loudly/s08-08-going-deeper"}]}]},{"slug":"read-the-pipeline","lessons":[{"slug":"files-without-fear","steps":[{"id":"read-the-pipeline/files-without-fear/s01-01-intro"},{"id":"read-the-pipeline/files-without-fear/s02-02-which-mode"},{"id":"read-the-pipeline/files-without-fear/s03-03-pathlib-and-manifests"},{"id":"read-the-pipeline/files-without-fear/s04-04-predict-the-suffix"},{"id":"read-the-pipeline/files-without-fear/s05-05-fill-the-mode"},{"id":"read-the-pipeline/files-without-fear/s06-06-fix-the-wrong-mode"},{"id":"read-the-pipeline/files-without-fear/s07-07-write-manifest-entry"},{"id":"read-the-pipeline/files-without-fear/s08-08-checkpoint"}]},{"slug":"talking-to-apis","steps":[{"id":"read-the-pipeline/talking-to-apis/s01-01-intro"},{"id":"read-the-pipeline/talking-to-apis/s02-02-which-status-family"},{"id":"read-the-pipeline/talking-to-apis/s03-03-parsing-the-json"},{"id":"read-the-pipeline/talking-to-apis/s04-04-predict-the-key"},{"id":"read-the-pipeline/talking-to-apis/s05-05-fill-the-method"},{"id":"read-the-pipeline/talking-to-apis/s06-06-fix-the-status-check"},{"id":"read-the-pipeline/talking-to-apis/s07-07-write-classify-status"},{"id":"read-the-pipeline/talking-to-apis/s08-08-checkpoint"}]},{"slug":"when-the-pipeline-breaks","steps":[{"id":"read-the-pipeline/when-the-pipeline-breaks/s01-01-intro"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s02-02-honor-the-retry-after"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s03-03-predict-the-status-handling"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s04-04-fill-the-raise-for-status"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s05-05-fix-the-missing-check"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s06-06-fix-the-broad-retry"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s07-07-checkpoint"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s08-08-going-deeper"}]}]},{"slug":"mutation-and-state","lessons":[{"slug":"why-it-breaks","steps":[{"id":"mutation-and-state/why-it-breaks/s01-01-intro"},{"id":"mutation-and-state/why-it-breaks/s02-02-which-mutates"},{"id":"mutation-and-state/why-it-breaks/s03-03-shared-reference"},{"id":"mutation-and-state/why-it-breaks/s04-04-fix-the-shared-list"},{"id":"mutation-and-state/why-it-breaks/s05-05-checkpoint"}]},{"slug":"copy-vs-reference","steps":[{"id":"mutation-and-state/copy-vs-reference/s01-01-deep-copy"},{"id":"mutation-and-state/copy-vs-reference/s02-02-predict-the-shallow"},{"id":"mutation-and-state/copy-vs-reference/s03-03-fix-the-shallow-bug"},{"id":"mutation-and-state/copy-vs-reference/s04-04-checkpoint"}]}]},{"slug":"llm-apis","lessons":[{"slug":"before-the-api-what-the-model-is-doing","steps":[{"id":"llm-apis/before-the-api-what-the-model-is-doing/s01-01-intro"}]},{"slug":"the-messages-pattern","steps":[{"id":"llm-apis/the-messages-pattern/s01-01-intro"},{"id":"llm-apis/the-messages-pattern/s02-02-which-role-is-yours"},{"id":"llm-apis/the-messages-pattern/s03-03-reading-the-response"},{"id":"llm-apis/the-messages-pattern/s04-04-predict-the-text"},{"id":"llm-apis/the-messages-pattern/s05-05-fill-the-role"},{"id":"llm-apis/the-messages-pattern/s06-06-fix-the-message-shape"},{"id":"llm-apis/the-messages-pattern/s07-07-fix-the-response-access"},{"id":"llm-apis/the-messages-pattern/s08-08-write-ask-claude"},{"id":"llm-apis/the-messages-pattern/s09-09-checkpoint"}]},{"slug":"reading-the-response","steps":[{"id":"llm-apis/reading-the-response/s01-01-intro"},{"id":"llm-apis/reading-the-response/s02-02-which-block-do-you-read"},{"id":"llm-apis/reading-the-response/s03-03-the-five-stop-reasons"},{"id":"llm-apis/reading-the-response/s04-04-predict-the-text"},{"id":"llm-apis/reading-the-response/s05-05-fill-the-text-extract"},{"id":"llm-apis/reading-the-response/s06-06-fix-the-content-zero"},{"id":"llm-apis/reading-the-response/s07-07-fix-the-stop-assumption"},{"id":"llm-apis/reading-the-response/s08-08-write-the-extractor"},{"id":"llm-apis/reading-the-response/s09-09-checkpoint"}]},{"slug":"the-model-picker","steps":[{"id":"llm-apis/the-model-picker/s01-01-why-model-choice-is-a-product-decision"},{"id":"llm-apis/the-model-picker/s02-02-haiku-territory"},{"id":"llm-apis/the-model-picker/s03-03-which-model-here"},{"id":"llm-apis/the-model-picker/s04-04-sonnet-territory"},{"id":"llm-apis/the-model-picker/s05-05-opus-and-the-thinking-budget"},{"id":"llm-apis/the-model-picker/s06-06-pick-the-model"},{"id":"llm-apis/the-model-picker/s07-07-write-the-model-routing-memo"},{"id":"llm-apis/the-model-picker/s08-08-checkpoint"}]}]},{"slug":"structured-output","lessons":[{"slug":"getting-json-back","steps":[{"id":"structured-output/getting-json-back/s01-01-intro"},{"id":"structured-output/getting-json-back/s02-02-why-validate"},{"id":"structured-output/getting-json-back/s03-03-pydantic-models"},{"id":"structured-output/getting-json-back/s04-04-predict-the-validated"},{"id":"structured-output/getting-json-back/s05-05-fill-the-field-type"},{"id":"structured-output/getting-json-back/s06-06-fix-the-required-field"},{"id":"structured-output/getting-json-back/s07-07-fix-the-type-mismatch"},{"id":"structured-output/getting-json-back/s08-08-write-parse-extraction"},{"id":"structured-output/getting-json-back/s09-09-checkpoint"}]},{"slug":"schemas-eat-prompts","steps":[{"id":"structured-output/schemas-eat-prompts/s01-01-the-trust-boundary"},{"id":"structured-output/schemas-eat-prompts/s02-02-three-shipped-breakages"},{"id":"structured-output/schemas-eat-prompts/s03-03-which-boundary-broke"},{"id":"structured-output/schemas-eat-prompts/s04-04-schema-as-prompt-replacement"},{"id":"structured-output/schemas-eat-prompts/s05-05-where-validation-belongs"},{"id":"structured-output/schemas-eat-prompts/s06-06-the-pydantic-migration-as-war-story"},{"id":"structured-output/schemas-eat-prompts/s07-07-write-the-boundary-audit"},{"id":"structured-output/schemas-eat-prompts/s08-08-checkpoint"}]},{"slug":"dataset-schema-validation","steps":[{"id":"structured-output/dataset-schema-validation/s01-01-validate-at-the-boundary"},{"id":"structured-output/dataset-schema-validation/s02-02-the-silent-drop"},{"id":"structured-output/dataset-schema-validation/s03-03-predict-the-missing-fields"},{"id":"structured-output/dataset-schema-validation/s04-04-fill-the-membership-test"},{"id":"structured-output/dataset-schema-validation/s05-05-fix-the-range-check"},{"id":"structured-output/dataset-schema-validation/s06-06-write-validate-batch"},{"id":"structured-output/dataset-schema-validation/s07-07-claims-with-receipts"},{"id":"structured-output/dataset-schema-validation/s08-08-checkpoint"}]}]},{"slug":"mcp","lessons":[{"slug":"what-mcp-is","steps":[{"id":"mcp/what-mcp-is/s01-01-intro"},{"id":"mcp/what-mcp-is/s02-02-which-side-is-the-server"},{"id":"mcp/what-mcp-is/s03-03-the-tool-list"},{"id":"mcp/what-mcp-is/s04-04-predict-the-tool-result"},{"id":"mcp/what-mcp-is/s05-05-fill-the-tool-name"},{"id":"mcp/what-mcp-is/s06-06-fix-the-missing-argument"},{"id":"mcp/what-mcp-is/s07-07-fix-the-result-shape"},{"id":"mcp/what-mcp-is/s08-08-write-call-tool"},{"id":"mcp/what-mcp-is/s09-09-checkpoint"}]},{"slug":"writing-a-tiny-mcp-server","steps":[{"id":"mcp/writing-a-tiny-mcp-server/s01-01-intro"},{"id":"mcp/writing-a-tiny-mcp-server/s02-02-which-message-is-call-tool"},{"id":"mcp/writing-a-tiny-mcp-server/s03-03-the-tool-registry"},{"id":"mcp/writing-a-tiny-mcp-server/s04-04-predict-the-list-tools-output"},{"id":"mcp/writing-a-tiny-mcp-server/s05-05-fill-the-input-schema"},{"id":"mcp/writing-a-tiny-mcp-server/s06-06-fix-the-missing-error-response"},{"id":"mcp/writing-a-tiny-mcp-server/s07-07-fix-the-validation-gap"},{"id":"mcp/writing-a-tiny-mcp-server/s08-08-write-the-call-tool-dispatcher"},{"id":"mcp/writing-a-tiny-mcp-server/s09-09-checkpoint"}]},{"slug":"why-mcp-won","steps":[{"id":"mcp/why-mcp-won/s01-01-the-graveyard-of-tool-protocols"},{"id":"mcp/why-mcp-won/s02-02-what-openai-plugins-got-wrong"},{"id":"mcp/why-mcp-won/s03-03-which-failure-mode"},{"id":"mcp/why-mcp-won/s04-04-the-mcp-bet"},{"id":"mcp/why-mcp-won/s05-05-the-network-effect"},{"id":"mcp/why-mcp-won/s06-06-spot-the-mcp-wedge"},{"id":"mcp/why-mcp-won/s07-07-write-the-protocol-postmortem"},{"id":"mcp/why-mcp-won/s08-08-checkpoint"}]}]},{"slug":"agent-loops","lessons":[{"slug":"the-loop","steps":[{"id":"agent-loops/the-loop/s01-01-intro"},{"id":"agent-loops/the-loop/s02-02-which-stop-reason"},{"id":"agent-loops/the-loop/s03-03-tool-result-shape"},{"id":"agent-loops/the-loop/s04-04-predict-the-next-call"},{"id":"agent-loops/the-loop/s05-05-fill-the-stop-condition"},{"id":"agent-loops/the-loop/s06-06-fix-the-tool-id"},{"id":"agent-loops/the-loop/s07-07-fix-the-loop-exit"},{"id":"agent-loops/the-loop/s08-08-write-the-loop"},{"id":"agent-loops/the-loop/s09-09-checkpoint"}]},{"slug":"multi-step-tools","steps":[{"id":"agent-loops/multi-step-tools/s01-01-intro"},{"id":"agent-loops/multi-step-tools/s02-02-which-tool-runs"},{"id":"agent-loops/multi-step-tools/s03-03-tool-registry"},{"id":"agent-loops/multi-step-tools/s04-04-predict-the-chain"},{"id":"agent-loops/multi-step-tools/s05-05-fill-the-dispatch"},{"id":"agent-loops/multi-step-tools/s06-06-fix-the-missing-tool"},{"id":"agent-loops/multi-step-tools/s07-07-fix-the-parallel-ids"},{"id":"agent-loops/multi-step-tools/s08-08-write-multi-tool-agent"},{"id":"agent-loops/multi-step-tools/s09-09-checkpoint"}]},{"slug":"routing","steps":[{"id":"agent-loops/routing/s01-01-intro"},{"id":"agent-loops/routing/s02-02-when-to-route"},{"id":"agent-loops/routing/s03-03-classifier-output"},{"id":"agent-loops/routing/s04-04-predict-the-route"},{"id":"agent-loops/routing/s05-05-fill-the-dispatch"},{"id":"agent-loops/routing/s06-06-fix-the-fallback"},{"id":"agent-loops/routing/s07-07-fix-the-loose-classifier"},{"id":"agent-loops/routing/s08-08-write-the-router"},{"id":"agent-loops/routing/s09-09-checkpoint"}]},{"slug":"evaluator-optimizer","steps":[{"id":"agent-loops/evaluator-optimizer/s01-01-intro"},{"id":"agent-loops/evaluator-optimizer/s02-02-when-it-earns-its-keep"},{"id":"agent-loops/evaluator-optimizer/s03-03-judge-output-shape"},{"id":"agent-loops/evaluator-optimizer/s04-04-predict-the-iterations"},{"id":"agent-loops/evaluator-optimizer/s05-05-fill-the-exit"},{"id":"agent-loops/evaluator-optimizer/s06-06-fix-the-judge-parsing"},{"id":"agent-loops/evaluator-optimizer/s07-07-fix-the-missing-cap"},{"id":"agent-loops/evaluator-optimizer/s08-08-write-the-evaluator-optimizer"},{"id":"agent-loops/evaluator-optimizer/s09-09-checkpoint"}]},{"slug":"why-every-framework-is-thirty-lines","steps":[{"id":"agent-loops/why-every-framework-is-thirty-lines/s01-01-the-frameworks-converged"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s02-02-langgraph-walkthrough"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s03-03-the-work-is-a-graph"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s04-04-which-framework-fits"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s05-05-vercel-ai-sdk-walkthrough"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s06-06-the-five-canonical-patterns"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s07-07-when-to-stay-framework-free"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s08-08-write-the-tradeoff-memo"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s09-09-checkpoint"}]},{"slug":"the-industry-map","steps":[{"id":"agent-loops/the-industry-map/s01-01-the-five-layers"},{"id":"agent-loops/the-industry-map/s02-02-which-layer-failed"},{"id":"agent-loops/the-industry-map/s03-03-the-shape-of-the-stack"}]}]},{"slug":"git-and-github","lessons":[{"slug":"the-three-states","steps":[{"id":"git-and-github/the-three-states/s01-01-intro"},{"id":"git-and-github/the-three-states/s02-02-which-command-stages"},{"id":"git-and-github/the-three-states/s03-03-status-and-diff"},{"id":"git-and-github/the-three-states/s04-04-predict-the-status"},{"id":"git-and-github/the-three-states/s05-05-fill-the-command"},{"id":"git-and-github/the-three-states/s06-06-fix-the-staged-files"},{"id":"git-and-github/the-three-states/s07-07-fix-the-pr-command"},{"id":"git-and-github/the-three-states/s08-08-write-status-summary"},{"id":"git-and-github/the-three-states/s09-09-checkpoint"}]},{"slug":"three-git-disasters-ai-shipped","steps":[{"id":"git-and-github/three-git-disasters-ai-shipped/s01-01-why-postmortems-teach-better"},{"id":"git-and-github/three-git-disasters-ai-shipped/s02-02-uber-2016-credentials-in-repo"},{"id":"git-and-github/three-git-disasters-ai-shipped/s03-03-which-control-would-have-caught-it"},{"id":"git-and-github/three-git-disasters-ai-shipped/s04-04-the-samsung-paste"},{"id":"git-and-github/three-git-disasters-ai-shipped/s05-05-the-cursor-git-add-dot-default"},{"id":"git-and-github/three-git-disasters-ai-shipped/s06-06-spot-the-failure-mode"},{"id":"git-and-github/three-git-disasters-ai-shipped/s07-07-write-your-pre-commit-checklist"},{"id":"git-and-github/three-git-disasters-ai-shipped/s08-08-checkpoint"}]},{"slug":"github-actions-for-model-checks","steps":[{"id":"git-and-github/github-actions-for-model-checks/s01-01-intro"},{"id":"git-and-github/github-actions-for-model-checks/s02-02-green-but-wrong"},{"id":"git-and-github/github-actions-for-model-checks/s03-03-predict-the-run"},{"id":"git-and-github/github-actions-for-model-checks/s04-04-fill-the-trigger"},{"id":"git-and-github/github-actions-for-model-checks/s05-05-fix-the-eval-gate"},{"id":"git-and-github/github-actions-for-model-checks/s06-06-write-run-job"},{"id":"git-and-github/github-actions-for-model-checks/s07-07-checkpoint"}]}]},{"slug":"secrets-and-env","lessons":[{"slug":"keeping-keys-safe","steps":[{"id":"secrets-and-env/keeping-keys-safe/s01-01-intro"},{"id":"secrets-and-env/keeping-keys-safe/s02-02-which-line-leaks"},{"id":"secrets-and-env/keeping-keys-safe/s03-03-loading-env-vars"},{"id":"secrets-and-env/keeping-keys-safe/s04-04-predict-the-key"},{"id":"secrets-and-env/keeping-keys-safe/s05-05-fill-the-getenv"},{"id":"secrets-and-env/keeping-keys-safe/s06-06-fix-the-hardcoded-key"},{"id":"secrets-and-env/keeping-keys-safe/s07-07-fix-the-missing-default"},{"id":"secrets-and-env/keeping-keys-safe/s08-08-write-load-config"},{"id":"secrets-and-env/keeping-keys-safe/s09-09-checkpoint"}]},{"slug":"ml-service-secrets","steps":[{"id":"secrets-and-env/ml-service-secrets/s01-01-intro"},{"id":"secrets-and-env/ml-service-secrets/s02-02-which-log-line-leaks"},{"id":"secrets-and-env/ml-service-secrets/s03-03-predict-the-scan"},{"id":"secrets-and-env/ml-service-secrets/s04-04-fill-the-lookup"},{"id":"secrets-and-env/ml-service-secrets/s05-05-fix-the-logged-key"},{"id":"secrets-and-env/ml-service-secrets/s06-06-write-scan-logs"},{"id":"secrets-and-env/ml-service-secrets/s07-07-checkpoint"}]}]},{"slug":"prompting","lessons":[{"slug":"builder-briefs-before-prompts","steps":[{"id":"prompting/builder-briefs-before-prompts/s01-01-intro"}]},{"slug":"the-prompt-craft","steps":[{"id":"prompting/the-prompt-craft/s01-01-intro"},{"id":"prompting/the-prompt-craft/s02-02-which-prompt-is-better"},{"id":"prompting/the-prompt-craft/s03-03-context-rot"},{"id":"prompting/the-prompt-craft/s04-04-predict-the-output"},{"id":"prompting/the-prompt-craft/s05-05-fill-the-constraint"},{"id":"prompting/the-prompt-craft/s06-06-fix-the-vague-prompt"},{"id":"prompting/the-prompt-craft/s07-07-fix-the-bloated-context"},{"id":"prompting/the-prompt-craft/s08-08-write-the-prompt"},{"id":"prompting/the-prompt-craft/s09-09-checkpoint"}]},{"slug":"few-shot-and-reasoning","steps":[{"id":"prompting/few-shot-and-reasoning/s01-01-intro"},{"id":"prompting/few-shot-and-reasoning/s02-02-when-few-shot-helps"},{"id":"prompting/few-shot-and-reasoning/s03-03-the-cot-trap"},{"id":"prompting/few-shot-and-reasoning/s04-04-predict-the-format-lock"},{"id":"prompting/few-shot-and-reasoning/s05-05-fill-the-example-format"},{"id":"prompting/few-shot-and-reasoning/s06-06-fix-the-cot-on-reasoning"},{"id":"prompting/few-shot-and-reasoning/s07-07-fix-the-format-mismatch"},{"id":"prompting/few-shot-and-reasoning/s08-08-write-the-prompt-builder"},{"id":"prompting/few-shot-and-reasoning/s09-09-checkpoint"}]},{"slug":"agent-config-files","steps":[{"id":"prompting/agent-config-files/s01-01-intro"},{"id":"prompting/agent-config-files/s02-02-which-belongs-in-the-file"},{"id":"prompting/agent-config-files/s03-03-anatomy-of-a-good-file"},{"id":"prompting/agent-config-files/s04-04-predict-which-rule-fires"},{"id":"prompting/agent-config-files/s05-05-fill-the-rule-block"},{"id":"prompting/agent-config-files/s06-06-fix-the-too-vague"},{"id":"prompting/agent-config-files/s07-07-fix-the-stale-pointer"},{"id":"prompting/agent-config-files/s08-08-write-a-claude-md"},{"id":"prompting/agent-config-files/s09-09-checkpoint"}]},{"slug":"what-aged-in-prompting","steps":[{"id":"prompting/what-aged-in-prompting/s01-01-prompting-as-a-moving-target"},{"id":"prompting/what-aged-in-prompting/s02-02-cot-then-vs-now"},{"id":"prompting/what-aged-in-prompting/s03-03-which-still-works"},{"id":"prompting/what-aged-in-prompting/s04-04-the-system-prompt-arms-race"},{"id":"prompting/what-aged-in-prompting/s05-05-spot-the-stale-technique"},{"id":"prompting/what-aged-in-prompting/s06-06-the-prompt-engineer-debate"},{"id":"prompting/what-aged-in-prompting/s07-07-write-the-cheatsheet-update"},{"id":"prompting/what-aged-in-prompting/s08-08-checkpoint"}]}]},{"slug":"agent-traces","lessons":[{"slug":"reading-the-trace","steps":[{"id":"agent-traces/reading-the-trace/s01-01-intro"},{"id":"agent-traces/reading-the-trace/s02-02-which-turn-failed"},{"id":"agent-traces/reading-the-trace/s03-03-the-stop-reasons"},{"id":"agent-traces/reading-the-trace/s04-04-predict-the-stop"},{"id":"agent-traces/reading-the-trace/s05-05-fill-the-tool-name"},{"id":"agent-traces/reading-the-trace/s06-06-fix-the-loop"},{"id":"agent-traces/reading-the-trace/s07-07-fix-the-arg-shape"},{"id":"agent-traces/reading-the-trace/s08-08-write-the-summarizer"},{"id":"agent-traces/reading-the-trace/s09-09-checkpoint"}]},{"slug":"trace-driven-debugging","steps":[{"id":"agent-traces/trace-driven-debugging/s01-01-intro"},{"id":"agent-traces/trace-driven-debugging/s02-02-which-pattern-is-this"},{"id":"agent-traces/trace-driven-debugging/s03-03-the-loop-signature"},{"id":"agent-traces/trace-driven-debugging/s04-04-predict-the-call-count"},{"id":"agent-traces/trace-driven-debugging/s05-05-fill-the-detector"},{"id":"agent-traces/trace-driven-debugging/s06-06-fix-the-tool-routing"},{"id":"agent-traces/trace-driven-debugging/s07-07-fix-the-prompt-bloat"},{"id":"agent-traces/trace-driven-debugging/s08-08-write-the-trace-summary"},{"id":"agent-traces/trace-driven-debugging/s09-09-checkpoint"}]},{"slug":"prediction-ids-and-inference-logs","steps":[{"id":"agent-traces/prediction-ids-and-inference-logs/s01-01-intro"},{"id":"agent-traces/prediction-ids-and-inference-logs/s02-02-pick-the-log-line"},{"id":"agent-traces/prediction-ids-and-inference-logs/s03-03-predict-the-ids"},{"id":"agent-traces/prediction-ids-and-inference-logs/s04-04-fill-the-join-key"},{"id":"agent-traces/prediction-ids-and-inference-logs/s05-05-fix-the-join"},{"id":"agent-traces/prediction-ids-and-inference-logs/s06-06-write-the-logger"},{"id":"agent-traces/prediction-ids-and-inference-logs/s07-07-checkpoint"}]}]},{"slug":"evals","lessons":[{"slug":"check-before-you-trust","steps":[{"id":"evals/check-before-you-trust/s01-01-intro"}]},{"slug":"writing-evals","steps":[{"id":"evals/writing-evals/s01-01-intro"},{"id":"evals/writing-evals/s02-02-which-eval-passes"},{"id":"evals/writing-evals/s03-03-eval-patterns"},{"id":"evals/writing-evals/s04-04-predict-the-pass-rate"},{"id":"evals/writing-evals/s05-05-fill-the-assertion"},{"id":"evals/writing-evals/s06-06-fix-the-flaky-eval"},{"id":"evals/writing-evals/s07-07-fix-the-overstrict-eval"},{"id":"evals/writing-evals/s08-08-write-the-suite"},{"id":"evals/writing-evals/s09-09-checkpoint"}]},{"slug":"llm-as-judge","steps":[{"id":"evals/llm-as-judge/s01-01-intro"},{"id":"evals/llm-as-judge/s02-02-pairwise-or-rubric"},{"id":"evals/llm-as-judge/s03-03-the-four-biases"},{"id":"evals/llm-as-judge/s04-04-predict-the-bias"},{"id":"evals/llm-as-judge/s05-05-fill-the-rubric-call"},{"id":"evals/llm-as-judge/s06-06-fix-the-likert-trap"},{"id":"evals/llm-as-judge/s07-07-fix-the-position-bias"},{"id":"evals/llm-as-judge/s08-08-write-the-rubric-judge"},{"id":"evals/llm-as-judge/s09-09-checkpoint"}]},{"slug":"the-rise-of-evals-as-a-discipline","steps":[{"id":"evals/the-rise-of-evals-as-a-discipline/s01-01-the-prompt-engineering-era"},{"id":"evals/the-rise-of-evals-as-a-discipline/s02-02-the-eval-turn"},{"id":"evals/the-rise-of-evals-as-a-discipline/s03-03-which-company-survives-a-model-swap"},{"id":"evals/the-rise-of-evals-as-a-discipline/s04-04-case-studies"},{"id":"evals/the-rise-of-evals-as-a-discipline/s05-05-eval-first-vs-prompt-first"},{"id":"evals/the-rise-of-evals-as-a-discipline/s06-06-why-the-eval-engineer-eats-the-prompt-engineer"},{"id":"evals/the-rise-of-evals-as-a-discipline/s07-07-write-the-eval-readiness-audit"},{"id":"evals/the-rise-of-evals-as-a-discipline/s08-08-checkpoint"}]},{"slug":"model-validation-gates","steps":[{"id":"evals/model-validation-gates/s01-01-intro"},{"id":"evals/model-validation-gates/s02-02-spot-the-theater"},{"id":"evals/model-validation-gates/s03-03-predict-the-decision"},{"id":"evals/model-validation-gates/s04-04-fill-the-regression-check"},{"id":"evals/model-validation-gates/s05-05-fix-the-theater-gate"},{"id":"evals/model-validation-gates/s06-06-write-the-gate-runner"},{"id":"evals/model-validation-gates/s07-07-checkpoint"}]}]},{"slug":"context-and-retrieval","lessons":[{"slug":"chunking-that-respects-structure","steps":[{"id":"context-and-retrieval/chunking-that-respects-structure/s01-01-intro"},{"id":"context-and-retrieval/chunking-that-respects-structure/s02-02-which-content-shape-breaks"},{"id":"context-and-retrieval/chunking-that-respects-structure/s03-03-the-recursive-splitter"},{"id":"context-and-retrieval/chunking-that-respects-structure/s04-04-predict-the-shred"},{"id":"context-and-retrieval/chunking-that-respects-structure/s05-05-fill-the-separator-list"},{"id":"context-and-retrieval/chunking-that-respects-structure/s06-06-fix-the-mid-sentence-cut"},{"id":"context-and-retrieval/chunking-that-respects-structure/s07-07-fix-the-missing-overlap"},{"id":"context-and-retrieval/chunking-that-respects-structure/s08-08-write-the-recursive-splitter"},{"id":"context-and-retrieval/chunking-that-respects-structure/s09-09-checkpoint"}]},{"slug":"embedding-that-fits-the-budget","steps":[{"id":"context-and-retrieval/embedding-that-fits-the-budget/s01-01-intro"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s02-02-which-model-fits"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s03-03-the-vector-space"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s04-04-predict-the-closest"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s05-05-fill-the-cosine-formula"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s06-06-fix-the-wrong-embedding-model"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s07-07-fix-the-missing-normalization"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s08-08-write-the-similarity-ranker"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s09-09-checkpoint"}]},{"slug":"retrieval-that-finds-the-right-thing","steps":[{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s01-01-intro"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s02-02-which-result-is-good"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s03-03-the-distance-threshold"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s04-04-predict-the-top-k"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s05-05-fill-the-cutoff"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s06-06-fix-the-no-threshold"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s07-07-fix-the-dup-results"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s08-08-write-the-ranker"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s09-09-checkpoint"}]},{"slug":"rag-vs-long-context-vs-fine-tune","steps":[{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s01-01-the-three-way-fork"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s02-02-rag-when-you-need-freshness"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s03-03-which-fork-for-this-scenario"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s04-04-long-context-when-the-corpus-fits"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s05-05-fine-tune-when-style-or-format-is-the-product"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s06-06-which-fork-dies-first"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s07-07-graph-vector-or-both"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s08-08-the-rubric"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s09-09-write-the-fork-decision"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s10-10-checkpoint"}]},{"slug":"retrieval-metrics-and-vector-db-shape","steps":[{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s01-01-the-eval-set"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s02-02-filter-or-search-first"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s03-03-predict-recall-precision"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s04-04-fill-the-reciprocal"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s05-05-fix-the-late-filter"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s06-06-write-recall-at-k"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s07-07-checkpoint"}]}]},{"slug":"production-tradeoffs","lessons":[{"slug":"prompt-caching-correctly","steps":[{"id":"production-tradeoffs/prompt-caching-correctly/s01-01-intro"},{"id":"production-tradeoffs/prompt-caching-correctly/s02-02-which-piece-caches"},{"id":"production-tradeoffs/prompt-caching-correctly/s03-03-the-cache-breakpoints"},{"id":"production-tradeoffs/prompt-caching-correctly/s04-04-predict-the-savings"},{"id":"production-tradeoffs/prompt-caching-correctly/s05-05-fill-the-cache-control"},{"id":"production-tradeoffs/prompt-caching-correctly/s06-06-fix-the-cache-buster"},{"id":"production-tradeoffs/prompt-caching-correctly/s07-07-fix-the-ttl-default"},{"id":"production-tradeoffs/prompt-caching-correctly/s08-08-write-the-cost-estimator"},{"id":"production-tradeoffs/prompt-caching-correctly/s09-09-checkpoint"}]},{"slug":"read-the-token-bill","steps":[{"id":"production-tradeoffs/read-the-token-bill/s01-01-intro"},{"id":"production-tradeoffs/read-the-token-bill/s02-02-which-prompt-costs-more"},{"id":"production-tradeoffs/read-the-token-bill/s03-03-the-shape-of-a-bill"},{"id":"production-tradeoffs/read-the-token-bill/s04-04-predict-the-monthly-spend"},{"id":"production-tradeoffs/read-the-token-bill/s05-05-fill-the-cost-formula"},{"id":"production-tradeoffs/read-the-token-bill/s06-06-fix-the-pricing-ratio-error"},{"id":"production-tradeoffs/read-the-token-bill/s07-07-fix-the-units-error"},{"id":"production-tradeoffs/read-the-token-bill/s08-08-write-the-cost-estimator"},{"id":"production-tradeoffs/read-the-token-bill/s09-09-checkpoint"}]},{"slug":"the-model-price-war","steps":[{"id":"production-tradeoffs/the-model-price-war/s01-01-the-price-curve"},{"id":"production-tradeoffs/the-model-price-war/s02-02-three-bets-and-how-they-aged"},{"id":"production-tradeoffs/the-model-price-war/s03-03-which-startup-survived"},{"id":"production-tradeoffs/the-model-price-war/s04-04-the-incumbent-trap"},{"id":"production-tradeoffs/the-model-price-war/s05-05-features-now-viable"},{"id":"production-tradeoffs/the-model-price-war/s06-06-pricing-for-a-war-you-cant-predict"},{"id":"production-tradeoffs/the-model-price-war/s07-07-write-the-cost-model"},{"id":"production-tradeoffs/the-model-price-war/s08-08-checkpoint"}]},{"slug":"batch-realtime-costs","steps":[{"id":"production-tradeoffs/batch-realtime-costs/s01-01-who-is-waiting"},{"id":"production-tradeoffs/batch-realtime-costs/s02-02-which-lane"},{"id":"production-tradeoffs/batch-realtime-costs/s03-03-predict-the-bill"},{"id":"production-tradeoffs/batch-realtime-costs/s04-04-fill-the-miss-rate"},{"id":"production-tradeoffs/batch-realtime-costs/s05-05-fix-the-free-discount"},{"id":"production-tradeoffs/batch-realtime-costs/s06-06-write-monthly-bill"},{"id":"production-tradeoffs/batch-realtime-costs/s07-07-checkpoint"}]}]},{"slug":"debugging-output","lessons":[{"slug":"read-the-trace-not-the-chat","steps":[{"id":"debugging-output/read-the-trace-not-the-chat/s01-01-intro"},{"id":"debugging-output/read-the-trace-not-the-chat/s02-02-where-to-look-first"},{"id":"debugging-output/read-the-trace-not-the-chat/s03-03-the-four-failure-classes"},{"id":"debugging-output/read-the-trace-not-the-chat/s04-04-predict-the-bad-turn"},{"id":"debugging-output/read-the-trace-not-the-chat/s05-05-fill-the-find-bad-turn"},{"id":"debugging-output/read-the-trace-not-the-chat/s06-06-fix-the-fix-the-symptom"},{"id":"debugging-output/read-the-trace-not-the-chat/s07-07-fix-the-blame-the-model"},{"id":"debugging-output/read-the-trace-not-the-chat/s08-08-write-the-trace-summary"},{"id":"debugging-output/read-the-trace-not-the-chat/s09-09-checkpoint"}]},{"slug":"the-four-breakage-classes","steps":[{"id":"debugging-output/the-four-breakage-classes/s01-01-intro"},{"id":"debugging-output/the-four-breakage-classes/s02-02-classify-the-refund-bug"},{"id":"debugging-output/the-four-breakage-classes/s03-03-the-trace-anatomy"},{"id":"debugging-output/the-four-breakage-classes/s04-04-predict-the-class"},{"id":"debugging-output/the-four-breakage-classes/s05-05-fill-the-class-name"},{"id":"debugging-output/the-four-breakage-classes/s06-06-fix-the-misclassification"},{"id":"debugging-output/the-four-breakage-classes/s07-07-fix-the-parse-mangle"},{"id":"debugging-output/the-four-breakage-classes/s08-08-write-the-classifier"},{"id":"debugging-output/the-four-breakage-classes/s09-09-checkpoint"}]},{"slug":"five-real-postmortems","steps":[{"id":"debugging-output/five-real-postmortems/s01-01-the-postmortem-template"},{"id":"debugging-output/five-real-postmortems/s02-02-air-canada-refund-lawsuit"},{"id":"debugging-output/five-real-postmortems/s03-03-dpd-and-nyc-mycity"},{"id":"debugging-output/five-real-postmortems/s04-04-classify-the-five"},{"id":"debugging-output/five-real-postmortems/s05-05-recruiter-json-mangle"},{"id":"debugging-output/five-real-postmortems/s06-06-glean-style-retrieval-miss"},{"id":"debugging-output/five-real-postmortems/s07-07-which-fix-catches-the-most"},{"id":"debugging-output/five-real-postmortems/s08-08-write-the-postmortem"},{"id":"debugging-output/five-real-postmortems/s09-09-checkpoint"}]},{"slug":"prediction-failure-triage","steps":[{"id":"debugging-output/prediction-failure-triage/s01-01-reproduce-classify-route"},{"id":"debugging-output/prediction-failure-triage/s02-02-first-move"},{"id":"debugging-output/prediction-failure-triage/s03-03-predict-the-route"},{"id":"debugging-output/prediction-failure-triage/s04-04-fill-the-owners"},{"id":"debugging-output/prediction-failure-triage/s05-05-fix-the-paraphrase-repro"},{"id":"debugging-output/prediction-failure-triage/s06-06-write-the-router"},{"id":"debugging-output/prediction-failure-triage/s07-07-checkpoint"}]}]},{"slug":"capstone","lessons":[{"slug":"pick-a-project-that-ships","steps":[{"id":"capstone/pick-a-project-that-ships/s01-01-the-builders-graveyard"},{"id":"capstone/pick-a-project-that-ships/s02-02-which-project-survives-contact-with-reality"},{"id":"capstone/pick-a-project-that-ships/s03-03-the-three-shapes-that-ship"},{"id":"capstone/pick-a-project-that-ships/s04-04-the-wedge-checklist"},{"id":"capstone/pick-a-project-that-ships/s05-05-spot-the-missing-wedge"},{"id":"capstone/pick-a-project-that-ships/s06-06-scope-down-the-capstone"},{"id":"capstone/pick-a-project-that-ships/s07-07-checkpoint"}]},{"slug":"build-a-cli-agent","steps":[{"id":"capstone/build-a-cli-agent/s01-01-intro"},{"id":"capstone/build-a-cli-agent/s02-02-which-piece-is-missing"},{"id":"capstone/build-a-cli-agent/s03-03-the-tool-loop"},{"id":"capstone/build-a-cli-agent/s04-04-predict-the-final-answer"},{"id":"capstone/build-a-cli-agent/s05-05-fill-the-stop-check"},{"id":"capstone/build-a-cli-agent/s06-06-fix-the-tool-dispatch"},{"id":"capstone/build-a-cli-agent/s07-07-fix-the-message-append"},{"id":"capstone/build-a-cli-agent/s08-08-the-logging-pattern"},{"id":"capstone/build-a-cli-agent/s09-09-write-the-tool-runner"},{"id":"capstone/build-a-cli-agent/s10-10-fix-the-loop-exit"},{"id":"capstone/build-a-cli-agent/s11-11-write-the-agent"},{"id":"capstone/build-a-cli-agent/s12-12-checkpoint"}]},{"slug":"wire-the-real-model","steps":[{"id":"capstone/wire-the-real-model/s01-01-intro"},{"id":"capstone/wire-the-real-model/s02-02-where-the-key-lives"},{"id":"capstone/wire-the-real-model/s03-03-the-sdk-response-shape"},{"id":"capstone/wire-the-real-model/s04-04-predict-the-block-type"},{"id":"capstone/wire-the-real-model/s05-05-fill-the-key-load"},{"id":"capstone/wire-the-real-model/s06-06-fix-the-hardcoded-key"},{"id":"capstone/wire-the-real-model/s07-07-fix-the-block-access"},{"id":"capstone/wire-the-real-model/s08-08-write-the-shape-adapter"},{"id":"capstone/wire-the-real-model/s09-09-checkpoint"}]},{"slug":"validate-tool-inputs","steps":[{"id":"capstone/validate-tool-inputs/s01-01-intro"},{"id":"capstone/validate-tool-inputs/s02-02-which-input-needs-validation"},{"id":"capstone/validate-tool-inputs/s03-03-the-validator-surface"},{"id":"capstone/validate-tool-inputs/s04-04-predict-the-error"},{"id":"capstone/validate-tool-inputs/s05-05-fill-the-validate-call"},{"id":"capstone/validate-tool-inputs/s06-06-fix-the-untyped-tool"},{"id":"capstone/validate-tool-inputs/s07-07-fix-the-error-leak"},{"id":"capstone/validate-tool-inputs/s08-08-write-the-validator"},{"id":"capstone/validate-tool-inputs/s09-09-checkpoint"}]},{"slug":"add-evals-and-traces","steps":[{"id":"capstone/add-evals-and-traces/s01-01-intro"},{"id":"capstone/add-evals-and-traces/s02-02-which-eval-fits"},{"id":"capstone/add-evals-and-traces/s03-03-the-trace-shape"},{"id":"capstone/add-evals-and-traces/s04-04-predict-the-pass-rate"},{"id":"capstone/add-evals-and-traces/s05-05-fill-the-trace-append"},{"id":"capstone/add-evals-and-traces/s06-06-fix-the-trace-mutation"},{"id":"capstone/add-evals-and-traces/s07-07-fix-the-tautological-eval"},{"id":"capstone/add-evals-and-traces/s08-08-write-the-eval-runner"},{"id":"capstone/add-evals-and-traces/s09-09-checkpoint"}]},{"slug":"wire-an-mcp-tool","steps":[{"id":"capstone/wire-an-mcp-tool/s01-01-intro"},{"id":"capstone/wire-an-mcp-tool/s02-02-when-to-use-mcp"},{"id":"capstone/wire-an-mcp-tool/s03-03-the-mcp-protocol"},{"id":"capstone/wire-an-mcp-tool/s04-04-predict-the-tool-list"},{"id":"capstone/wire-an-mcp-tool/s05-05-fill-the-tools-call"},{"id":"capstone/wire-an-mcp-tool/s06-06-fix-the-result-shape"},{"id":"capstone/wire-an-mcp-tool/s07-07-fix-the-iserror-flag"},{"id":"capstone/wire-an-mcp-tool/s08-08-write-the-mcp-bridge"},{"id":"capstone/wire-an-mcp-tool/s09-09-checkpoint"}]},{"slug":"portfolio-ml-system-option","steps":[{"id":"capstone/portfolio-ml-system-option/s01-01-the-other-capstone"},{"id":"capstone/portfolio-ml-system-option/s02-02-what-reviewers-open"},{"id":"capstone/portfolio-ml-system-option/s03-03-predict-the-gate"},{"id":"capstone/portfolio-ml-system-option/s04-04-fill-the-gate"},{"id":"capstone/portfolio-ml-system-option/s05-05-fix-the-train-accuracy-gate"},{"id":"capstone/portfolio-ml-system-option/s06-06-write-the-scope-check"},{"id":"capstone/portfolio-ml-system-option/s07-07-checkpoint"}]}]},{"slug":"agent-harnesses","lessons":[{"slug":"project-workspaces-not-product-names","steps":[{"id":"agent-harnesses/project-workspaces-not-product-names/s01-01-intro"}]},{"slug":"what-a-harness-is","steps":[{"id":"agent-harnesses/what-a-harness-is/s01-01-intro"},{"id":"agent-harnesses/what-a-harness-is/s02-02-which-is-a-harness"},{"id":"agent-harnesses/what-a-harness-is/s03-03-the-four-layers"},{"id":"agent-harnesses/what-a-harness-is/s04-04-predict-the-layer"},{"id":"agent-harnesses/what-a-harness-is/s05-05-fill-the-dispatch"},{"id":"agent-harnesses/what-a-harness-is/s06-06-fix-the-no-retry"},{"id":"agent-harnesses/what-a-harness-is/s07-07-fix-the-hardcoded-registry"},{"id":"agent-harnesses/what-a-harness-is/s08-08-write-mini-harness"},{"id":"agent-harnesses/what-a-harness-is/s09-09-checkpoint"}]},{"slug":"architecting-an-ai-native-workflow","steps":[{"id":"agent-harnesses/architecting-an-ai-native-workflow/s01-01-intro"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s02-02-which-workflow-fits"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s03-03-mapping-the-workflow"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s04-04-predict-the-readiness-score"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s05-05-fill-the-required-fields"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s06-06-fix-the-implicit-policy"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s07-07-fix-the-wrong-metric"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s08-08-write-the-workflow-validator"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s09-09-checkpoint"}]},{"slug":"five-industries-walked-through","steps":[{"id":"agent-harnesses/five-industries-walked-through/s01-01-the-case-study-method"},{"id":"agent-harnesses/five-industries-walked-through/s02-02-home-services"},{"id":"agent-harnesses/five-industries-walked-through/s03-03-which-task-belongs-to-the-agent"},{"id":"agent-harnesses/five-industries-walked-through/s04-04-insurance-brokerage"},{"id":"agent-harnesses/five-industries-walked-through/s05-05-recruiting-firm"},{"id":"agent-harnesses/five-industries-walked-through/s06-06-which-industry-wedges-fastest"},{"id":"agent-harnesses/five-industries-walked-through/s07-07-the-incumbent-trap"},{"id":"agent-harnesses/five-industries-walked-through/s08-08-write-the-process-debt-audit"},{"id":"agent-harnesses/five-industries-walked-through/s09-09-checkpoint"}]},{"slug":"incident-runbook-from-the-repo","steps":[{"id":"agent-harnesses/incident-runbook-from-the-repo/s01-01-intro"},{"id":"agent-harnesses/incident-runbook-from-the-repo/s02-02-which-is-a-runbook"},{"id":"agent-harnesses/incident-runbook-from-the-repo/s03-03-detect-decide-writeback"},{"id":"agent-harnesses/incident-runbook-from-the-repo/s04-04-predict-the-decision"},{"id":"agent-harnesses/incident-runbook-from-the-repo/s05-05-fill-the-decision-rule"},{"id":"agent-harnesses/incident-runbook-from-the-repo/s06-06-fix-the-wiki-page"},{"id":"agent-harnesses/incident-runbook-from-the-repo/s07-07-fix-the-unnamed-human"},{"id":"agent-harnesses/incident-runbook-from-the-repo/s08-08-write-the-runbook"},{"id":"agent-harnesses/incident-runbook-from-the-repo/s09-09-checkpoint"}]}]},{"slug":"ai-image-generation","lessons":[{"slug":"the-image-model-landscape","steps":[{"id":"ai-image-generation/the-image-model-landscape/s01-01-the-six-families"},{"id":"ai-image-generation/the-image-model-landscape/s02-02-flux-vs-midjourney"},{"id":"ai-image-generation/the-image-model-landscape/s03-03-pick-the-model"},{"id":"ai-image-generation/the-image-model-landscape/s04-04-nano-banana-and-the-batch-economics"},{"id":"ai-image-generation/the-image-model-landscape/s05-05-text-in-image"},{"id":"ai-image-generation/the-image-model-landscape/s06-06-which-model-for-text"},{"id":"ai-image-generation/the-image-model-landscape/s07-07-the-decision-tree"},{"id":"ai-image-generation/the-image-model-landscape/s08-08-write-pick-image-model"},{"id":"ai-image-generation/the-image-model-landscape/s09-09-checkpoint"}]},{"slug":"prompting-for-real-output","steps":[{"id":"ai-image-generation/prompting-for-real-output/s01-01-why-the-girl-with-red-hair-fails"},{"id":"ai-image-generation/prompting-for-real-output/s02-02-the-eight-knobs"},{"id":"ai-image-generation/prompting-for-real-output/s03-03-fix-the-vague-prompt"},{"id":"ai-image-generation/prompting-for-real-output/s04-04-controlnet-and-reference"},{"id":"ai-image-generation/prompting-for-real-output/s05-05-which-knob-is-missing"},{"id":"ai-image-generation/prompting-for-real-output/s06-06-failure-modes"},{"id":"ai-image-generation/prompting-for-real-output/s07-07-write-score-image-prompt"},{"id":"ai-image-generation/prompting-for-real-output/s08-08-checkpoint"}]},{"slug":"the-image-pipeline","steps":[{"id":"ai-image-generation/the-image-pipeline/s01-01-the-six-stages"},{"id":"ai-image-generation/the-image-pipeline/s02-02-batch-and-filter"},{"id":"ai-image-generation/the-image-pipeline/s03-03-which-stage-is-broken"},{"id":"ai-image-generation/the-image-pipeline/s04-04-format-and-platform"},{"id":"ai-image-generation/the-image-pipeline/s05-05-fill-the-pipeline"},{"id":"ai-image-generation/the-image-pipeline/s06-06-write-cost-for-batch"},{"id":"ai-image-generation/the-image-pipeline/s07-07-checkpoint"}]}]},{"slug":"ai-video-generation","lessons":[{"slug":"the-video-model-lineup","steps":[{"id":"ai-video-generation/the-video-model-lineup/s01-01-the-landscape"},{"id":"ai-video-generation/the-video-model-lineup/s02-02-sora-and-veo"},{"id":"ai-video-generation/the-video-model-lineup/s03-03-higgsfield-and-camera-control"},{"id":"ai-video-generation/the-video-model-lineup/s04-04-the-second-wave"},{"id":"ai-video-generation/the-video-model-lineup/s05-05-text-vs-image-vs-video"},{"id":"ai-video-generation/the-video-model-lineup/s06-06-which-lane"},{"id":"ai-video-generation/the-video-model-lineup/s07-07-pick-the-model"},{"id":"ai-video-generation/the-video-model-lineup/s08-08-write-pick-video-model"},{"id":"ai-video-generation/the-video-model-lineup/s09-09-checkpoint"}]},{"slug":"camera-control-and-motion","steps":[{"id":"ai-video-generation/camera-control-and-motion/s01-01-the-static-camera-problem"},{"id":"ai-video-generation/camera-control-and-motion/s02-02-shot-vocabulary"},{"id":"ai-video-generation/camera-control-and-motion/s03-03-which-shot-is-which"},{"id":"ai-video-generation/camera-control-and-motion/s04-04-camera-moves"},{"id":"ai-video-generation/camera-control-and-motion/s05-05-the-i2v-keyframe-pattern"},{"id":"ai-video-generation/camera-control-and-motion/s06-06-spot-the-generic-prompt"},{"id":"ai-video-generation/camera-control-and-motion/s07-07-write-audit-shot-list"},{"id":"ai-video-generation/camera-control-and-motion/s08-08-checkpoint"}]},{"slug":"the-cost-math","steps":[{"id":"ai-video-generation/the-cost-math/s01-01-the-unit-economics"},{"id":"ai-video-generation/the-cost-math/s02-02-the-retake-rate-reality"},{"id":"ai-video-generation/the-cost-math/s03-03-which-tier-for-this-budget"},{"id":"ai-video-generation/the-cost-math/s04-04-the-60-second-promo"},{"id":"ai-video-generation/the-cost-math/s05-05-write-cost-per-minute"},{"id":"ai-video-generation/the-cost-math/s06-06-checkpoint"}]}]},{"slug":"programmatic-design","lessons":[{"slug":"why-programmatic","steps":[{"id":"programmatic-design/why-programmatic/s01-01-intro"},{"id":"programmatic-design/why-programmatic/s02-02-the-three-jobs"},{"id":"programmatic-design/why-programmatic/s03-03-which-task-is-code"},{"id":"programmatic-design/why-programmatic/s04-04-parametric-vs-one-shot"},{"id":"programmatic-design/why-programmatic/s05-05-the-classifier"},{"id":"programmatic-design/why-programmatic/s06-06-write-classify-video-task"},{"id":"programmatic-design/why-programmatic/s07-07-checkpoint"}]},{"slug":"hyperframes-and-remotion","steps":[{"id":"programmatic-design/hyperframes-and-remotion/s01-01-the-two-tools"},{"id":"programmatic-design/hyperframes-and-remotion/s02-02-hyperframes-tour"},{"id":"programmatic-design/hyperframes-and-remotion/s03-03-hyperframes-cli"},{"id":"programmatic-design/hyperframes-and-remotion/s04-04-remotion-tour"},{"id":"programmatic-design/hyperframes-and-remotion/s05-05-the-decision-tree"},{"id":"programmatic-design/hyperframes-and-remotion/s06-06-pick-the-tool"},{"id":"programmatic-design/hyperframes-and-remotion/s07-07-claude-design-html"},{"id":"programmatic-design/hyperframes-and-remotion/s08-08-write-pick-tool"},{"id":"programmatic-design/hyperframes-and-remotion/s09-09-checkpoint"}]},{"slug":"the-ai-native-design-pipeline","steps":[{"id":"programmatic-design/the-ai-native-design-pipeline/s01-01-the-seven-steps"},{"id":"programmatic-design/the-ai-native-design-pipeline/s02-02-where-ai-fits-each-step"},{"id":"programmatic-design/the-ai-native-design-pipeline/s03-03-which-model-which-step"},{"id":"programmatic-design/the-ai-native-design-pipeline/s04-04-website-to-hyperframes"},{"id":"programmatic-design/the-ai-native-design-pipeline/s05-05-the-cost-model"},{"id":"programmatic-design/the-ai-native-design-pipeline/s06-06-score-the-pipeline"},{"id":"programmatic-design/the-ai-native-design-pipeline/s07-07-write-score-pipeline"},{"id":"programmatic-design/the-ai-native-design-pipeline/s08-08-checkpoint"}]}]},{"slug":"harness-engineering","lessons":[{"slug":"the-harness-engineering-mindset","steps":[{"id":"harness-engineering/the-harness-engineering-mindset/s01-01-the-model-is-only-one-input"},{"id":"harness-engineering/the-harness-engineering-mindset/s02-02-the-six-pieces-of-a-harness"},{"id":"harness-engineering/the-harness-engineering-mindset/s03-03-which-piece-is-missing"},{"id":"harness-engineering/the-harness-engineering-mindset/s04-04-the-skill-issue-reframe"},{"id":"harness-engineering/the-harness-engineering-mindset/s05-05-pick-the-fix"},{"id":"harness-engineering/the-harness-engineering-mindset/s06-06-the-harness-gap"},{"id":"harness-engineering/the-harness-engineering-mindset/s07-07-write-the-harness-inventory"},{"id":"harness-engineering/the-harness-engineering-mindset/s08-08-checkpoint"}]},{"slug":"the-ratchet","steps":[{"id":"harness-engineering/the-ratchet/s01-01-failures-are-signals-not-flukes"},{"id":"harness-engineering/the-ratchet/s02-02-every-line-traces-to-a-failure"},{"id":"harness-engineering/the-ratchet/s03-03-which-rule-earned-its-line"},{"id":"harness-engineering/the-ratchet/s04-04-working-backwards-from-behavior"},{"id":"harness-engineering/the-ratchet/s05-05-the-ratchet-log"},{"id":"harness-engineering/the-ratchet/s06-06-spot-the-cruft"},{"id":"harness-engineering/the-ratchet/s07-07-write-the-ratchet-update"},{"id":"harness-engineering/the-ratchet/s08-08-audit-the-ratchet"},{"id":"harness-engineering/the-ratchet/s09-09-checkpoint"}]},{"slug":"context-engineering","steps":[{"id":"harness-engineering/context-engineering/s01-01-the-context-rot"},{"id":"harness-engineering/context-engineering/s02-02-compaction"},{"id":"harness-engineering/context-engineering/s03-03-tool-call-offloading"},{"id":"harness-engineering/context-engineering/s04-04-progressive-disclosure"},{"id":"harness-engineering/context-engineering/s05-05-which-technique-fits"},{"id":"harness-engineering/context-engineering/s06-06-the-memory-hierarchy"},{"id":"harness-engineering/context-engineering/s07-07-classify-the-data"},{"id":"harness-engineering/context-engineering/s08-08-write-the-context-budget"},{"id":"harness-engineering/context-engineering/s09-09-checkpoint"}]},{"slug":"long-horizon-execution","steps":[{"id":"harness-engineering/long-horizon-execution/s01-01-the-40-step-problem"},{"id":"harness-engineering/long-horizon-execution/s02-02-loops-intercepting-early-stops"},{"id":"harness-engineering/long-horizon-execution/s03-03-planning-as-a-separate-step"},{"id":"harness-engineering/long-horizon-execution/s04-04-splits-generation-vs-evaluation"},{"id":"harness-engineering/long-horizon-execution/s05-05-which-pattern-fixes-the-failure"},{"id":"harness-engineering/long-horizon-execution/s06-06-hooks-as-the-enforcement-layer"},{"id":"harness-engineering/long-horizon-execution/s07-07-the-type-check-backpressure-pattern"},{"id":"harness-engineering/long-horizon-execution/s08-08-write-the-hook-router"},{"id":"harness-engineering/long-horizon-execution/s09-09-write-the-completion-guard"},{"id":"harness-engineering/long-horizon-execution/s10-10-checkpoint"}]},{"slug":"the-haas-shift","steps":[{"id":"harness-engineering/the-haas-shift/s01-01-llm-apis-to-harness-apis"},{"id":"harness-engineering/the-haas-shift/s02-02-the-convergence"},{"id":"harness-engineering/the-haas-shift/s03-03-which-haas-fits"},{"id":"harness-engineering/the-haas-shift/s04-04-harnesses-dont-shrink-they-move"},{"id":"harness-engineering/the-haas-shift/s05-05-the-overfitting-feedback-loop"},{"id":"harness-engineering/the-haas-shift/s06-06-mcp-security-as-prompt-injection"},{"id":"harness-engineering/the-haas-shift/s07-07-which-component-is-debt"},{"id":"harness-engineering/the-haas-shift/s08-08-write-the-build-vs-buy"},{"id":"harness-engineering/the-haas-shift/s09-09-checkpoint"}]},{"slug":"pipeline-boundaries","steps":[{"id":"harness-engineering/pipeline-boundaries/s01-01-the-boundary-is-an-artifact"},{"id":"harness-engineering/pipeline-boundaries/s02-02-spot-the-risky-handoff"},{"id":"harness-engineering/pipeline-boundaries/s03-03-predict-the-gate"},{"id":"harness-engineering/pipeline-boundaries/s04-04-fill-the-gate"},{"id":"harness-engineering/pipeline-boundaries/s05-05-fix-the-overtrust"},{"id":"harness-engineering/pipeline-boundaries/s06-06-write-the-advance-check"},{"id":"harness-engineering/pipeline-boundaries/s07-07-checkpoint"}]}]},{"slug":"intro-to-terminal","lessons":[{"slug":"what-the-terminal-is","steps":[{"id":"intro-to-terminal/what-the-terminal-is/s01-01-not-a-cockpit"},{"id":"intro-to-terminal/what-the-terminal-is/s02-02-what-it-actually-is"},{"id":"intro-to-terminal/what-the-terminal-is/s03-03-open-it"},{"id":"intro-to-terminal/what-the-terminal-is/s04-04-which-app"},{"id":"intro-to-terminal/what-the-terminal-is/s05-05-the-prompt"},{"id":"intro-to-terminal/what-the-terminal-is/s06-06-what-a-command-is"},{"id":"intro-to-terminal/what-the-terminal-is/s07-07-checkpoint"}]},{"slug":"moving-around","steps":[{"id":"intro-to-terminal/moving-around/s01-01-the-filesystem-is-a-tree"},{"id":"intro-to-terminal/moving-around/s02-02-pwd-and-ls"},{"id":"intro-to-terminal/moving-around/s03-03-where-am-i"},{"id":"intro-to-terminal/moving-around/s04-04-read-this-listing"},{"id":"intro-to-terminal/moving-around/s05-05-cd"},{"id":"intro-to-terminal/moving-around/s06-06-follow-the-moves"},{"id":"intro-to-terminal/moving-around/s07-07-pick-the-command"},{"id":"intro-to-terminal/moving-around/s08-08-checkpoint"}]},{"slug":"making-things","steps":[{"id":"intro-to-terminal/making-things/s01-01-mkdir-and-touch"},{"id":"intro-to-terminal/making-things/s02-02-folder-or-file"},{"id":"intro-to-terminal/making-things/s03-03-cat-and-echo"},{"id":"intro-to-terminal/making-things/s04-04-reading-a-file"},{"id":"intro-to-terminal/making-things/s05-05-rm-carefully"},{"id":"intro-to-terminal/making-things/s06-06-the-no-undo-rule"},{"id":"intro-to-terminal/making-things/s07-07-string-it-together"},{"id":"intro-to-terminal/making-things/s08-08-checkpoint"}]}]},{"slug":"intro-to-claude-cli","lessons":[{"slug":"what-the-claude-cli-is","steps":[{"id":"intro-to-claude-cli/what-the-claude-cli-is/s01-01-the-glass-is-gone"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s02-02-chat-vs-cli"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s03-03-which-tool"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s04-04-the-agent-loop"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s05-05-read-the-loop"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s06-06-permission-and-limits"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s07-07-checkpoint"}]},{"slug":"install-and-sign-in","steps":[{"id":"intro-to-claude-cli/install-and-sign-in/s01-01-two-ways-to-install"},{"id":"intro-to-claude-cli/install-and-sign-in/s02-02-run-the-installer"},{"id":"intro-to-claude-cli/install-and-sign-in/s03-03-install-check"},{"id":"intro-to-claude-cli/install-and-sign-in/s04-04-sign-in"},{"id":"intro-to-claude-cli/install-and-sign-in/s05-05-what-sign-in-did"},{"id":"intro-to-claude-cli/install-and-sign-in/s06-06-auth-check"},{"id":"intro-to-claude-cli/install-and-sign-in/s07-07-account-not-api-key"},{"id":"intro-to-claude-cli/install-and-sign-in/s08-08-checkpoint"}]},{"slug":"your-first-session","steps":[{"id":"intro-to-claude-cli/your-first-session/s01-01-start-in-a-folder"},{"id":"intro-to-claude-cli/your-first-session/s02-02-your-first-instruction"},{"id":"intro-to-claude-cli/your-first-session/s03-03-watch-the-loop"},{"id":"intro-to-claude-cli/your-first-session/s04-04-read-what-it-did"},{"id":"intro-to-claude-cli/your-first-session/s05-05-one-shot-or-interactive"},{"id":"intro-to-claude-cli/your-first-session/s06-06-chat-or-cli-decision"},{"id":"intro-to-claude-cli/your-first-session/s07-07-where-this-goes"},{"id":"intro-to-claude-cli/your-first-session/s08-08-checkpoint"}]}]},{"slug":"intro-to-codex-cli","lessons":[{"slug":"another-tool-same-idea","steps":[{"id":"intro-to-codex-cli/another-tool-same-idea/s01-01-why-a-second-tool"},{"id":"intro-to-codex-cli/another-tool-same-idea/s02-02-what-carries-over"},{"id":"intro-to-codex-cli/another-tool-same-idea/s03-03-what-carries-over-check"},{"id":"intro-to-codex-cli/another-tool-same-idea/s04-04-what-is-different"},{"id":"intro-to-codex-cli/another-tool-same-idea/s05-05-spot-the-difference"},{"id":"intro-to-codex-cli/another-tool-same-idea/s06-06-when-to-reach-for-which"},{"id":"intro-to-codex-cli/another-tool-same-idea/s07-07-checkpoint"}]},{"slug":"install-and-sign-in","steps":[{"id":"intro-to-codex-cli/install-and-sign-in/s01-01-the-install-paths"},{"id":"intro-to-codex-cli/install-and-sign-in/s02-02-installing-it"},{"id":"intro-to-codex-cli/install-and-sign-in/s03-03-verify-it"},{"id":"intro-to-codex-cli/install-and-sign-in/s04-04-install-check"},{"id":"intro-to-codex-cli/install-and-sign-in/s05-05-sign-in"},{"id":"intro-to-codex-cli/install-and-sign-in/s06-06-account-vs-api-key"},{"id":"intro-to-codex-cli/install-and-sign-in/s07-07-auth-check"},{"id":"intro-to-codex-cli/install-and-sign-in/s08-08-checkpoint"}]},{"slug":"your-first-session","steps":[{"id":"intro-to-codex-cli/your-first-session/s01-01-start-and-instruct"},{"id":"intro-to-codex-cli/your-first-session/s02-02-the-loop-again"},{"id":"intro-to-codex-cli/your-first-session/s03-03-read-what-it-did"},{"id":"intro-to-codex-cli/your-first-session/s04-04-usage-patterns"},{"id":"intro-to-codex-cli/your-first-session/s05-05-usage-scenario"},{"id":"intro-to-codex-cli/your-first-session/s06-06-running-both"},{"id":"intro-to-codex-cli/your-first-session/s07-07-where-this-goes"},{"id":"intro-to-codex-cli/your-first-session/s08-08-checkpoint"}]}]},{"slug":"team-skills","lessons":[{"slug":"what-a-skill-is","steps":[{"id":"team-skills/what-a-skill-is/s01-01-the-re-explaining-problem"},{"id":"team-skills/what-a-skill-is/s02-02-a-skill-defined"},{"id":"team-skills/what-a-skill-is/s03-03-a-real-skill"},{"id":"team-skills/what-a-skill-is/s04-04-what-a-skill-is-check"},{"id":"team-skills/what-a-skill-is/s05-05-skill-vs-prompt-vs-agent"},{"id":"team-skills/what-a-skill-is/s06-06-skill-or-agent"},{"id":"team-skills/what-a-skill-is/s07-07-checkpoint"}]},{"slug":"skills-as-a-team-thing","steps":[{"id":"team-skills/skills-as-a-team-thing/s01-01-the-team-problem"},{"id":"team-skills/skills-as-a-team-thing/s02-02-org-wide-skills"},{"id":"team-skills/skills-as-a-team-thing/s03-03-sharing-and-the-directory"},{"id":"team-skills/skills-as-a-team-thing/s04-04-deployment-check"},{"id":"team-skills/skills-as-a-team-thing/s05-05-collaborating-through-skills"},{"id":"team-skills/skills-as-a-team-thing/s06-06-sharing-scenario"},{"id":"team-skills/skills-as-a-team-thing/s07-07-checkpoint"}]},{"slug":"role-playbooks","steps":[{"id":"team-skills/role-playbooks/s01-01-hr-skills"},{"id":"team-skills/role-playbooks/s02-02-legal-skills"},{"id":"team-skills/role-playbooks/s03-03-operations-skills"},{"id":"team-skills/role-playbooks/s04-04-match-the-skill"},{"id":"team-skills/role-playbooks/s05-05-skill-or-one-off"},{"id":"team-skills/role-playbooks/s06-06-skill-or-one-off-check"},{"id":"team-skills/role-playbooks/s07-07-checkpoint"}]},{"slug":"governance-and-trust","steps":[{"id":"team-skills/governance-and-trust/s01-01-a-skill-can-do-real-things"},{"id":"team-skills/governance-and-trust/s02-02-treat-it-like-software"},{"id":"team-skills/governance-and-trust/s03-03-evaluate-before-you-deploy"},{"id":"team-skills/governance-and-trust/s04-04-access-control"},{"id":"team-skills/governance-and-trust/s05-05-governance-scenario"},{"id":"team-skills/governance-and-trust/s06-06-a-rollout-checklist"},{"id":"team-skills/governance-and-trust/s07-07-checkpoint"}]}]},{"slug":"dataframes-numpy-pandas","lessons":[{"slug":"ml-package-map","steps":[{"id":"dataframes-numpy-pandas/ml-package-map/s01-01-intro"},{"id":"dataframes-numpy-pandas/ml-package-map/s02-02-match-the-tool"},{"id":"dataframes-numpy-pandas/ml-package-map/s03-03-predict-sys-modules"},{"id":"dataframes-numpy-pandas/ml-package-map/s04-04-fill-the-registry-key"},{"id":"dataframes-numpy-pandas/ml-package-map/s05-05-fix-the-swapped-map"},{"id":"dataframes-numpy-pandas/ml-package-map/s06-06-write-find-unused"},{"id":"dataframes-numpy-pandas/ml-package-map/s07-07-checkpoint"}]},{"slug":"array-and-series-shapes","steps":[{"id":"dataframes-numpy-pandas/array-and-series-shapes/s01-01-intro"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s02-02-choose-the-risk"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s03-03-predict-the-checks"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s04-04-fill-the-return"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s05-05-fix-the-overtrust"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s06-06-write-count-ready"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s07-07-checkpoint"}]},{"slug":"dataframe-selection-and-cleaning","steps":[{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s01-01-intro"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s02-02-choose-the-risk"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s03-03-predict-the-checks"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s04-04-fill-the-return"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s05-05-fix-the-overtrust"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s06-06-write-count-ready"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s07-07-checkpoint"}]},{"slug":"missing-values-and-types","steps":[{"id":"dataframes-numpy-pandas/missing-values-and-types/s01-01-intro"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s02-02-choose-the-risk"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s03-03-predict-the-checks"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s04-04-fill-the-return"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s05-05-fix-the-overtrust"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s06-06-write-count-ready"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s07-07-checkpoint"}]},{"slug":"groupby-joins-and-features","steps":[{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s01-01-intro"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s02-02-choose-the-risk"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s03-03-predict-the-checks"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s04-04-fill-the-return"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s05-05-fix-the-overtrust"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s06-06-write-count-ready"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s07-07-checkpoint"}]},{"slug":"mission-api-to-dataframe","steps":[{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s01-01-intro"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s02-02-choose-the-risk"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s03-03-predict-the-checks"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s04-04-fill-the-return"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s05-05-fix-the-overtrust"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s06-06-write-count-ready"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s07-07-checkpoint"}]}]},{"slug":"sql-for-ml-datasets","lessons":[{"slug":"select-filter-aggregate","steps":[{"id":"sql-for-ml-datasets/select-filter-aggregate/s01-01-intro"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s02-02-choose-the-risk"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s03-03-predict-the-checks"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s04-04-fill-the-return"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s05-05-fix-the-overtrust"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s06-06-write-count-ready"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s07-07-checkpoint"}]},{"slug":"joins-and-label-windows","steps":[{"id":"sql-for-ml-datasets/joins-and-label-windows/s01-01-intro"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s02-02-choose-the-risk"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s03-03-predict-the-checks"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s04-04-fill-the-return"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s05-05-fix-the-overtrust"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s06-06-write-count-ready"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s07-07-checkpoint"}]},{"slug":"sql-quality-checks","steps":[{"id":"sql-for-ml-datasets/sql-quality-checks/s01-01-intro"},{"id":"sql-for-ml-datasets/sql-quality-checks/s02-02-choose-the-risk"},{"id":"sql-for-ml-datasets/sql-quality-checks/s03-03-predict-the-checks"},{"id":"sql-for-ml-datasets/sql-quality-checks/s04-04-fill-the-return"},{"id":"sql-for-ml-datasets/sql-quality-checks/s05-05-fix-the-overtrust"},{"id":"sql-for-ml-datasets/sql-quality-checks/s06-06-write-count-ready"},{"id":"sql-for-ml-datasets/sql-quality-checks/s07-07-checkpoint"}]},{"slug":"mission-sql-feature-query-lab","steps":[{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s01-01-intro"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s02-02-choose-the-risk"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s03-03-predict-the-checks"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s04-04-fill-the-return"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s05-05-fix-the-overtrust"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s06-06-write-count-ready"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s07-07-checkpoint"}]}]},{"slug":"dataset-pipelines","lessons":[{"slug":"csv-jsonl-parquet-tradeoffs","steps":[{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s01-01-intro"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s02-02-choose-the-risk"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s03-03-predict-the-checks"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s04-04-fill-the-return"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s05-05-fix-the-overtrust"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s06-06-write-count-ready"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s07-07-checkpoint"}]},{"slug":"pagination-retries-checkpoints","steps":[{"id":"dataset-pipelines/pagination-retries-checkpoints/s01-01-intro"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s02-02-choose-the-risk"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s03-03-predict-the-checks"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s04-04-fill-the-return"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s05-05-fix-the-overtrust"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s06-06-write-count-ready"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s07-07-checkpoint"}]},{"slug":"schema-drift-and-contracts","steps":[{"id":"dataset-pipelines/schema-drift-and-contracts/s01-01-intro"},{"id":"dataset-pipelines/schema-drift-and-contracts/s02-02-choose-the-risk"},{"id":"dataset-pipelines/schema-drift-and-contracts/s03-03-predict-the-checks"},{"id":"dataset-pipelines/schema-drift-and-contracts/s04-04-fill-the-return"},{"id":"dataset-pipelines/schema-drift-and-contracts/s05-05-fix-the-overtrust"},{"id":"dataset-pipelines/schema-drift-and-contracts/s06-06-write-count-ready"},{"id":"dataset-pipelines/schema-drift-and-contracts/s07-07-checkpoint"}]},{"slug":"pipeline-orchestration-shape","steps":[{"id":"dataset-pipelines/pipeline-orchestration-shape/s01-01-intro"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s02-02-choose-the-risk"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s03-03-predict-the-checks"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s04-04-fill-the-return"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s05-05-fix-the-overtrust"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s06-06-write-count-ready"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s07-07-checkpoint"}]},{"slug":"mission-api-to-dataset-pipeline","steps":[{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s01-01-intro"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s02-02-choose-the-risk"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s03-03-predict-the-checks"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s04-04-fill-the-return"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s05-05-fix-the-overtrust"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s06-06-write-count-ready"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s07-07-checkpoint"}]}]},{"slug":"ml-math-and-stats","lessons":[{"slug":"vectors-matrices-dot-products","steps":[{"id":"ml-math-and-stats/vectors-matrices-dot-products/s01-01-intro"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s02-02-choose-the-risk"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s03-03-predict-the-checks"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s04-04-fill-the-return"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s05-05-fix-the-overtrust"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s06-06-write-count-ready"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s07-07-checkpoint"}]},{"slug":"probability-and-baselines","steps":[{"id":"ml-math-and-stats/probability-and-baselines/s01-01-intro"},{"id":"ml-math-and-stats/probability-and-baselines/s02-02-choose-the-risk"},{"id":"ml-math-and-stats/probability-and-baselines/s03-03-predict-the-checks"},{"id":"ml-math-and-stats/probability-and-baselines/s04-04-fill-the-return"},{"id":"ml-math-and-stats/probability-and-baselines/s05-05-fix-the-overtrust"},{"id":"ml-math-and-stats/probability-and-baselines/s06-06-write-count-ready"},{"id":"ml-math-and-stats/probability-and-baselines/s07-07-checkpoint"}]},{"slug":"distributions-sampling-variance","steps":[{"id":"ml-math-and-stats/distributions-sampling-variance/s01-01-intro"},{"id":"ml-math-and-stats/distributions-sampling-variance/s02-02-choose-the-risk"},{"id":"ml-math-and-stats/distributions-sampling-variance/s03-03-predict-the-checks"},{"id":"ml-math-and-stats/distributions-sampling-variance/s04-04-fill-the-return"},{"id":"ml-math-and-stats/distributions-sampling-variance/s05-05-fix-the-overtrust"},{"id":"ml-math-and-stats/distributions-sampling-variance/s06-06-write-variance"},{"id":"ml-math-and-stats/distributions-sampling-variance/s07-07-checkpoint"}]},{"slug":"bias-variance-and-leakage","steps":[{"id":"ml-math-and-stats/bias-variance-and-leakage/s01-01-intro"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s02-02-choose-the-risk"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s03-03-predict-the-checks"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s04-04-fill-the-return"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s05-05-fix-the-overtrust"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s06-06-write-diagnose"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s07-07-checkpoint"}]},{"slug":"mission-statistical-sanity-check","steps":[{"id":"ml-math-and-stats/mission-statistical-sanity-check/s01-01-intro"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s02-02-choose-the-risk"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s03-03-predict-the-checks"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s04-04-fill-the-return"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s05-05-fix-the-overtrust"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s06-06-write-count-ready"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s07-07-checkpoint"}]}]},{"slug":"supervised-learning-workflows","lessons":[{"slug":"labels-features-splits","steps":[{"id":"supervised-learning-workflows/labels-features-splits/s01-01-intro"},{"id":"supervised-learning-workflows/labels-features-splits/s02-02-choose-the-risk"},{"id":"supervised-learning-workflows/labels-features-splits/s03-03-predict-the-checks"},{"id":"supervised-learning-workflows/labels-features-splits/s04-04-fill-the-return"},{"id":"supervised-learning-workflows/labels-features-splits/s05-05-fix-the-overtrust"},{"id":"supervised-learning-workflows/labels-features-splits/s06-06-write-split-audit"},{"id":"supervised-learning-workflows/labels-features-splits/s07-07-checkpoint"}]},{"slug":"baselines-before-fancy-models","steps":[{"id":"supervised-learning-workflows/baselines-before-fancy-models/s01-01-intro"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s02-02-choose-the-risk"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s03-03-predict-the-checks"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s04-04-fill-the-return"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s05-05-fix-the-overtrust"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s06-06-write-baseline-comparison"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s07-07-checkpoint"}]},{"slug":"training-and-prediction-loop","steps":[{"id":"supervised-learning-workflows/training-and-prediction-loop/s01-01-intro"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s02-02-choose-the-risk"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s03-03-predict-the-checks"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s04-04-fill-the-return"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s05-05-fix-the-overtrust"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s06-06-write-train"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s07-07-checkpoint"}]},{"slug":"overfitting-and-regularization","steps":[{"id":"supervised-learning-workflows/overfitting-and-regularization/s01-01-intro"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s02-02-choose-the-risk"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s03-03-predict-the-checks"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s04-04-fill-the-return"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s05-05-fix-the-overtrust"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s06-06-write-overfit-audit"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s07-07-checkpoint"}]},{"slug":"mission-baseline-model-showdown","steps":[{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s01-01-intro"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s02-02-choose-the-risk"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s03-03-predict-the-checks"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s04-04-fill-the-return"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s05-05-fix-the-overtrust"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s06-06-write-classifier-brief"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s07-07-checkpoint"}]}]},{"slug":"unsupervised-learning-and-embeddings","lessons":[{"slug":"clustering-without-labels","steps":[{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s01-01-intro"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s02-02-choose-the-risk"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s03-03-predict-the-checks"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s04-04-fill-the-return"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s05-05-fix-the-overtrust"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s06-06-write-cluster-summary"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s07-07-checkpoint"}]},{"slug":"dimensionality-and-distance","steps":[{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s01-01-intro"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s02-02-choose-the-risk"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s03-03-predict-the-checks"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s04-04-fill-the-return"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s05-05-fix-the-overtrust"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s06-06-write-euclidean"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s07-07-checkpoint"}]},{"slug":"embeddings-as-features","steps":[{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s01-01-intro"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s02-02-choose-the-risk"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s03-03-predict-the-checks"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s04-04-fill-the-return"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s05-05-fix-the-overtrust"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s06-06-write-retrieval-receipt"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s07-07-checkpoint"}]},{"slug":"mission-tiny-recommender","steps":[{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s01-01-intro"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s02-02-choose-the-risk"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s03-03-predict-the-checks"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s04-04-fill-the-return"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s05-05-fix-the-overtrust"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s06-06-write-recommendation-card"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s07-07-checkpoint"}]}]},{"slug":"metrics-and-error-analysis","lessons":[{"slug":"confusion-matrix","steps":[{"id":"metrics-and-error-analysis/confusion-matrix/s01-01-intro"},{"id":"metrics-and-error-analysis/confusion-matrix/s02-02-choose-the-risk"},{"id":"metrics-and-error-analysis/confusion-matrix/s03-03-predict-the-checks"},{"id":"metrics-and-error-analysis/confusion-matrix/s04-04-fill-the-return"},{"id":"metrics-and-error-analysis/confusion-matrix/s05-05-fix-the-overtrust"},{"id":"metrics-and-error-analysis/confusion-matrix/s06-06-write-count-ready"},{"id":"metrics-and-error-analysis/confusion-matrix/s07-07-checkpoint"}]},{"slug":"precision-recall-thresholds","steps":[{"id":"metrics-and-error-analysis/precision-recall-thresholds/s01-01-intro"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s02-02-choose-the-risk"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s03-03-predict-the-checks"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s04-04-fill-the-return"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s05-05-fix-the-overtrust"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s06-06-write-count-ready"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s07-07-checkpoint"}]},{"slug":"slice-based-error-analysis","steps":[{"id":"metrics-and-error-analysis/slice-based-error-analysis/s01-01-intro"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s02-02-choose-the-risk"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s03-03-predict-the-checks"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s04-04-fill-the-return"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s05-05-fix-the-overtrust"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s06-06-write-count-ready"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s07-07-checkpoint"}]},{"slug":"regression-metrics-and-residuals","steps":[{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s01-01-intro"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s02-02-choose-the-risk"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s03-03-predict-the-checks"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s04-04-fill-the-return"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s05-05-fix-the-overtrust"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s06-06-write-count-ready"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s07-07-checkpoint"}]},{"slug":"mission-confusion-matrix-triage","steps":[{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s01-01-intro"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s02-02-choose-the-risk"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s03-03-predict-the-checks"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s04-04-fill-the-return"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s05-05-fix-the-overtrust"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s06-06-write-count-ready"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s07-07-checkpoint"}]}]},{"slug":"pytorch-tensors-and-autograd","lessons":[{"slug":"tensors-and-shapes","steps":[{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s01-01-intro"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s02-02-choose-the-risk"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s03-03-predict-the-checks"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s04-04-fill-the-return"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s05-05-fix-the-overtrust"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s06-06-write-count-ready"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s07-07-checkpoint"}]},{"slug":"broadcasting-and-vectorization","steps":[{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s01-01-intro"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s02-02-choose-the-risk"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s03-03-predict-the-checks"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s04-04-fill-the-return"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s05-05-fix-the-overtrust"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s06-06-write-count-ready"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s07-07-checkpoint"}]},{"slug":"autograd-mental-model","steps":[{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s01-01-intro"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s02-02-choose-the-risk"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s03-03-predict-the-checks"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s04-04-fill-the-return"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s05-05-fix-the-overtrust"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s06-06-write-count-ready"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s07-07-checkpoint"}]},{"slug":"mission-tensor-playground","steps":[{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s01-01-intro"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s02-02-choose-the-risk"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s03-03-predict-the-checks"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s04-04-fill-the-return"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s05-05-fix-the-overtrust"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s06-06-write-count-ready"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s07-07-checkpoint"}]}]},{"slug":"training-loops-and-optimizers","lessons":[{"slug":"loss-forward-backward-step","steps":[{"id":"training-loops-and-optimizers/loss-forward-backward-step/s01-01-intro"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s02-02-choose-the-risk"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s03-03-predict-the-checks"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s04-04-fill-the-return"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s05-05-fix-the-overtrust"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s06-06-write-count-ready"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s07-07-checkpoint"}]},{"slug":"gradient-descent-by-hand","steps":[{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s01-01-intro"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s02-02-choose-the-risk"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s03-03-predict-the-checks"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s04-04-fill-the-return"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s05-05-fix-the-overtrust"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s06-06-write-count-ready"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s07-07-checkpoint"}]},{"slug":"optimizers-and-schedulers","steps":[{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s01-01-intro"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s02-02-choose-the-risk"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s03-03-predict-the-checks"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s04-04-fill-the-return"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s05-05-fix-the-overtrust"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s06-06-write-count-ready"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s07-07-checkpoint"}]},{"slug":"checkpoints-and-reproducibility","steps":[{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s01-01-intro"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s02-02-choose-the-risk"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s03-03-predict-the-checks"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s04-04-fill-the-return"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s05-05-fix-the-overtrust"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s06-06-write-count-ready"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s07-07-checkpoint"}]},{"slug":"mission-overfit-then-recover","steps":[{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s01-01-intro"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s02-02-choose-the-risk"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s03-03-predict-the-checks"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s04-04-fill-the-return"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s05-05-fix-the-overtrust"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s06-06-write-count-ready"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s07-07-checkpoint"}]}]},{"slug":"deep-learning-architectures","lessons":[{"slug":"cnns-and-local-patterns","steps":[{"id":"deep-learning-architectures/cnns-and-local-patterns/s01-01-intro"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s02-02-choose-the-risk"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s03-03-predict-the-checks"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s04-04-fill-the-return"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s05-05-fix-the-overtrust"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s06-06-write-count-ready"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s07-07-checkpoint"}]},{"slug":"tokenizers-and-context-budget","steps":[{"id":"deep-learning-architectures/tokenizers-and-context-budget/s01-01-intro"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s02-02-choose-the-risk"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s03-03-predict-the-checks"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s04-04-fill-the-return"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s05-05-fix-the-overtrust"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s06-06-write-count-ready"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s07-07-checkpoint"}]},{"slug":"attention-and-transformer-blocks","steps":[{"id":"deep-learning-architectures/attention-and-transformer-blocks/s01-01-intro"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s02-02-choose-the-risk"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s03-03-predict-the-checks"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s04-04-fill-the-return"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s05-05-fix-the-overtrust"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s06-06-write-count-ready"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s07-07-checkpoint"}]},{"slug":"decoding-kv-cache-quantization","steps":[{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s01-01-intro"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s02-02-choose-the-risk"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s03-03-predict-the-checks"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s04-04-fill-the-return"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s05-05-fix-the-overtrust"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s06-06-write-count-ready"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s07-07-checkpoint"}]},{"slug":"mission-architecture-tradeoff-note","steps":[{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s01-01-intro"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s02-02-choose-the-risk"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s03-03-predict-the-checks"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s04-04-fill-the-return"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s05-05-fix-the-overtrust"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s06-06-write-count-ready"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s07-07-checkpoint"}]}]},{"slug":"feature-experiments-registries","lessons":[{"slug":"feature-pipeline-contracts","steps":[{"id":"feature-experiments-registries/feature-pipeline-contracts/s01-01-intro"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s02-02-choose-the-risk"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s03-03-predict-the-checks"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s04-04-fill-the-return"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s05-05-fix-the-overtrust"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s06-06-write-count-ready"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s07-07-checkpoint"}]},{"slug":"train-inference-skew","steps":[{"id":"feature-experiments-registries/train-inference-skew/s01-01-intro"},{"id":"feature-experiments-registries/train-inference-skew/s02-02-choose-the-risk"},{"id":"feature-experiments-registries/train-inference-skew/s03-03-predict-the-checks"},{"id":"feature-experiments-registries/train-inference-skew/s04-04-fill-the-return"},{"id":"feature-experiments-registries/train-inference-skew/s05-05-fix-the-overtrust"},{"id":"feature-experiments-registries/train-inference-skew/s06-06-write-count-ready"},{"id":"feature-experiments-registries/train-inference-skew/s07-07-checkpoint"}]},{"slug":"experiment-tracker-lite","steps":[{"id":"feature-experiments-registries/experiment-tracker-lite/s01-01-intro"},{"id":"feature-experiments-registries/experiment-tracker-lite/s02-02-choose-the-risk"},{"id":"feature-experiments-registries/experiment-tracker-lite/s03-03-predict-the-checks"},{"id":"feature-experiments-registries/experiment-tracker-lite/s04-04-fill-the-return"},{"id":"feature-experiments-registries/experiment-tracker-lite/s05-05-fix-the-overtrust"},{"id":"feature-experiments-registries/experiment-tracker-lite/s06-06-write-count-ready"},{"id":"feature-experiments-registries/experiment-tracker-lite/s07-07-checkpoint"}]},{"slug":"model-registry-shape","steps":[{"id":"feature-experiments-registries/model-registry-shape/s01-01-intro"},{"id":"feature-experiments-registries/model-registry-shape/s02-02-choose-the-risk"},{"id":"feature-experiments-registries/model-registry-shape/s03-03-predict-the-checks"},{"id":"feature-experiments-registries/model-registry-shape/s04-04-fill-the-return"},{"id":"feature-experiments-registries/model-registry-shape/s05-05-fix-the-overtrust"},{"id":"feature-experiments-registries/model-registry-shape/s06-06-write-count-ready"},{"id":"feature-experiments-registries/model-registry-shape/s07-07-checkpoint"}]},{"slug":"mission-feature-pipeline-and-tracker","steps":[{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s01-01-intro"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s02-02-choose-the-risk"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s03-03-predict-the-checks"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s04-04-fill-the-return"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s05-05-fix-the-overtrust"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s06-06-write-count-ready"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s07-07-checkpoint"}]}]},{"slug":"model-serving-and-mlops","lessons":[{"slug":"fastapi-inference-shape","steps":[{"id":"model-serving-and-mlops/fastapi-inference-shape/s01-01-intro"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s06-06-write-count-ready"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s07-07-checkpoint"}]},{"slug":"batch-vs-realtime-serving","steps":[{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s01-01-intro"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s06-06-write-count-ready"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s07-07-checkpoint"}]},{"slug":"docker-runtime-and-config","steps":[{"id":"model-serving-and-mlops/docker-runtime-and-config/s01-01-intro"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s06-06-write-count-ready"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s07-07-checkpoint"}]},{"slug":"github-actions-validation-gates","steps":[{"id":"model-serving-and-mlops/github-actions-validation-gates/s01-01-intro"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s06-06-write-failing-checks"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s07-07-checkpoint"}]},{"slug":"continuous-training-and-rollout","steps":[{"id":"model-serving-and-mlops/continuous-training-and-rollout/s01-01-intro"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s06-06-write-count-ready"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s07-07-checkpoint"}]},{"slug":"mission-fastapi-model-server","steps":[{"id":"model-serving-and-mlops/mission-fastapi-model-server/s01-01-intro"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s06-06-write-count-ready"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s07-07-checkpoint"}]}]},{"slug":"monitoring-cloud-portfolio","lessons":[{"slug":"structured-logs-and-alerts","steps":[{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s01-01-intro"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s06-06-write-count-ready"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s07-07-checkpoint"}]},{"slug":"data-and-concept-drift","steps":[{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s01-01-intro"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s06-06-write-count-ready"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s07-07-checkpoint"}]},{"slug":"retraining-triggers","steps":[{"id":"monitoring-cloud-portfolio/retraining-triggers/s01-01-intro"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s06-06-write-count-ready"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s07-07-checkpoint"}]},{"slug":"cloud-gpu-kubernetes-cost","steps":[{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s01-01-intro"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s06-06-write-count-ready"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s07-07-checkpoint"}]},{"slug":"portfolio-architecture-docs","steps":[{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s01-01-intro"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s06-06-write-count-ready"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s07-07-checkpoint"}]},{"slug":"mission-final-portfolio-ml-system","steps":[{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s01-01-intro"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s06-06-write-decide"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s07-07-checkpoint"}]}]},{"slug":"graphic-design-studio","lessons":[{"slug":"prompting-like-an-art-director","steps":[{"id":"graphic-design-studio/prompting-like-an-art-director/s01-01-the-prompt-is-a-creative-brief"},{"id":"graphic-design-studio/prompting-like-an-art-director/s02-02-style-language"},{"id":"graphic-design-studio/prompting-like-an-art-director/s03-03-which-tool-for-the-headline"},{"id":"graphic-design-studio/prompting-like-an-art-director/s04-04-predict-the-scaffold"},{"id":"graphic-design-studio/prompting-like-an-art-director/s05-05-negative-space-and-the-iteration-loop"},{"id":"graphic-design-studio/prompting-like-an-art-director/s06-06-fix-the-mood-word-prompt"},{"id":"graphic-design-studio/prompting-like-an-art-director/s07-07-write-prompt-linter"},{"id":"graphic-design-studio/prompting-like-an-art-director/s08-08-checkpoint"}]},{"slug":"the-design-kit","steps":[{"id":"graphic-design-studio/the-design-kit/s01-01-the-brand-bible-as-data"},{"id":"graphic-design-studio/the-design-kit/s02-02-fill-the-palette-line"},{"id":"graphic-design-studio/the-design-kit/s03-03-banned-looks"},{"id":"graphic-design-studio/the-design-kit/s04-04-predict-the-negatives"},{"id":"graphic-design-studio/the-design-kit/s05-05-which-field-failed"},{"id":"graphic-design-studio/the-design-kit/s06-06-fix-the-crossed-kit"},{"id":"graphic-design-studio/the-design-kit/s07-07-write-contrast-check"},{"id":"graphic-design-studio/the-design-kit/s08-08-tokens-for-the-interface"},{"id":"graphic-design-studio/the-design-kit/s09-09-write-token-audit"},{"id":"graphic-design-studio/the-design-kit/s10-10-checkpoint"}]},{"slug":"fifty-assets-one-brand","steps":[{"id":"graphic-design-studio/fifty-assets-one-brand/s01-01-the-consistency-problem"},{"id":"graphic-design-studio/fifty-assets-one-brand/s02-02-seeds-the-same-dice-twice"},{"id":"graphic-design-studio/fifty-assets-one-brand/s03-03-why-does-the-mascot-drift"},{"id":"graphic-design-studio/fifty-assets-one-brand/s04-04-predict-the-batch-plan"},{"id":"graphic-design-studio/fifty-assets-one-brand/s05-05-the-qa-filter"},{"id":"graphic-design-studio/fifty-assets-one-brand/s06-06-defend-the-pick"},{"id":"graphic-design-studio/fifty-assets-one-brand/s07-07-fix-the-qa-threshold"},{"id":"graphic-design-studio/fifty-assets-one-brand/s08-08-write-palette-distance"},{"id":"graphic-design-studio/fifty-assets-one-brand/s09-09-checkpoint"}]}]},{"slug":"campaign-studio","lessons":[{"slug":"voice-pack-as-data","steps":[{"id":"campaign-studio/voice-pack-as-data/s01-01-the-campaign-and-the-gate"},{"id":"campaign-studio/voice-pack-as-data/s02-02-lint-the-draft"},{"id":"campaign-studio/voice-pack-as-data/s03-03-predict-the-violations"},{"id":"campaign-studio/voice-pack-as-data/s04-04-testable-or-mood"},{"id":"campaign-studio/voice-pack-as-data/s05-05-fill-the-linter"},{"id":"campaign-studio/voice-pack-as-data/s06-06-variants-as-records"},{"id":"campaign-studio/voice-pack-as-data/s07-07-write-lint-copy"},{"id":"campaign-studio/voice-pack-as-data/s08-08-checkpoint"}]},{"slug":"batch-plan-and-calendar","steps":[{"id":"campaign-studio/batch-plan-and-calendar/s01-01-slots-before-prompts"},{"id":"campaign-studio/batch-plan-and-calendar/s02-02-which-field-is-missing"},{"id":"campaign-studio/batch-plan-and-calendar/s03-03-write-batch-plan"},{"id":"campaign-studio/batch-plan-and-calendar/s04-04-the-calendar-is-data"},{"id":"campaign-studio/batch-plan-and-calendar/s05-05-predict-the-queue"},{"id":"campaign-studio/batch-plan-and-calendar/s06-06-fix-the-gate"},{"id":"campaign-studio/batch-plan-and-calendar/s07-07-checkpoint"}]},{"slug":"the-numbers-pass","steps":[{"id":"campaign-studio/the-numbers-pass/s01-01-a-report-is-a-draft-over-data"},{"id":"campaign-studio/the-numbers-pass/s02-02-predict-the-verdict"},{"id":"campaign-studio/the-numbers-pass/s03-03-write-verify-numbers"},{"id":"campaign-studio/the-numbers-pass/s04-04-denominators-and-the-ftc"},{"id":"campaign-studio/the-numbers-pass/s05-05-what-may-not-ship"},{"id":"campaign-studio/the-numbers-pass/s06-06-fill-the-policy-router"},{"id":"campaign-studio/the-numbers-pass/s07-07-checkpoint"}]}]},{"slug":"support-studio","lessons":[{"slug":"the-policy-pack","steps":[{"id":"support-studio/the-policy-pack/s01-01-the-reply-that-binds-the-company"},{"id":"support-studio/the-policy-pack/s02-02-the-pack-as-data"},{"id":"support-studio/the-policy-pack/s03-03-spot-the-banned-promise"},{"id":"support-studio/the-policy-pack/s04-04-predict-the-lint"},{"id":"support-studio/the-policy-pack/s05-05-fix-the-stale-citation"},{"id":"support-studio/the-policy-pack/s06-06-the-review-loop-evidence"},{"id":"support-studio/the-policy-pack/s07-07-write-lint-draft"},{"id":"support-studio/the-policy-pack/s08-08-checkpoint"}]},{"slug":"golden-evals-earn-autonomy","steps":[{"id":"support-studio/golden-evals-earn-autonomy/s01-01-the-klarna-arc"},{"id":"support-studio/golden-evals-earn-autonomy/s02-02-a-macro-is-three-things"},{"id":"support-studio/golden-evals-earn-autonomy/s03-03-inside-the-lane"},{"id":"support-studio/golden-evals-earn-autonomy/s04-04-predict-the-eval-run"},{"id":"support-studio/golden-evals-earn-autonomy/s05-05-fill-the-promotion-rule"},{"id":"support-studio/golden-evals-earn-autonomy/s06-06-write-run-golden-evals"},{"id":"support-studio/golden-evals-earn-autonomy/s07-07-checkpoint"}]},{"slug":"the-escalation-boundary","steps":[{"id":"support-studio/the-escalation-boundary/s01-01-drafting-is-not-spending"},{"id":"support-studio/the-escalation-boundary/s02-02-triggers-as-conditions"},{"id":"support-studio/the-escalation-boundary/s03-03-route-this-ticket"},{"id":"support-studio/the-escalation-boundary/s04-04-predict-the-routing"},{"id":"support-studio/the-escalation-boundary/s05-05-fix-the-refund-hole"},{"id":"support-studio/the-escalation-boundary/s06-06-disclosure-and-the-kill-switch"},{"id":"support-studio/the-escalation-boundary/s07-07-write-route-ticket"},{"id":"support-studio/the-escalation-boundary/s08-08-checkpoint"}]}]},{"slug":"copy-studio","lessons":[{"slug":"the-voice-fingerprint","steps":[{"id":"copy-studio/the-voice-fingerprint/s01-01-the-client-you-inherit"},{"id":"copy-studio/the-voice-fingerprint/s02-02-the-fingerprint"},{"id":"copy-studio/the-voice-fingerprint/s03-03-predict-the-ceiling"},{"id":"copy-studio/the-voice-fingerprint/s04-04-the-samples-say-otherwise"},{"id":"copy-studio/the-voice-fingerprint/s05-05-fill-derive-the-rules"},{"id":"copy-studio/the-voice-fingerprint/s06-06-the-pack-with-provenance"},{"id":"copy-studio/the-voice-fingerprint/s07-07-write-fingerprint"},{"id":"copy-studio/the-voice-fingerprint/s08-08-checkpoint"}]},{"slug":"variants-that-hold-the-line","steps":[{"id":"copy-studio/variants-that-hold-the-line/s01-01-twelve-by-thursday"},{"id":"copy-studio/variants-that-hold-the-line/s02-02-the-drift-problem"},{"id":"copy-studio/variants-that-hold-the-line/s03-03-predict-the-drift-report"},{"id":"copy-studio/variants-that-hold-the-line/s04-04-which-subject-line-ships"},{"id":"copy-studio/variants-that-hold-the-line/s05-05-fix-the-channel-check"},{"id":"copy-studio/variants-that-hold-the-line/s06-06-the-kill-log"},{"id":"copy-studio/variants-that-hold-the-line/s07-07-write-voice-gate"},{"id":"copy-studio/variants-that-hold-the-line/s08-08-checkpoint"}]},{"slug":"the-claim-check","steps":[{"id":"copy-studio/the-claim-check/s01-01-forty-one-of-seventy-seven"},{"id":"copy-studio/the-claim-check/s02-02-what-counts-as-checkable"},{"id":"copy-studio/the-claim-check/s03-03-predict-the-claim-list"},{"id":"copy-studio/the-claim-check/s04-04-what-closes-a-row"},{"id":"copy-studio/the-claim-check/s05-05-fill-the-ship-decision"},{"id":"copy-studio/the-claim-check/s06-06-the-corrections-log"},{"id":"copy-studio/the-claim-check/s07-07-write-ship-gate"},{"id":"copy-studio/the-claim-check/s08-08-checkpoint"}]}]},{"slug":"analytics-studio","lessons":[{"slug":"the-cleaning-log","steps":[{"id":"analytics-studio/the-cleaning-log/s01-01-where-did-those-rows-go"},{"id":"analytics-studio/the-cleaning-log/s02-02-the-log-as-data"},{"id":"analytics-studio/the-cleaning-log/s03-03-refuse-the-silent-step"},{"id":"analytics-studio/the-cleaning-log/s04-04-predict-the-receipts"},{"id":"analytics-studio/the-cleaning-log/s05-05-fill-the-null-decision"},{"id":"analytics-studio/the-cleaning-log/s06-06-fix-the-vanishing-rows"},{"id":"analytics-studio/the-cleaning-log/s07-07-write-the-null-ledger"},{"id":"analytics-studio/the-cleaning-log/s08-08-checkpoint"}]},{"slug":"every-number-has-an-address","steps":[{"id":"analytics-studio/every-number-has-an-address/s01-01-the-plausible-wrong-number"},{"id":"analytics-studio/every-number-has-an-address/s02-02-sample-first-then-full"},{"id":"analytics-studio/every-number-has-an-address/s03-03-which-check-catches-it"},{"id":"analytics-studio/every-number-has-an-address/s04-04-predict-the-fan-out"},{"id":"analytics-studio/every-number-has-an-address/s05-05-fill-the-trace"},{"id":"analytics-studio/every-number-has-an-address/s06-06-write-audit-the-memo"},{"id":"analytics-studio/every-number-has-an-address/s07-07-checkpoint"}]},{"slug":"challenge-then-rerun","steps":[{"id":"analytics-studio/challenge-then-rerun/s01-01-the-confident-recommendation"},{"id":"analytics-studio/challenge-then-rerun/s02-02-the-triad-as-a-checklist"},{"id":"analytics-studio/challenge-then-rerun/s03-03-name-the-leg"},{"id":"analytics-studio/challenge-then-rerun/s04-04-predict-the-rerun"},{"id":"analytics-studio/challenge-then-rerun/s05-05-fix-the-drifting-memo"},{"id":"analytics-studio/challenge-then-rerun/s06-06-write-score-the-challenge"},{"id":"analytics-studio/challenge-then-rerun/s07-07-checkpoint"}]}]},{"slug":"delivery-studio","lessons":[{"slug":"the-spec-is-the-build","steps":[{"id":"delivery-studio/the-spec-is-the-build/s01-01-the-clarifying-question-is-gone"},{"id":"delivery-studio/the-spec-is-the-build/s02-02-the-ticket-as-data"},{"id":"delivery-studio/the-spec-is-the-build/s03-03-spot-the-untestable-check"},{"id":"delivery-studio/the-spec-is-the-build/s04-04-predict-the-spec-lint"},{"id":"delivery-studio/the-spec-is-the-build/s05-05-fill-the-scope-fence"},{"id":"delivery-studio/the-spec-is-the-build/s06-06-inside-or-outside-the-frontier"},{"id":"delivery-studio/the-spec-is-the-build/s07-07-write-lint-spec"},{"id":"delivery-studio/the-spec-is-the-build/s08-08-checkpoint"}]},{"slug":"acceptance-before-the-build","steps":[{"id":"delivery-studio/acceptance-before-the-build/s01-01-the-ninety-seven-thousand-dollar-skip"},{"id":"delivery-studio/acceptance-before-the-build/s02-02-evidence-against-the-checks"},{"id":"delivery-studio/acceptance-before-the-build/s03-03-which-check-catches-it"},{"id":"delivery-studio/acceptance-before-the-build/s04-04-fix-the-vibes-review"},{"id":"delivery-studio/acceptance-before-the-build/s05-05-predict-the-review-run"},{"id":"delivery-studio/acceptance-before-the-build/s06-06-rework-is-a-list-not-a-feeling"},{"id":"delivery-studio/acceptance-before-the-build/s07-07-write-review-deliverable"},{"id":"delivery-studio/acceptance-before-the-build/s08-08-checkpoint"}]},{"slug":"status-from-artifacts","steps":[{"id":"delivery-studio/status-from-artifacts/s01-01-watermelon-status"},{"id":"delivery-studio/status-from-artifacts/s02-02-derive-the-rag"},{"id":"delivery-studio/status-from-artifacts/s03-03-spot-the-artifact"},{"id":"delivery-studio/status-from-artifacts/s04-04-fill-the-worst-of"},{"id":"delivery-studio/status-from-artifacts/s05-05-predict-the-rollup"},{"id":"delivery-studio/status-from-artifacts/s06-06-the-not-ready-call"},{"id":"delivery-studio/status-from-artifacts/s07-07-write-readiness-gate"},{"id":"delivery-studio/status-from-artifacts/s08-08-checkpoint"}]},{"slug":"the-hours-saved-receipt","steps":[{"id":"delivery-studio/the-hours-saved-receipt/s01-01-the-token-chart-is-not-a-receipt"},{"id":"delivery-studio/the-hours-saved-receipt/s02-02-the-receipt-as-data"},{"id":"delivery-studio/the-hours-saved-receipt/s03-03-which-line-survives-staff-meeting"},{"id":"delivery-studio/the-hours-saved-receipt/s04-04-fix-the-vanity-chart"},{"id":"delivery-studio/the-hours-saved-receipt/s05-05-predict-the-receipt-run"},{"id":"delivery-studio/the-hours-saved-receipt/s06-06-only-accepted-ships-count"},{"id":"delivery-studio/the-hours-saved-receipt/s07-07-write-hours-receipt"},{"id":"delivery-studio/the-hours-saved-receipt/s08-08-checkpoint"}]}]},{"slug":"people-studio","lessons":[{"slug":"the-data-firewall","steps":[{"id":"people-studio/the-data-firewall/s01-01-the-rule-that-cannot-live-in-memory"},{"id":"people-studio/the-data-firewall/s02-02-the-firewall-as-data"},{"id":"people-studio/the-data-firewall/s03-03-which-payload-passes"},{"id":"people-studio/the-data-firewall/s04-04-predict-the-scan"},{"id":"people-studio/the-data-firewall/s05-05-fix-the-leaky-gate"},{"id":"people-studio/the-data-firewall/s06-06-employee-relations-takes-nothing"},{"id":"people-studio/the-data-firewall/s07-07-write-prompt-gate"},{"id":"people-studio/the-data-firewall/s08-08-checkpoint"}]},{"slug":"briefs-with-receipts","steps":[{"id":"people-studio/briefs-with-receipts/s01-01-the-most-automated-desk"},{"id":"people-studio/briefs-with-receipts/s02-02-a-claim-with-a-receipt"},{"id":"people-studio/briefs-with-receipts/s03-03-sourced-or-inference"},{"id":"people-studio/briefs-with-receipts/s04-04-predict-the-receipt-audit"},{"id":"people-studio/briefs-with-receipts/s05-05-fill-the-lookup"},{"id":"people-studio/briefs-with-receipts/s06-06-the-screening-line"},{"id":"people-studio/briefs-with-receipts/s07-07-write-audit-brief"},{"id":"people-studio/briefs-with-receipts/s08-08-checkpoint"}]},{"slug":"the-paper-trail","steps":[{"id":"people-studio/the-paper-trail/s01-01-policy-is-legal-surface"},{"id":"people-studio/the-paper-trail/s02-02-the-boundary-as-a-state-machine"},{"id":"people-studio/the-paper-trail/s03-03-who-signs-off"},{"id":"people-studio/the-paper-trail/s04-04-predict-the-version-log"},{"id":"people-studio/the-paper-trail/s05-05-fix-the-shortcut"},{"id":"people-studio/the-paper-trail/s06-06-the-evidence-trail"},{"id":"people-studio/the-paper-trail/s07-07-write-decision-record"},{"id":"people-studio/the-paper-trail/s08-08-checkpoint"}]}]},{"slug":"ops-studio","lessons":[{"slug":"the-sop-is-data","steps":[{"id":"ops-studio/the-sop-is-data/s01-01-the-two-weeks-notice-problem"},{"id":"ops-studio/the-sop-is-data/s02-02-the-sop-as-data"},{"id":"ops-studio/the-sop-is-data/s03-03-spot-the-invented-step"},{"id":"ops-studio/the-sop-is-data/s04-04-fill-the-sop-lint"},{"id":"ops-studio/the-sop-is-data/s05-05-predict-the-diff"},{"id":"ops-studio/the-sop-is-data/s06-06-the-diff-is-the-change-control"},{"id":"ops-studio/the-sop-is-data/s07-07-write-lint-sop"},{"id":"ops-studio/the-sop-is-data/s08-08-checkpoint"}]},{"slug":"auto-run-or-named-approval","steps":[{"id":"ops-studio/auto-run-or-named-approval/s01-01-the-checklist-that-lied"},{"id":"ops-studio/auto-run-or-named-approval/s02-02-the-schedule-is-data"},{"id":"ops-studio/auto-run-or-named-approval/s03-03-which-item-may-auto-run"},{"id":"ops-studio/auto-run-or-named-approval/s04-04-fix-the-silent-auto-complete"},{"id":"ops-studio/auto-run-or-named-approval/s05-05-predict-the-exception-queue"},{"id":"ops-studio/auto-run-or-named-approval/s06-06-exceptions-are-a-queue-not-a-shrug"},{"id":"ops-studio/auto-run-or-named-approval/s07-07-write-run-checklist"},{"id":"ops-studio/auto-run-or-named-approval/s08-08-checkpoint"}]},{"slug":"numbers-with-receipts","steps":[{"id":"ops-studio/numbers-with-receipts/s01-01-the-number-that-moves-money"},{"id":"ops-studio/numbers-with-receipts/s02-02-recompute-dont-trust"},{"id":"ops-studio/numbers-with-receipts/s03-03-which-claim-has-no-receipt"},{"id":"ops-studio/numbers-with-receipts/s04-04-predict-the-verification"},{"id":"ops-studio/numbers-with-receipts/s05-05-fill-the-receipt-check"},{"id":"ops-studio/numbers-with-receipts/s06-06-the-anomaly-is-a-question"},{"id":"ops-studio/numbers-with-receipts/s07-07-write-verify-report"},{"id":"ops-studio/numbers-with-receipts/s08-08-checkpoint"}]},{"slug":"the-pii-proxy","steps":[{"id":"ops-studio/the-pii-proxy/s01-01-the-ticket-that-left-the-building"},{"id":"ops-studio/the-pii-proxy/s02-02-mask-then-unmask"},{"id":"ops-studio/the-pii-proxy/s03-03-which-payload-may-leave"},{"id":"ops-studio/the-pii-proxy/s04-04-fill-the-leftover-check"},{"id":"ops-studio/the-pii-proxy/s05-05-predict-the-two-sides"},{"id":"ops-studio/the-pii-proxy/s06-06-unmask-is-a-named-human-act"},{"id":"ops-studio/the-pii-proxy/s07-07-write-the-proxy"},{"id":"ops-studio/the-pii-proxy/s08-08-checkpoint"}]}]},{"slug":"legal-studio","lessons":[{"slug":"the-cite-check-gate","steps":[{"id":"legal-studio/the-cite-check-gate/s01-01-the-brief-that-reads-perfectly"},{"id":"legal-studio/the-cite-check-gate/s02-02-three-verdicts-as-data"},{"id":"legal-studio/the-cite-check-gate/s03-03-which-cite-needs-no-check"},{"id":"legal-studio/the-cite-check-gate/s04-04-predict-the-verification-run"},{"id":"legal-studio/the-cite-check-gate/s05-05-fix-the-verifier-that-asks-again"},{"id":"legal-studio/the-cite-check-gate/s06-06-write-verify-authorities"},{"id":"legal-studio/the-cite-check-gate/s07-07-checkpoint"}]},{"slug":"clause-extraction-with-pinpoints","steps":[{"id":"legal-studio/clause-extraction-with-pinpoints/s01-01-a-claim-without-a-pinpoint-is-a-rumor"},{"id":"legal-studio/clause-extraction-with-pinpoints/s02-02-the-playbook-as-data"},{"id":"legal-studio/clause-extraction-with-pinpoints/s03-03-which-row-is-unusable"},{"id":"legal-studio/clause-extraction-with-pinpoints/s04-04-predict-the-triage"},{"id":"legal-studio/clause-extraction-with-pinpoints/s05-05-fill-the-walk-away-route"},{"id":"legal-studio/clause-extraction-with-pinpoints/s06-06-write-triage-extraction"},{"id":"legal-studio/clause-extraction-with-pinpoints/s07-07-checkpoint"}]},{"slug":"privilege-rules-and-the-review-log","steps":[{"id":"legal-studio/privilege-rules-and-the-review-log/s01-01-privilege-is-a-routing-question"},{"id":"legal-studio/privilege-rules-and-the-review-log/s02-02-the-routing-table"},{"id":"legal-studio/privilege-rules-and-the-review-log/s03-03-which-request-is-the-waiver-risk"},{"id":"legal-studio/privilege-rules-and-the-review-log/s04-04-predict-the-router"},{"id":"legal-studio/privilege-rules-and-the-review-log/s05-05-fix-the-privilege-hole"},{"id":"legal-studio/privilege-rules-and-the-review-log/s06-06-the-log-that-survives-a-motion"},{"id":"legal-studio/privilege-rules-and-the-review-log/s07-07-write-log-ready"},{"id":"legal-studio/privilege-rules-and-the-review-log/s08-08-checkpoint"}]}]},{"slug":"agent-studio","lessons":[{"slug":"the-tool-contract","steps":[{"id":"agent-studio/the-tool-contract/s01-01-the-feature-you-ship-today"},{"id":"agent-studio/the-tool-contract/s02-02-the-schema-is-the-blast-radius"},{"id":"agent-studio/the-tool-contract/s03-03-spot-the-loose-tool"},{"id":"agent-studio/the-tool-contract/s04-04-predict-the-validation"},{"id":"agent-studio/the-tool-contract/s05-05-fix-the-yolo-rerun"},{"id":"agent-studio/the-tool-contract/s06-06-write-validate-call"},{"id":"agent-studio/the-tool-contract/s07-07-checkpoint"}]},{"slug":"the-loop-with-a-meter","steps":[{"id":"agent-studio/the-loop-with-a-meter/s01-01-a-loop-that-cannot-stop-is-an-invoice"},{"id":"agent-studio/the-loop-with-a-meter/s02-02-caps-are-part-of-the-loop"},{"id":"agent-studio/the-loop-with-a-meter/s03-03-why-did-it-stop"},{"id":"agent-studio/the-loop-with-a-meter/s04-04-predict-the-stop-reason"},{"id":"agent-studio/the-loop-with-a-meter/s05-05-fill-the-budget-check"},{"id":"agent-studio/the-loop-with-a-meter/s06-06-the-trace-is-the-receipt"},{"id":"agent-studio/the-loop-with-a-meter/s07-07-write-run-loop"},{"id":"agent-studio/the-loop-with-a-meter/s08-08-checkpoint"}]},{"slug":"evals-before-it-works","steps":[{"id":"agent-studio/evals-before-it-works/s01-01-green-before-it-runs"},{"id":"agent-studio/evals-before-it-works/s02-02-a-golden-case-is-a-past-incident"},{"id":"agent-studio/evals-before-it-works/s03-03-which-case-catches-it"},{"id":"agent-studio/evals-before-it-works/s04-04-predict-the-eval-run"},{"id":"agent-studio/evals-before-it-works/s05-05-fix-the-router-to-green"},{"id":"agent-studio/evals-before-it-works/s06-06-the-ratchet-entry"},{"id":"agent-studio/evals-before-it-works/s07-07-write-lint-the-ratchet"},{"id":"agent-studio/evals-before-it-works/s08-08-checkpoint"}]}]}]},"detail":{"number":16,"slug":"agent-loops","title":"agent loops — tool_use and the request/tool/respond cycle","blurb":"an agent isn't magic. it's a while loop. learn the actual cycle claude code, cursor, and every other agent uses: model returns tool_use, you run the tool, you send the result back, repeat until end_turn.","overview":"$27","status":"live","lessons":[{"slug":"the-loop","title":"stop_reason, tool_use, tool_result — the loop every agent runs","estMinutes":9,"prerequisites":[],"status":"live","steps":[{"type":"read","id":"agent-loops/the-loop/s01-01-intro","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":95,"concept":"agent-loop-overview","body":"$28","cta":"Got it","code":"# one round-trip in an agent loop, hardcoded.\n# the model returned a tool_use block instead of stopping.\nresponse = {\n \"role\": \"assistant\",\n \"stop_reason\": \"tool_use\",\n \"content\": [\n {\"type\": \"text\", \"text\": \"Let me check the weather.\"},\n {\n \"type\": \"tool_use\",\n \"id\": \"toolu_01\",\n \"name\": \"get_weather\",\n \"input\": {\"city\": \"Tokyo\"},\n },\n ],\n}\n\n# is the model done, or do we owe it a tool result?\nif response[\"stop_reason\"] == \"tool_use\":\n # find the tool_use block in content\n for block in response[\"content\"]:\n if block[\"type\"] == \"tool_use\":\n print(f\"call {block['name']} with {block['input']}\")\n print(f\"tool_use_id = {block['id']}\")\n","runnable":true},{"type":"mc","id":"agent-loops/the-loop/s02-02-which-stop-reason","xp":2,"hint":[],"personalize":false,"phase":"warmup","estSeconds":50,"concept":"stop-reason-tool-use","prompt":"The agent loop just got back a response with `stop_reason ==\n\"tool_use\"`. What does your code do next?\n","options":[{"id":"a","label":"Print the text and exit","explain":"That's what `end_turn` means. `tool_use` means the model is asking you to run a tool — exiting now skips the actual work."},{"id":"b","label":"Run the requested tool, append a tool_result block, and call the model again"},{"id":"c","label":"Retry the same call","explain":"Re-sending the identical messages will produce the same tool_use response. You'd loop forever without ever running the tool."},{"id":"d","label":"Raise an error","explain":"`tool_use` isn't an error condition — it's the normal mid-conversation signal. Errors come back as exceptions or `isError: true` from MCP."}],"answerIds":["b"],"shuffle":false,"code":"# the model's response includes both a text block and a tool_use block\nresponse = {\n \"stop_reason\": \"tool_use\",\n \"content\": [\n {\"type\": \"text\", \"text\": \"I'll look that up.\"},\n {\"type\": \"tool_use\", \"id\": \"toolu_a\", \"name\": \"search\", \"input\": {\"q\": \"best ramen tokyo\"}},\n ],\n}\nprint(response[\"stop_reason\"])\n","runnable":true},{"type":"read","id":"agent-loops/the-loop/s03-03-tool-result-shape","xp":2,"hint":[],"personalize":false,"phase":"build","estSeconds":80,"concept":"tool-result-message-shape","body":"$29","cta":"Got it","code":"# the conversation so far — what the agent loop has accumulated.\n# note how the assistant's tool_use becomes the user's tool_result\n# in the very next turn.\n\nmessages = [\n {\"role\": \"user\", \"content\": \"What's the weather in Tokyo?\"},\n {\n \"role\": \"assistant\",\n \"content\": [\n {\"type\": \"text\", \"text\": \"Let me check.\"},\n {\n \"type\": \"tool_use\",\n \"id\": \"toolu_01\",\n \"name\": \"get_weather\",\n \"input\": {\"city\": \"Tokyo\"},\n },\n ],\n },\n {\n \"role\": \"user\",\n \"content\": [\n {\n \"type\": \"tool_result\",\n \"tool_use_id\": \"toolu_01\",\n \"content\": \"72F, partly cloudy\",\n }\n ],\n },\n]\n\nfor turn in messages:\n print(turn[\"role\"])\n","runnable":true},{"type":"predict","id":"agent-loops/the-loop/s04-04-predict-the-next-call","xp":3,"hint":[{"level":1,"body":"The loop calls the model, increments `iterations`, then checks `stop_reason`. It exits on `end_turn`, which is the third call.","cost":0},{"level":2,"body":"Three iterations: tool_use, tool_use, end_turn.","cost":0}],"personalize":false,"phase":"build","estSeconds":80,"concept":"agent-loop-iteration-count","code":"# mock a tiny agent loop. fake_model returns tool_use twice, then end_turn.\ncall_count = {\"n\": 0}\n\ndef fake_model(messages):\n call_count[\"n\"] += 1\n if call_count[\"n\"] < 3:\n return {\"stop_reason\": \"tool_use\"}\n return {\"stop_reason\": \"end_turn\"}\n\nmessages = [{\"role\": \"user\", \"content\": \"do the thing\"}]\niterations = 0\nwhile True:\n response = fake_model(messages)\n iterations += 1\n if response[\"stop_reason\"] == \"end_turn\":\n break\n messages.append({\"role\": \"assistant\", \"content\": []})\n messages.append({\"role\": \"user\", \"content\": [{\"type\": \"tool_result\"}]})\n\nprint(f\"iterations: {iterations}\")\n","prompt":"Read the editor. The fake model returns `tool_use` on calls 1 and 2,\nthen `end_turn` on call 3. How many times does the loop iterate\nbefore exiting?\n","grader":{"kind":"stdout-equality","expected":"iterations: 3","normalize":"collapse-trailing-newline"}},{"type":"fill","id":"agent-loops/the-loop/s05-05-fill-the-stop-condition","xp":3,"hint":[{"level":1,"body":"Two-word string with an underscore. It signals the model is done speaking.","cost":0},{"level":2,"body":"`end_turn` — the standard \"done\" signal in the messages API.","cost":0}],"personalize":false,"phase":"build","estSeconds":65,"concept":"end-turn-stop-reason","prompt":"Cursor wrote an agent loop and left the exit condition unfilled.\nThe loop should break when the model is finished — that is, when\nthe response's `stop_reason` equals which string?\n\nStop reason: ___\n","code":"def fake_model(messages):\n return {\"stop_reason\": \"end_turn\", \"content\": [{\"type\": \"text\", \"text\": \"done\"}]}\n\nmessages = [{\"role\": \"user\", \"content\": \"hi\"}]\nwhile True:\n response = fake_model(messages)\n if response[\"stop_reason\"] == \"___\":\n print(response[\"content\"][0][\"text\"])\n break\n # ... handle tool_use ...\n","blanks":[{"id":"keyword","accept":["end_turn"],"caseSensitive":true,"normalize":"trim"}]},{"type":"fix","id":"agent-loops/the-loop/s06-06-fix-the-tool-id","xp":4,"hint":[{"level":1,"body":"Read the id from the model's `tool_use` block instead of hardcoding `\"1\"`.","cost":0},{"level":2,"body":"Change line 9 to `\"tool_use_id\": tool_use_block[\"id\"],`. The id flows from the model, not your code.","cost":0}],"personalize":false,"phase":"build","estSeconds":95,"concept":"tool-use-id-must-match","brokenCode":"tool_use_block = {\n \"type\": \"tool_use\",\n \"id\": \"toolu_42\",\n \"name\": \"search\",\n \"input\": {\"q\": \"tokyo\"},\n}\n\nreply = {\n \"tool_use_id\": \"1\",\n \"content\": \"ok\",\n}\n\nif reply[\"tool_use_id\"] != tool_use_block[\"id\"]:\n raise ValueError(\"tool_use_id mismatch\")\nprint(\"matched\")\n","prompt":"The agent loop built a `tool_result` with a hardcoded `tool_use_id`\nof `\"1\"`, but the model emitted `\"toolu_42\"`. The fake server raises\na `ValueError` on mismatch. Fix line 9 so the id is read from the\n`tool_use` block dynamically.\n\nExpected output:\n```\nmatched\n```\n","grader":{"kind":"stdout-equality","expected":"matched","normalize":"collapse-trailing-newline"},"bugLines":[9],"revealAfter":4},{"type":"fix","id":"agent-loops/the-loop/s07-07-fix-the-loop-exit","xp":4,"hint":[{"level":1,"body":"The break condition is inverted. `tool_use` means \"I want to call a tool, keep the loop running so you can give me the result.\" `end_turn` means \"I'm done, exit.\"","cost":0},{"level":2,"body":"Change line 14 to `if response[\"stop_reason\"] == \"end_turn\":`. The loop body (append the assistant response, fetch next) only runs on tool_use turns.","cost":0}],"personalize":false,"phase":"build","estSeconds":100,"concept":"agent-loop-must-check-stop-reason","brokenCode":"responses = iter([\n {\"stop_reason\": \"tool_use\", \"content\": []},\n {\"stop_reason\": \"end_turn\", \"content\": [{\"type\": \"text\", \"text\": \"done\"}]},\n])\n\ndef fake_model(messages):\n return next(responses)\n\nmessages = [{\"role\": \"user\", \"content\": \"go\"}]\n\nwhile len(messages) < 10:\n response = fake_model(messages)\n if response[\"stop_reason\"] == \"tool_use\":\n break\n messages.append({\"role\": \"assistant\", \"content\": response[\"content\"]})\n\ntext_blocks = [b for b in response[\"content\"] if b.get(\"type\") == \"text\"]\nprint(text_blocks[0][\"text\"] if text_blocks else \"no text\")\n","prompt":"AI wrote an agent loop that BREAKS on `tool_use`. That's backwards —\n`tool_use` is the signal to KEEP GOING (run the tool, append the\nresult, loop again). `end_turn` is the signal to stop. The fake\nmodel returns one `tool_use` then one `end_turn`. The broken loop\nbails on the first response and never sees `\"done\"`.\n\nFix line 12 so the loop breaks on `end_turn`, runs the loop body\nfor the `tool_use` turn, and prints the final text.\n\nExpected output:\n```\ndone\n```\n","grader":{"kind":"stdout-equality","expected":"done","normalize":"collapse-trailing-newline"},"bugLines":[14],"revealAfter":5},{"type":"write","id":"agent-loops/the-loop/s08-08-write-the-loop","xp":5,"hint":[{"level":1,"body":"Loop with `for _ in range(5)`. Inside: call the model, branch on `stop_reason`. On `end_turn` return the text. On `tool_use` append both turns.","cost":0},{"level":2,"body":"When appending the tool_result, read `tool_use_id` from `response[\"content\"][0][\"id\"]` — never hardcode it.","cost":0}],"personalize":false,"phase":"build","estSeconds":150,"concept":"agent-loop-implementation","prompt":"Build the actual agent loop. Write `run_agent(question)` that:\n\n- Starts with a `messages` list containing one user turn with `question`.\n- Loops up to 5 times. Each iteration calls `fake_model(messages)`.\n- If `response[\"stop_reason\"]` is `\"end_turn\"`, return the text from\n `response[\"content\"][0][\"text\"]`.\n- If it's `\"tool_use\"`, append the assistant turn AND a user turn\n containing a `tool_result` with `content=\"42\"` (we'll pretend every\n tool returns 42). Then loop again.\n- If the cap is hit without `end_turn`, return the string `\"capped\"`.\n\nThen call it once and print the result.\n\nExpected output:\n```\nthe answer is 42\n```\n","starter":"call_count = {\"n\": 0}\n\ndef fake_model(messages):\n call_count[\"n\"] += 1\n if call_count[\"n\"] == 1:\n return {\n \"stop_reason\": \"tool_use\",\n \"content\": [{\"type\": \"tool_use\", \"id\": \"toolu_a\", \"name\": \"calc\", \"input\": {}}],\n }\n return {\n \"stop_reason\": \"end_turn\",\n \"content\": [{\"type\": \"text\", \"text\": \"the answer is 42\"}],\n }\n\n# define run_agent(question) below\n\nprint(run_agent(\"what's the meaning of life?\"))\n","grader":{"kind":"stdout-equality","expected":"the answer is 42","normalize":"collapse-trailing-newline"},"solution":"$2a","hiddenTests":[]},{"type":"checkpoint","id":"agent-loops/the-loop/s09-09-checkpoint","xp":8,"hint":[],"personalize":false,"phase":"check","estSeconds":180,"concept":"agent-loop-checkpoint","prompt":"Final drill. Build a slightly bigger agent loop. Write\n`run_agent(question, max_iterations)` that:\n\n- Maintains a `messages` list, starting with one user turn.\n- Calls `fake_model(messages)` up to `max_iterations` times.\n- On `stop_reason == \"end_turn\"`: return the dict\n `{\"ok\": True, \"answer\": , \"rounds\": }`.\n- On `stop_reason == \"tool_use\"`: run the tool by calling\n `fake_tool(name, input)`, append the assistant turn + a user turn\n with a `tool_result` carrying the tool's output, increment count,\n and continue.\n- If `max_iterations` is exhausted: return\n `{\"ok\": False, \"answer\": \"capped\", \"rounds\": max_iterations}`.\n\nThen the script will run two cases and print each result.\n\nExpected output:\n```\nok=True rounds=2 answer=area is 50\nok=False rounds=2 answer=capped\n```\n","starter":"$2b","grader":{"kind":"stdout-equality","expected":"ok=True rounds=2 answer=area is 50\nok=False rounds=2 answer=capped\n","normalize":"collapse-trailing-newline"},"solution":"$2c","passThreshold":0.66}],"xpTotal":32},{"slug":"multi-step-tools","title":"Multi-step tools — when one tool isn't enough","estMinutes":11,"prerequisites":["the-loop"],"status":"live","steps":[{"type":"read","id":"agent-loops/multi-step-tools/s01-01-intro","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":110,"concept":"tool-registry-pattern","body":"$2d","cta":"Got it","code":"# the model gets a *list* of tools each turn. it picks one (or several).\n# your job is to dispatch the call to the right Python function.\n\nTOOLS = {\n \"search\": lambda q: f\"3 results for {q}\",\n \"read_page\": lambda url: f\"

\",\n \"summarize\": lambda text: f\"summary: {text[:20]}...\",\n}\n\n# one assistant turn — the model picked `search`.\ntool_use = {\"type\": \"tool_use\", \"id\": \"toolu_a\", \"name\": \"search\", \"input\": {\"q\": \"ramen tokyo\"}}\n\n# dispatch is a one-line lookup, not a giant if/elif.\nfn = TOOLS[tool_use[\"name\"]]\nresult = fn(**tool_use[\"input\"])\nprint(result)\n","runnable":true},{"type":"mc","id":"agent-loops/multi-step-tools/s02-02-which-tool-runs","xp":2,"hint":[{"level":1,"body":"Read the Anthropic docs on parallel tool use: one assistant turn → one user turn that bundles a tool_result per tool_use_id.","cost":0}],"personalize":false,"phase":"warmup","estSeconds":50,"concept":"tool-dispatch-by-name","prompt":"The assistant turn above contains three `tool_use` blocks in a single\nresponse. Your loop is about to handle this turn. What does it do?\n","options":[{"id":"a","label":"Pick the first tool, run it, ignore the rest","explain":"You'd silently drop two of the model's requests and the next API call would fail — the model expects a `tool_result` for every `tool_use_id` it emitted."},{"id":"b","label":"Run all three tools and reply with one user turn containing three `tool_result` blocks"},{"id":"c","label":"Send three separate user turns, one tool_result each","explain":"The protocol pairs each model turn with one user turn. Multiple results go *inside* one user turn as a list of `tool_result` blocks."},{"id":"d","label":"Refuse and ask the model to pick just one tool at a time","explain":"Parallel tool calls are a normal feature, not a misuse. The model emits them when it knows the calls are independent."}],"answerIds":["b"],"shuffle":false,"code":"# the model returned three tool_use blocks in ONE assistant turn.\nTOOLS = {\n \"search\": lambda q: f\"hits for {q}\",\n \"read_page\": lambda url: f\"<{url}>\",\n \"summarize\": lambda text: text[:10],\n}\n\nassistant_turn = {\n \"stop_reason\": \"tool_use\",\n \"content\": [\n {\"type\": \"tool_use\", \"id\": \"toolu_1\", \"name\": \"search\", \"input\": {\"q\": \"ramen\"}},\n {\"type\": \"tool_use\", \"id\": \"toolu_2\", \"name\": \"read_page\", \"input\": {\"url\": \"https://x.com\"}},\n {\"type\": \"tool_use\", \"id\": \"toolu_3\", \"name\": \"summarize\", \"input\": {\"text\": \"long story\"}},\n ],\n}\n\nfor block in assistant_turn[\"content\"]:\n print(block[\"name\"])\n","runnable":true},{"type":"read","id":"agent-loops/multi-step-tools/s03-03-tool-registry","xp":2,"hint":[],"personalize":false,"phase":"build","estSeconds":90,"concept":"tool-definitions-vs-tool-functions","body":"$2e","cta":"Got it","code":"# two halves of \"a tool\". the model needs the SCHEMA to know it exists.\n# your code needs the FUNCTION to actually run it.\n\nTOOL_DEFINITIONS = [\n {\n \"name\": \"search\",\n \"description\": \"Search the web. Returns up to 3 result URLs.\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\"q\": {\"type\": \"string\"}},\n \"required\": [\"q\"],\n },\n },\n {\n \"name\": \"read_page\",\n \"description\": \"Fetch a URL and return its plain text.\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\"url\": {\"type\": \"string\"}},\n \"required\": [\"url\"],\n },\n },\n]\n\nTOOL_FUNCTIONS = {\n \"search\": lambda q: f\"3 results for {q}\",\n \"read_page\": lambda url: f\"\",\n}\n\n# send TOOL_DEFINITIONS to the model. dispatch through TOOL_FUNCTIONS.\nfor d in TOOL_DEFINITIONS:\n print(f\"{d['name']}: {d['description']}\")\n","runnable":true},{"type":"predict","id":"agent-loops/multi-step-tools/s04-04-predict-the-chain","xp":3,"hint":[{"level":1,"body":"The loop only prints inside the `end_turn` branch. The third response is `end_turn` with one text block.","cost":0},{"level":2,"body":"`r[\"content\"][0][\"text\"]` on the third iteration.","cost":0}],"personalize":false,"phase":"build","estSeconds":80,"concept":"tool-chain-by-model","code":"$2f","prompt":"Read the editor. The fake model returns three responses in sequence:\n`search` → `read_page` → `end_turn`. The loop dispatches each tool\nthrough the `TOOLS` registry. What single line of text gets printed?\n","grader":{"kind":"stdout-equality","expected":"found a 2026 ramen guide","normalize":"collapse-trailing-newline"}},{"type":"fill","id":"agent-loops/multi-step-tools/s05-05-fill-the-dispatch","xp":3,"hint":[{"level":1,"body":"Two pieces: look up the function with `TOOLS[name]`, then call it with `**input` to unpack the dict as keyword arguments.","cost":0},{"level":2,"body":"`TOOLS[tool_use[\"name\"]](**tool_use[\"input\"])` — registry lookup, then unpack-call.","cost":0}],"personalize":false,"phase":"build","estSeconds":65,"concept":"tool-dispatch-via-registry","prompt":"The agent loop knows the model picked a tool, but the dispatch line\nis unfilled. Look up the function in `TOOLS` by the tool's `name`\nand call it with the tool's `input` dict unpacked as kwargs.\n\nFill in the missing expression so the loop prints `3 hits for ramen`.\n\nExpected output:\n```\n3 hits for ramen\n```\n","code":"TOOLS = {\n \"search\": lambda q: f\"3 hits for {q}\",\n}\n\ntool_use = {\"type\": \"tool_use\", \"id\": \"t1\", \"name\": \"search\", \"input\": {\"q\": \"ramen\"}}\n\nresult = ___\nprint(result)\n","blanks":[{"id":"dispatch","accept":["TOOLS[tool_use[\"name\"]](**tool_use[\"input\"])","TOOLS[tool_use['name']](**tool_use['input'])"],"caseSensitive":true,"normalize":"trim"}]},{"type":"fix","id":"agent-loops/multi-step-tools/s06-06-fix-the-missing-tool","xp":4,"hint":[{"level":1,"body":"Use `TOOLS.get(name)` instead of `TOOLS[name]`. If it returns `None`, build the error string yourself instead of calling.","cost":0},{"level":2,"body":"Replace the dispatch line with:\n\n```python\nfn = TOOLS.get(tool_use[\"name\"])\nresult = fn(**tool_use[\"input\"]) if fn else f\"unknown tool: {tool_use['name']}\"\n```\n","cost":0}],"personalize":false,"phase":"build","estSeconds":90,"concept":"unknown-tool-fail-loud","brokenCode":"TOOLS = {\n \"search\": lambda q: f\"hits for {q}\",\n}\n\ntool_use = {\n \"type\": \"tool_use\",\n \"id\": \"t1\",\n \"name\": \"web_search\", # not in TOOLS!\n \"input\": {\"q\": \"ramen\"},\n}\n\n# this line crashes with KeyError on unknown names.\nresult = TOOLS[tool_use[\"name\"]](**tool_use[\"input\"])\nprint(result)\n","prompt":"The model hallucinated a tool name — `web_search` — that isn't in\nthe registry. The current dispatch raises `KeyError` deep inside\nthe loop, which crashes the agent and loses the whole conversation.\n\nFix the dispatch so an unknown tool returns the string\n`\"unknown tool: \"` as the tool result instead. The model can\nthen read that and recover (it usually retries with a real tool\nname).\n\nExpected output:\n```\nunknown tool: web_search\n```\n","grader":{"kind":"stdout-equality","expected":"unknown tool: web_search","normalize":"collapse-trailing-newline"},"bugLines":[12],"revealAfter":4},{"type":"fix","id":"agent-loops/multi-step-tools/s07-07-fix-the-parallel-ids","xp":4,"hint":[{"level":1,"body":"The id of *this* tool_use block is `tu[\"id\"]`, not the literal `\"t1\"`. Read it from the loop variable.","cost":0},{"level":2,"body":"Change line 14 to `\"tool_use_id\": tu[\"id\"],` — every tool_result carries the id of the tool_use it answers.","cost":0}],"personalize":false,"phase":"build","estSeconds":110,"concept":"parallel-tool-result-ids","brokenCode":"TOOLS = {\n \"read_page\": lambda url: f\"<{url}>\",\n}\n\nassistant_content = [\n {\"type\": \"tool_use\", \"id\": \"t1\", \"name\": \"read_page\", \"input\": {\"url\": \"a.com\"}},\n {\"type\": \"tool_use\", \"id\": \"t2\", \"name\": \"read_page\", \"input\": {\"url\": \"b.com\"}},\n]\n\nresults = []\nfor tu in assistant_content:\n out = TOOLS[tu[\"name\"]](**tu[\"input\"])\n results.append({\n \"type\": \"tool_result\",\n \"tool_use_id\": \"t1\", # bug: hardcoded\n \"content\": out,\n })\n\nprint([r[\"tool_use_id\"] for r in results])\n","prompt":"The model returned two `tool_use` blocks in one assistant turn —\nclassic parallel tool calls. The reply builder runs both tools but\nhardcodes both `tool_use_id` values to `\"t1\"`. The next API call\nwill 400 because tool ids must match the originals (`t1` and `t2`).\n\nFix the loop so each tool result carries its own original id.\n\nExpected output:\n```\n['t1', 't2']\n```\n","grader":{"kind":"stdout-equality","expected":"['t1', 't2']","normalize":"collapse-trailing-newline"},"bugLines":[14],"revealAfter":4},{"type":"write","id":"agent-loops/multi-step-tools/s08-08-write-multi-tool-agent","xp":5,"hint":[{"level":1,"body":"Same loop shape as `01-the-loop`. The new part: iterate over every `tool_use` block in `response[\"content\"]`, dispatch through `TOOLS`, and collect a list of `tool_result` blocks for one user turn.","cost":0},{"level":2,"body":"Filter `response[\"content\"]` to just `tool_use` blocks, build a parallel list of `tool_result` blocks (one per id), and put that whole list in `{\"role\": \"user\", \"content\": tool_results}`.","cost":0}],"personalize":false,"phase":"build","estSeconds":180,"concept":"multi-tool-agent-loop","prompt":"Build the multi-tool agent. Write `run_agent(question)` that:\n\n- Starts with a `messages` list containing one user turn with `question`.\n- Loops up to 5 times, calling `fake_model(messages)` each iteration.\n- On `stop_reason == \"end_turn\"`: return `response[\"content\"][0][\"text\"]`.\n- On `stop_reason == \"tool_use\"`: there may be ONE OR MORE `tool_use`\n blocks in `response[\"content\"]`. For each one, look up the tool in\n the `TOOLS` registry and call it with `**block[\"input\"]`. Build a\n list of `tool_result` blocks (each with the matching `tool_use_id`)\n and append both turns:\n\n 1. The full assistant turn (`{\"role\": \"assistant\", \"content\": response[\"content\"]}`)\n 2. One user turn whose `content` is the list of tool_result blocks.\n\n- If the cap is hit without `end_turn`, return `\"capped\"`.\n\nThen call it once and print the result.\n\nExpected output:\n```\nfound: Tokyo's best ramen shops, 2026 guide.\n```\n","starter":"TOOLS = {\n \"search\": lambda q: \"https://news.example/ramen\",\n \"read_page\": lambda url: \"Tokyo's best ramen shops, 2026 guide.\",\n}\n\nresponses = iter([\n {\"stop_reason\": \"tool_use\", \"content\": [\n {\"type\": \"tool_use\", \"id\": \"t1\", \"name\": \"search\", \"input\": {\"q\": \"ramen tokyo\"}}\n ]},\n {\"stop_reason\": \"tool_use\", \"content\": [\n {\"type\": \"tool_use\", \"id\": \"t2\", \"name\": \"read_page\",\n \"input\": {\"url\": \"https://news.example/ramen\"}}\n ]},\n {\"stop_reason\": \"end_turn\", \"content\": [\n {\"type\": \"text\", \"text\": \"found: Tokyo's best ramen shops, 2026 guide.\"}\n ]},\n])\n\ndef fake_model(messages):\n return next(responses)\n\n# define run_agent(question) below\n\nprint(run_agent(\"find the latest ramen news\"))\n","grader":{"kind":"stdout-equality","expected":"found: Tokyo's best ramen shops, 2026 guide.","normalize":"collapse-trailing-newline"},"solution":"$30","hiddenTests":[]},{"type":"checkpoint","id":"agent-loops/multi-step-tools/s09-09-checkpoint","xp":8,"hint":[],"personalize":false,"phase":"check","estSeconds":200,"concept":"multi-tool-agent-checkpoint","prompt":"$31","starter":"$32","grader":{"kind":"stdout-equality","expected":"ok=True rounds=3 tools=3 answer=found 2 sources on ramen\nok=False rounds=2 tools=2 answer=capped\n","normalize":"collapse-trailing-newline"},"solution":"$33","passThreshold":0.66}],"xpTotal":32},{"slug":"routing","title":"Routing — pick the path before doing the work","estMinutes":11,"prerequisites":["the-loop","multi-step-tools"],"status":"live","steps":[{"type":"read","id":"agent-loops/routing/s01-01-intro","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":110,"concept":"routing-pattern","body":"$34","cta":"Got it","code":"# routing in five lines: ask a small model \"what is this?\", then dispatch.\n\ndef classify(question):\n # in real code, this is an LLM call constrained to one of the keys.\n if \"refund\" in question.lower(): return \"billing\"\n if \"error\" in question.lower(): return \"technical\"\n return \"general\"\n\nROUTES = {\n \"billing\": lambda q: f\"[billing agent] processing: {q}\",\n \"technical\": lambda q: f\"[tech agent] debugging: {q}\",\n \"general\": lambda q: f\"[general agent] answering: {q}\",\n}\n\ndef route(question):\n category = classify(question)\n handler = ROUTES[category]\n return handler(question)\n\nprint(route(\"I need a refund for my order\"))\nprint(route(\"Getting a 500 error\"))\nprint(route(\"How does this work?\"))\n","runnable":true},{"type":"mc","id":"agent-loops/routing/s02-02-when-to-route","xp":2,"hint":[{"level":1,"body":"Routing earns its keep when categories have *different tools, prompts, or costs*. Same tools + same prompt = no route needed.","cost":0}],"personalize":false,"phase":"warmup","estSeconds":55,"concept":"when-routing-helps","prompt":"Four features above. Which one is the strongest fit for a routing\npattern (LLM classifier → dispatch table)?\n","options":[{"id":"a","label":"A — single fixed message templates","explain":"Two outcomes hardcoded by status. A classifier here adds latency and cost for no decision the model makes."},{"id":"b","label":"B — support inbox with different categories and tool sets"},{"id":"c","label":"C — search with a user-selected category dropdown","explain":"The user already classified for you. Adding a model classifier on top is pure waste."},{"id":"d","label":"D — one chatbot, one corpus, every question","explain":"All paths read the same docs. There's no specialist to dispatch to."}],"answerIds":["b"],"shuffle":false,"code":"# four hypothetical features. only some of them benefit from a router.\nfeatures = {\n \"A\": \"A status-check page that says either 'all systems normal' or 'incident: '.\",\n \"B\": \"A support inbox that handles billing, technical, and account questions, each with its own tools.\",\n \"C\": \"A search box where the user picks 'docs', 'people', or 'orders' from a dropdown before searching.\",\n \"D\": \"A chatbot that answers ANY product question by reading the same docs corpus.\",\n}\nfor k, v in features.items():\n print(f\"{k}: {v}\")\n","runnable":true},{"type":"read","id":"agent-loops/routing/s03-03-classifier-output","xp":2,"hint":[],"personalize":false,"phase":"build","estSeconds":95,"concept":"constrained-classifier-output","body":"$35","cta":"Got it","code":"# the classifier MUST return a value the dispatch table knows about.\n# two ways to make that happen: an enum of allowed strings, or a\n# JSON schema with `enum`. both ride on top of structured output.\n\nALLOWED = {\"billing\", \"technical\", \"general\"}\n\nclassifier_schema = {\n \"type\": \"object\",\n \"properties\": {\n \"category\": {\"type\": \"string\", \"enum\": list(ALLOWED)},\n \"confidence\": {\"type\": \"number\"},\n },\n \"required\": [\"category\"],\n}\n\n# what the model returns (constrained to the schema):\nclassification = {\"category\": \"billing\", \"confidence\": 0.91}\n\nprint(f\"category: {classification['category']}\")\nprint(f\"in allowed set: {classification['category'] in ALLOWED}\")\n","runnable":true},{"type":"predict","id":"agent-loops/routing/s04-04-predict-the-route","xp":3,"hint":[{"level":1,"body":"`fake_classify` returns the mapped category, then `ROUTES[category](question)` runs the matching specialist. Each input prints its category prefix + the question.","cost":0},{"level":2,"body":"Output the three formatted strings in the order the questions appear.","cost":0}],"personalize":false,"phase":"build","estSeconds":75,"concept":"route-dispatch-execution","code":"# a tiny router. the fake classifier returns a fixed category for each input.\ndef fake_classify(question):\n mapping = {\n \"I want a refund\": \"billing\",\n \"Site is down\": \"technical\",\n \"What's your name?\": \"general\",\n }\n return mapping.get(question, \"general\")\n\nROUTES = {\n \"billing\": lambda q: f\"BILLING -> {q}\",\n \"technical\": lambda q: f\"TECH -> {q}\",\n \"general\": lambda q: f\"GEN -> {q}\",\n}\n\ndef route(question):\n category = fake_classify(question)\n return ROUTES[category](question)\n\nfor q in [\"I want a refund\", \"Site is down\", \"What's your name?\"]:\n print(route(q))\n","prompt":"Read the editor. The fake classifier maps each input to a fixed\ncategory, and the router dispatches through `ROUTES`. What three\nlines does the loop print, in order?\n","grader":{"kind":"stdout-equality","expected":"BILLING -> I want a refund\nTECH -> Site is down\nGEN -> What's your name?\n","normalize":"collapse-trailing-newline"}},{"type":"fill","id":"agent-loops/routing/s05-05-fill-the-dispatch","xp":3,"hint":[{"level":1,"body":"Two pieces: look up the handler with `ROUTES[category]`, then call it with `question`.","cost":0},{"level":2,"body":"`ROUTES[category](question)` — registry lookup, then call with the user input.","cost":0}],"personalize":false,"phase":"build","estSeconds":60,"concept":"route-table-lookup","prompt":"The router classifies the question and is about to dispatch to the\nmatching specialist. Fill in the missing expression so the right\nhandler runs.\n\nExpected output:\n```\nTECH -> server is on fire\n```\n","code":"ROUTES = {\n \"billing\": lambda q: f\"BILL -> {q}\",\n \"technical\": lambda q: f\"TECH -> {q}\",\n \"general\": lambda q: f\"GEN -> {q}\",\n}\n\ncategory = \"technical\"\nquestion = \"server is on fire\"\n\nresult = ___\nprint(result)\n","blanks":[{"id":"dispatch","accept":["ROUTES[category](question)"],"caseSensitive":true,"normalize":"trim"}]},{"type":"fix","id":"agent-loops/routing/s06-06-fix-the-fallback","xp":4,"hint":[{"level":1,"body":"Use `ROUTES.get(category, ROUTES[\"general\"])` — `dict.get` with a default returns the general handler when the key is missing.","cost":0},{"level":2,"body":"Replace line 11 with:\n\n```python\nhandler = ROUTES.get(category, ROUTES[\"general\"])\nresult = handler(question)\n```\n","cost":0}],"personalize":false,"phase":"build","estSeconds":90,"concept":"router-fallback-default","brokenCode":"ROUTES = {\n \"billing\": lambda q: f\"BILL -> {q}\",\n \"technical\": lambda q: f\"TECH -> {q}\",\n \"general\": lambda q: f\"GEN -> {q}\",\n}\n\ncategory = \"unknown\" # the classifier drifted\nquestion = \"what time do you open\"\n\n# this raises KeyError on unknown categories.\nresult = ROUTES[category](question)\nprint(result)\n","prompt":"The classifier just returned `\"unknown\"` — a category not in the\ndispatch table. The current code raises `KeyError` and the user\ngets a 500. Fix the dispatch so an unknown category falls back to\nthe `\"general\"` route instead.\n\nExpected output:\n```\nGEN -> what time do you open\n```\n","grader":{"kind":"stdout-equality","expected":"GEN -> what time do you open","normalize":"collapse-trailing-newline"},"bugLines":[11],"revealAfter":4},{"type":"fix","id":"agent-loops/routing/s07-07-fix-the-loose-classifier","xp":4,"hint":[{"level":1,"body":"Two steps: `.strip()` to remove whitespace, `.lower()` to normalize case. Both are string methods on `raw_category`.","cost":0},{"level":2,"body":"Change line 11 to `category = raw_category.strip().lower()`.","cost":0}],"personalize":false,"phase":"build","estSeconds":105,"concept":"classifier-output-normalization","brokenCode":"ROUTES = {\n \"billing\": lambda q: f\"BILL -> {q}\",\n \"technical\": lambda q: f\"TECH -> {q}\",\n \"general\": lambda q: f\"GEN -> {q}\",\n}\n\nraw_category = \" Billing \" # what the model returned\nquestion = \"refund please\"\n\n# normalize raw_category here so the lookup works.\ncategory = raw_category\nhandler = ROUTES.get(category, ROUTES[\"general\"])\nprint(handler(question))\n","prompt":"The classifier model returned `\" Billing \"` — capitalized, with\nwhitespace. The dispatch table only knows `\"billing\"` (lowercase,\ntrimmed). Without normalization, this falls through to the\nfallback even though the classifier got it right.\n\nFix the normalization so the category is lowercased AND stripped\nbefore the dispatch lookup.\n\nExpected output:\n```\nBILL -> refund please\n```\n","grader":{"kind":"stdout-equality","expected":"BILL -> refund please","normalize":"collapse-trailing-newline"},"bugLines":[11],"revealAfter":4},{"type":"write","id":"agent-loops/routing/s08-08-write-the-router","xp":5,"hint":[{"level":1,"body":"Three lines: classify, normalize, dispatch. Use `.strip().lower()` on the classifier output and `ROUTES.get(category, ROUTES[\"general\"])` for the dispatch.","cost":0},{"level":2,"body":"```python\ndef route(question):\n raw = fake_classifier(question)\n category = raw.strip().lower()\n handler = ROUTES.get(category, ROUTES[\"general\"])\n return handler(question)\n```\n","cost":0}],"personalize":false,"phase":"build","estSeconds":180,"concept":"full-router-implementation","prompt":"Build a full router. Write `route(question)` that:\n\n- Calls `fake_classifier(question)` to get a raw category string.\n- Normalizes it: `.strip().lower()`.\n- If the normalized category is in `ROUTES`, run that handler with\n the question.\n- Otherwise run `ROUTES[\"general\"]` as the fallback.\n- Returns the handler's output.\n\nThe classifier may return messy strings (`\"Billing\"`, `\" general\n\"`, `\"unknown_category\"`). Your router should handle all three.\n\nThen the script will run four cases. Expected output:\n```\nBILL -> I want a refund\nTECH -> 500 error on checkout\nGEN -> what time do you open\nGEN -> something completely off-topic\n```\n","starter":"ROUTES = {\n \"billing\": lambda q: f\"BILL -> {q}\",\n \"technical\": lambda q: f\"TECH -> {q}\",\n \"general\": lambda q: f\"GEN -> {q}\",\n}\n\nclassifier_outputs = iter([\"Billing\", \"technical\", \" general \", \"unknown_category\"])\n\ndef fake_classifier(question):\n return next(classifier_outputs)\n\n# define route(question) below\n\nquestions = [\n \"I want a refund\",\n \"500 error on checkout\",\n \"what time do you open\",\n \"something completely off-topic\",\n]\nfor q in questions:\n print(route(q))\n","grader":{"kind":"stdout-equality","expected":"BILL -> I want a refund\nTECH -> 500 error on checkout\nGEN -> what time do you open\nGEN -> something completely off-topic\n","normalize":"collapse-trailing-newline"},"solution":"ROUTES = {\n \"billing\": lambda q: f\"BILL -> {q}\",\n \"technical\": lambda q: f\"TECH -> {q}\",\n \"general\": lambda q: f\"GEN -> {q}\",\n}\n\nclassifier_outputs = iter([\"Billing\", \"technical\", \" general \", \"unknown_category\"])\n\ndef fake_classifier(question):\n return next(classifier_outputs)\n\ndef route(question):\n raw = fake_classifier(question)\n category = raw.strip().lower()\n handler = ROUTES.get(category, ROUTES[\"general\"])\n return handler(question)\n\nquestions = [\n \"I want a refund\",\n \"500 error on checkout\",\n \"what time do you open\",\n \"something completely off-topic\",\n]\nfor q in questions:\n print(route(q))\n","hiddenTests":[]},{"type":"checkpoint","id":"agent-loops/routing/s09-09-checkpoint","xp":8,"hint":[],"personalize":false,"phase":"check","estSeconds":200,"concept":"routing-checkpoint","prompt":"Final drill. Build a routing layer that wraps the agent loop you\nalready know. Write `run_router(question)` that:\n\n- Calls `fake_classifier(question)` and normalizes the output\n (`.strip().lower()`).\n- Looks up the normalized category in `SPECIALISTS`. If missing,\n fall back to `\"general\"`.\n- Calls the specialist function with `question`. Each specialist\n returns a dict shaped like `{\"answer\": \"...\", \"tools_used\": N}`.\n- Returns a single dict:\n `{\"category\": , \"answer\": , \"tools_used\": }`.\n\nNote: the *category used* is what you actually dispatched to —\nthat's `\"general\"` when the classifier produced something unknown,\nnot the raw classifier string.\n\nThe script runs four cases. Expected output:\n```\ncategory=billing tools=1 answer=refunded\ncategory=technical tools=2 answer=restarted\ncategory=general tools=0 answer=hours: 9-5\ncategory=general tools=0 answer=fallback answer\n```\n","starter":"def billing_specialist(q):\n return {\"answer\": \"refunded\", \"tools_used\": 1}\n\ndef technical_specialist(q):\n return {\"answer\": \"restarted\", \"tools_used\": 2}\n\ndef general_specialist(q):\n if \"open\" in q or \"hour\" in q:\n return {\"answer\": \"hours: 9-5\", \"tools_used\": 0}\n return {\"answer\": \"fallback answer\", \"tools_used\": 0}\n\nSPECIALISTS = {\n \"billing\": billing_specialist,\n \"technical\": technical_specialist,\n \"general\": general_specialist,\n}\n\nclassifier_outputs = iter([\"BILLING\", \" technical \", \"general\", \"weird_unknown\"])\n\ndef fake_classifier(question):\n return next(classifier_outputs)\n\n# define run_router(question) below\n\nquestions = [\n \"Refund my subscription\",\n \"Site is throwing 500s\",\n \"What hours are you open?\",\n \"totally off topic\",\n]\nfor q in questions:\n r = run_router(q)\n print(f\"category={r['category']} tools={r['tools_used']} answer={r['answer']}\")\n","grader":{"kind":"stdout-equality","expected":"category=billing tools=1 answer=refunded\ncategory=technical tools=2 answer=restarted\ncategory=general tools=0 answer=hours: 9-5\ncategory=general tools=0 answer=fallback answer\n","normalize":"collapse-trailing-newline"},"solution":"$36","passThreshold":0.66}],"xpTotal":32},{"slug":"evaluator-optimizer","title":"Evaluator-optimizer — write a draft, let a judge critique it, revise","estMinutes":12,"prerequisites":["the-loop","multi-step-tools","routing"],"status":"live","steps":[{"type":"read","id":"agent-loops/evaluator-optimizer/s01-01-intro","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":120,"concept":"evaluator-optimizer-pattern","body":"$37","cta":"Got it","code":"# the loop in 8 lines: draft, judge, revise on REJECT, exit on PASS.\n\ndef write_with_judge(task, generator, judge, max_iters=5):\n feedback = None\n draft = None\n for i in range(max_iters):\n draft = generator(task, feedback=feedback)\n verdict = judge(draft)\n if verdict[\"status\"] == \"PASS\":\n return draft, i + 1\n feedback = verdict[\"feedback\"]\n return draft, max_iters\n\n# mock generator + judge so the function runs without a real model.\ndrafts = iter([\"Cats are animals.\", \"Cats are small carnivorous mammals.\"])\nverdicts = iter([\n {\"status\": \"REJECT\", \"feedback\": \"Too short. Add detail.\"},\n {\"status\": \"PASS\", \"feedback\": \"Clear and detailed.\"},\n])\n\nresult, iters = write_with_judge(\n \"Describe cats\",\n generator=lambda task, feedback=None: next(drafts),\n judge=lambda draft: next(verdicts),\n)\nprint(f\"final draft (iter {iters}): {result}\")\n","runnable":true},{"type":"mc","id":"agent-loops/evaluator-optimizer/s02-02-when-it-earns-its-keep","xp":2,"hint":[{"level":1,"body":"Which tasks have qualitative judgment built in (style, completeness, tone) versus a single numerical right answer?","cost":0}],"personalize":false,"phase":"warmup","estSeconds":55,"concept":"evaluator-optimizer-fit","prompt":"Anthropic's two signals: (1) human-articulated feedback would\ndemonstrably improve the response, and (2) the LLM can produce that\nkind of feedback. Which TWO features above are the strongest fit for\nthe evaluator-optimizer pattern?\n","options":[{"id":"a_and_c","label":"A (translator) and C (code reviewer)"},{"id":"a_and_b","label":"A (translator) and B (arithmetic)","explain":"Arithmetic has a single correct answer. A judge can't 'critique' a wrong number into a right one — you need a calculator, not a critic."},{"id":"c_and_d","label":"C (code reviewer) and D (weather)","explain":"Weather is a lookup with a ground-truth value. There's nothing to refine — either you have the right number from the API or you don't."},{"id":"b_and_d","label":"B (arithmetic) and D (weather)","explain":"Both are exact-answer tasks. Iterative refinement adds latency and cost without changing correctness."}],"answerIds":["a_and_c"],"shuffle":false,"code":"# four hypothetical features. only some benefit from this pattern.\nfeatures = {\n \"A\": \"A translator that should preserve nuance and tone across attempts.\",\n \"B\": \"An arithmetic helper that returns the answer to '17 * 23'.\",\n \"C\": \"A code reviewer that emits a refactored version of a function.\",\n \"D\": \"A weather lookup that returns the current temp for a city.\",\n}\nfor k, v in features.items():\n print(f\"{k}: {v}\")\n","runnable":true},{"type":"read","id":"agent-loops/evaluator-optimizer/s03-03-judge-output-shape","xp":2,"hint":[],"personalize":false,"phase":"build","estSeconds":100,"concept":"structured-judge-output","body":"$38","cta":"Got it","code":"# the judge's output is the load-bearing contract of this pattern.\n# parse it as structured data, never as natural language.\n\n# GOOD — structured verdict, easy to branch on:\ngood_verdict = {\n \"status\": \"REJECT\",\n \"feedback\": \"The second paragraph contradicts the first. Pick one.\",\n \"score\": 4,\n}\n\n# BAD — free-text verdict you'd parse with `if \"good\" in verdict.lower()`:\nbad_verdict = \"This is mostly good but not great, I'd say a 6/10 if you fixed the contradiction.\"\n\n# which one do you want to dispatch on?\nif good_verdict[\"status\"] == \"PASS\":\n print(\"done\")\nelse:\n print(\"revise:\", good_verdict[\"feedback\"])\n","runnable":true},{"type":"predict","id":"agent-loops/evaluator-optimizer/s04-04-predict-the-iterations","xp":3,"hint":[{"level":1,"body":"Trace it: gen → judge (REJECT, iter 1) → gen → judge (REJECT, iter 2) → gen → judge (PASS, iter 3). The PASS branch returns `i + 1`.","cost":0},{"level":2,"body":"Three iterations: two rejections plus the accepting one.","cost":0}],"personalize":false,"phase":"build","estSeconds":80,"concept":"evaluator-optimizer-iteration-trace","code":"# the judge rejects the first two drafts, then accepts the third.\n\ndrafts = iter([\n \"Cats are animals.\",\n \"Cats are small mammals.\",\n \"Cats are small carnivorous mammals kept as pets worldwide.\",\n])\nverdicts = iter([\n {\"status\": \"REJECT\", \"feedback\": \"Too short.\"},\n {\"status\": \"REJECT\", \"feedback\": \"Add detail about diet.\"},\n {\"status\": \"PASS\", \"feedback\": \"Detailed and accurate.\"},\n])\n\ndef gen(task, feedback=None):\n return next(drafts)\n\ndef judge(draft):\n return next(verdicts)\n\ndef write_with_judge(task, max_iters=5):\n draft = gen(task)\n for i in range(max_iters):\n verdict = judge(draft)\n if verdict[\"status\"] == \"PASS\":\n return i + 1\n draft = gen(task, feedback=verdict[\"feedback\"])\n return max_iters\n\niterations = write_with_judge(\"Describe cats\")\nprint(f\"iterations: {iterations}\")\n","prompt":"Read the editor. The fake judge rejects drafts 1 and 2, then accepts\ndraft 3. The function returns the iteration count when the judge\nPASSES. How many iterations does it take?\n","grader":{"kind":"stdout-equality","expected":"iterations: 3","normalize":"collapse-trailing-newline"}},{"type":"fill","id":"agent-loops/evaluator-optimizer/s05-05-fill-the-exit","xp":3,"hint":[{"level":1,"body":"The judge returns a dict like `{\"status\": \"PASS\", ...}`. Read the `status` key and compare it to the literal string.","cost":0},{"level":2,"body":"`verdict[\"status\"] == \"PASS\"` — read the field, compare to the string the schema enforces.","cost":0}],"personalize":false,"phase":"build","estSeconds":65,"concept":"pass-status-exit","prompt":"The evaluator-optimizer loop is missing its exit condition. Fill in\nthe comparison so the function returns the draft as soon as the\njudge's `status` field equals `\"PASS\"`.\n\nExpected output:\n```\ndone in 2 iters\n```\n","code":"drafts = iter([\"v1\", \"v2\"])\nverdicts = iter([\n {\"status\": \"REJECT\", \"feedback\": \"needs work\"},\n {\"status\": \"PASS\", \"feedback\": \"ok\"},\n])\n\ndef gen(task, feedback=None): return next(drafts)\ndef judge(draft): return next(verdicts)\n\ndef loop(task, max_iters=5):\n draft = gen(task)\n for i in range(max_iters):\n verdict = judge(draft)\n if ___:\n return f\"done in {i + 1} iters\"\n draft = gen(task, feedback=verdict[\"feedback\"])\n return \"capped\"\n\nprint(loop(\"describe x\"))\n","blanks":[{"id":"exit","accept":["verdict[\"status\"] == \"PASS\"","verdict['status'] == 'PASS'"],"caseSensitive":true,"normalize":"trim"}]},{"type":"fix","id":"agent-loops/evaluator-optimizer/s06-06-fix-the-judge-parsing","xp":4,"hint":[{"level":1,"body":"Use `verdict.startswith(\"PASS\")` instead of `\"good\" in verdict.lower()`. Production code uses structured output for exactly this reason — substring matches lie.","cost":0},{"level":2,"body":"Change line 4 to:\n\n```python\nif verdict.startswith(\"PASS\"):\n```\n","cost":0}],"personalize":false,"phase":"build","estSeconds":100,"concept":"free-text-parsing-bug","brokenCode":"verdict = \"REJECT: this is not good, please fix the second sentence\"\n\n# bug: substring match on \"good\" matches \"not good\" too.\nif \"good\" in verdict.lower():\n print(\"accepted, shipping\")\nelse:\n print(\"rejected, will revise\")\n","prompt":"The judge is now returning free-text verdicts (the model wasn't\nforced into structured output). The current loop checks\n`if \"good\" in verdict.lower()` — and the judge said *\"this is **not**\ngood, please fix it\"*. The substring match sees `\"good\"`, exits the\nloop happily, and ships the broken draft.\n\nFix the parsing so it requires the verdict to START with the\nliteral token `\"PASS\"` (case-sensitive). Anything else is a reject.\n\nExpected output:\n```\nrejected, will revise\n```\n","grader":{"kind":"stdout-equality","expected":"rejected, will revise","normalize":"collapse-trailing-newline"},"bugLines":[4],"revealAfter":4},{"type":"fix","id":"agent-loops/evaluator-optimizer/s07-07-fix-the-missing-cap","xp":4,"hint":[{"level":1,"body":"Replace `while True:` with `for _ in range(3):` so the loop bails after 3 iterations. The trailing `return \"capped\"` will then run.","cost":0},{"level":2,"body":"Change line 11 to:\n\n```python\nfor _ in range(3):\n```\n\nThat makes the bottom `return \"capped\"` reachable when no PASS comes.\n","cost":0}],"personalize":false,"phase":"build","estSeconds":100,"concept":"max-iterations-safety-rail","brokenCode":"drafts = iter([\"v1\", \"v2\", \"v3\", \"v4\"])\nverdicts = iter([\n {\"status\": \"REJECT\", \"feedback\": \"no\"},\n {\"status\": \"REJECT\", \"feedback\": \"still no\"},\n {\"status\": \"REJECT\", \"feedback\": \"really no\"},\n])\n\ndef gen(task, feedback=None): return next(drafts)\ndef judge(draft): return next(verdicts)\n\ndef loop(task):\n draft = gen(task)\n while True: # bug: no cap\n verdict = judge(draft)\n if verdict[\"status\"] == \"PASS\":\n return draft\n draft = gen(task, feedback=verdict[\"feedback\"])\n return \"capped\"\n\nprint(loop(\"describe x\"))\n","prompt":"This loop has no `max_iters` ceiling. The judge in this test case is\nstubborn — it never says PASS. As written, the `while True` runs\nuntil the iter pool is empty and Python raises `StopIteration`\n(which in production would mean burning through your API budget).\n\nFix the loop to bail out after at most 3 iterations, returning the\nstring `\"capped\"` if no PASS was reached.\n\nExpected output:\n```\ncapped\n```\n","grader":{"kind":"stdout-equality","expected":"capped","normalize":"collapse-trailing-newline"},"bugLines":[13],"revealAfter":4},{"type":"write","id":"agent-loops/evaluator-optimizer/s08-08-write-the-evaluator-optimizer","xp":5,"hint":[{"level":1,"body":"Track `feedback` as a local variable starting at `None`. Inside the loop: call `gen(task, feedback=feedback)`, judge the result, branch on PASS, otherwise update `feedback = verdict[\"feedback\"]`. After the loop: return the last draft with `passed=False`.","cost":0},{"level":2,"body":"```python\ndef write_with_judge(task, max_iters):\n feedback = None\n draft = None\n for i in range(max_iters):\n draft = gen(task, feedback=feedback)\n verdict = judge(draft)\n if verdict[\"status\"] == \"PASS\":\n return {\"draft\": draft, \"iters\": i + 1, \"passed\": True}\n feedback = verdict[\"feedback\"]\n return {\"draft\": draft, \"iters\": max_iters, \"passed\": False}\n```\n","cost":0}],"personalize":false,"phase":"build","estSeconds":180,"concept":"full-evaluator-optimizer","prompt":"Build the full evaluator-optimizer. Write `write_with_judge(task, max_iters)` that:\n\n- Maintains `feedback = None` (initial draft has no prior feedback).\n- Loops up to `max_iters` times. Each iteration:\n - Calls `gen(task, feedback=feedback)` to produce a draft.\n - Calls `judge(draft)` to get a verdict dict.\n - If `verdict[\"status\"] == \"PASS\"`: return a dict\n `{\"draft\": , \"iters\": , \"passed\": True}`.\n - Otherwise update `feedback = verdict[\"feedback\"]` and continue.\n- If `max_iters` is exhausted without a PASS, return\n `{\"draft\": , \"iters\": max_iters, \"passed\": False}`.\n\nThe script will run two cases (one that PASSes on iter 2, one that\nexhausts the cap). Expected output:\n```\npassed=True iters=2 draft=Cats are small carnivorous mammals.\npassed=False iters=2 draft=v2\n```\n","starter":"good_drafts = iter([\n \"Cats are animals.\",\n \"Cats are small carnivorous mammals.\",\n])\ngood_verdicts = iter([\n {\"status\": \"REJECT\", \"feedback\": \"Too short.\"},\n {\"status\": \"PASS\", \"feedback\": \"Detailed.\"},\n])\n\nbad_drafts = iter([\"v1\", \"v2\"])\nbad_verdicts = iter([\n {\"status\": \"REJECT\", \"feedback\": \"no\"},\n {\"status\": \"REJECT\", \"feedback\": \"still no\"},\n])\n\ncurrent = {\"drafts\": good_drafts, \"verdicts\": good_verdicts}\n\ndef gen(task, feedback=None):\n return next(current[\"drafts\"])\n\ndef judge(draft):\n return next(current[\"verdicts\"])\n\n# define write_with_judge(task, max_iters) below\n\ncurrent[\"drafts\"], current[\"verdicts\"] = good_drafts, good_verdicts\nr = write_with_judge(\"describe cats\", 3)\nprint(f\"passed={r['passed']} iters={r['iters']} draft={r['draft']}\")\n\ncurrent[\"drafts\"], current[\"verdicts\"] = bad_drafts, bad_verdicts\nr = write_with_judge(\"describe cats\", 2)\nprint(f\"passed={r['passed']} iters={r['iters']} draft={r['draft']}\")\n","grader":{"kind":"stdout-equality","expected":"passed=True iters=2 draft=Cats are small carnivorous mammals.\npassed=False iters=2 draft=v2\n","normalize":"collapse-trailing-newline"},"solution":"$39","hiddenTests":[]},{"type":"checkpoint","id":"agent-loops/evaluator-optimizer/s09-09-checkpoint","xp":8,"hint":[],"personalize":false,"phase":"check","estSeconds":200,"concept":"evaluator-optimizer-checkpoint","prompt":"Final drill. Build a production-grade evaluator-optimizer that\ntracks the full revision history. Write\n`optimize(task, max_iters)` that:\n\n- Maintains `feedback = None` and `history = []`.\n- Loops up to `max_iters` times. Each iteration:\n - Calls `gen(task, feedback=feedback)` to produce a draft.\n - Calls `judge(draft)` to get a verdict dict.\n - Appends one entry to `history`:\n `{\"iter\": <1-based>, \"draft\": , \"status\": , \"feedback\": }`.\n - On PASS: return\n `{\"final_draft\": draft, \"iters\": , \"passed\": True, \"history\": history}`.\n - On REJECT: update `feedback = verdict[\"feedback\"]` and continue.\n- On cap exhaustion without a PASS: return\n `{\"final_draft\": , \"iters\": max_iters, \"passed\": False, \"history\": history}`.\n\nThe script will run two cases. Expected output:\n```\npassed=True iters=3 history_len=3 final=v3-good\npassed=False iters=2 history_len=2 final=stub2\n```\n","starter":"$3a","grader":{"kind":"stdout-equality","expected":"passed=True iters=3 history_len=3 final=v3-good\npassed=False iters=2 history_len=2 final=stub2\n","normalize":"collapse-trailing-newline"},"solution":"$3b","passThreshold":0.66}],"xpTotal":32},{"slug":"why-every-framework-is-thirty-lines","title":"Why every framework is the same thirty lines — and what that means for buy-vs-build","estMinutes":20,"prerequisites":["evaluator-optimizer"],"status":"live","steps":[{"type":"read","id":"agent-loops/why-every-framework-is-thirty-lines/s01-01-the-frameworks-converged","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":220,"concept":"framework-convergence","body":"$3c","cta":"Got it","runnable":true},{"type":"read","id":"agent-loops/why-every-framework-is-thirty-lines/s02-02-langgraph-walkthrough","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":200,"concept":"langgraph-walkthrough","body":"$3d","cta":"Got it","runnable":true},{"type":"read","id":"agent-loops/why-every-framework-is-thirty-lines/s03-03-the-work-is-a-graph","xp":1,"hint":[],"personalize":false,"phase":"build","estSeconds":150,"concept":"workflow-graphs-and-gates","body":"$3e","cta":"Got it","code":"steps = {\n \"draft\": {\"needs\": [], \"done\": False},\n \"check\": {\"needs\": [\"draft\"], \"done\": False},\n \"send\": {\"needs\": [\"check\"], \"done\": False, \"gate\": \"human approval\"},\n}\n\nready = [name for name, s in steps.items()\n if not s[\"done\"] and all(steps[n][\"done\"] for n in s[\"needs\"])]\nprint(\"runnable now:\", ready)\n","runnable":true},{"type":"mc","id":"agent-loops/why-every-framework-is-thirty-lines/s04-04-which-framework-fits","xp":2,"hint":[{"level":1,"body":"Match each project to its dominant constraint. Long-running + Python = LangGraph (checkpointing). Solo + deadline = framework-free. Next.js + streaming = Vercel AI SDK. Named multi-agent handoffs = CrewAI.","cost":0}],"personalize":false,"phase":"build","estSeconds":70,"concept":"framework-fit-scenarios","prompt":"Four real project profiles. Each one points at a different\nframework (or none). Which mapping is correct?\n","options":[{"id":"all_langgraph","label":"All four → LangGraph (it's the most popular)","explain":"Popularity isn't fit. A Next.js streaming chatbot wants the Vercel AI SDK. A solo prototype wants framework-free. Defaulting to LangGraph for everything is exactly the failure mode Anthropic warned about."},{"id":"correct_split","label":"LangGraph → research agent; framework-free → solo prototype; Vercel AI SDK → Next.js chatbot; CrewAI → 5-agent newsroom"},{"id":"all_vercel","label":"All four → Vercel AI SDK (streaming is universal)","explain":"Streaming matters for chat UX, not for a 60-minute research run. And Vercel AI SDK is a TypeScript SDK — a Python research agent doesn't live in its ecosystem."},{"id":"prototype_gets_langgraph","label":"LangGraph for everything except the prototype, which gets CrewAI","explain":"Backwards. CrewAI's whole pitch is multi-agent role-play, which is exactly the newsroom sim — not a solo prototype. The prototype wants the least framework, not a different one."}],"answerIds":["correct_split"],"shuffle":false,"code":"# four project profiles. each wants a different answer.\nprojects = [\n {\n \"name\": \"Research agent\",\n \"detail\": \"60-min runs, must survive server restarts, Python backend\",\n },\n {\n \"name\": \"Solo SaaS prototype\",\n \"detail\": \"2-week deadline, one dev, single-file script, throwaway code\",\n },\n {\n \"name\": \"Next.js chatbot\",\n \"detail\": \"streaming responses, React UI, deployed to Vercel\",\n },\n {\n \"name\": \"5-agent newsroom sim\",\n \"detail\": \"editor + 4 writers with named handoffs, role-based prompts\",\n },\n]\nfor p in projects:\n print(f\"- {p['name']:24} | {p['detail']}\")\n","runnable":true},{"type":"read","id":"agent-loops/why-every-framework-is-thirty-lines/s05-05-vercel-ai-sdk-walkthrough","xp":1,"hint":[],"personalize":false,"phase":"build","estSeconds":200,"concept":"vercel-ai-sdk-walkthrough","body":"$3f","cta":"Got it","runnable":true},{"type":"read","id":"agent-loops/why-every-framework-is-thirty-lines/s06-06-the-five-canonical-patterns","xp":1,"hint":[],"personalize":false,"phase":"build","estSeconds":220,"concept":"anthropic-five-patterns","body":"$40","cta":"Got it","runnable":true},{"type":"mc","id":"agent-loops/why-every-framework-is-thirty-lines/s07-07-when-to-stay-framework-free","xp":2,"hint":[{"level":1,"body":"Frameworks earn their weight on large teams + long horizons + durability needs. They cost more than they save on prototypes, single-file scripts, and teams that don't yet understand the underlying loop.","cost":0}],"personalize":false,"phase":"build","estSeconds":70,"concept":"framework-free-when","prompt":"Adopting a framework adds setup tax, vocabulary, version churn,\nand lock-in. In which case below does that tax come out NEGATIVE\n— i.e., framework-free is strictly better?\n","options":[{"id":"only_b","label":"Only B (12 engineers, 18-month roadmap)","explain":"Inverted. That's exactly the profile where a framework's coordination value pays back. Framework-free works when scope is small, not when scope is large."},{"id":"a_c_d","label":"A, C, and D — small scope, single-file, or learning the loop"},{"id":"only_c","label":"Only C (single-file script)","explain":"Half right. The single-file script doesn't need a framework — but the 2-week MVP and the learning team are both also worse off with one. Setup tax exceeds value in all three."},{"id":"none_all_need_one","label":"None of them — every agent needs a framework","explain":"This is the marketing pitch, not the reality. Cursor, Devin, Perplexity, and most shipped agent products run custom loops. 'Every agent needs a framework' is false."}],"answerIds":["a_c_d"],"shuffle":false,"code":"# four situations. in which one does adding a framework make things WORSE?\ncases = [\n {\n \"id\": \"A\",\n \"team\": \"Solo founder, 2-week MVP, throwaway code\",\n },\n {\n \"id\": \"B\",\n \"team\": \"12 engineers, 18-month roadmap, durable agents\",\n },\n {\n \"id\": \"C\",\n \"team\": \"Single-file Python script that runs nightly via cron\",\n },\n {\n \"id\": \"D\",\n \"team\": \"Eng team learning agent loops for the first time\",\n },\n]\nfor c in cases:\n print(f\"{c['id']}: {c['team']}\")\n","runnable":true},{"type":"write","id":"agent-loops/why-every-framework-is-thirty-lines/s08-08-write-the-tradeoff-memo","xp":5,"hint":[{"level":1,"body":"Four `if` statements in order. First-match-wins, so use early `return` inside each branch. The fallback at the bottom returns `\"crewai\"` with no condition.","cost":0},{"level":2,"body":"Solo MVP hits rule 1 (team_size=1 <= 2 AND days=10 <= 14) → framework-free. Research agent fails rule 1 (team_size=6), fails rule 2 (no streaming), passes rule 3 (durable + python) → langgraph.","cost":0}],"personalize":false,"phase":"build","estSeconds":200,"concept":"framework-recommendation-rules","prompt":"$41","starter":"solo_mvp = {\n \"team_size\": 1,\n \"language\": \"python\",\n \"needs_streaming\": False,\n \"needs_durable_state\": False,\n \"time_to_first_demo_days\": 10,\n}\n\nresearch_agent = {\n \"team_size\": 6,\n \"language\": \"python\",\n \"needs_streaming\": False,\n \"needs_durable_state\": True,\n \"time_to_first_demo_days\": 90,\n}\n\n# define recommend_framework(project) below\n\nprint(f\"Solo MVP: {recommend_framework(solo_mvp)}\")\nprint(f\"Research agent: {recommend_framework(research_agent)}\")\n","grader":{"kind":"stdout-equality","expected":"Solo MVP: framework-free\nResearch agent: langgraph\n","normalize":"collapse-trailing-newline"},"solution":"solo_mvp = {\n \"team_size\": 1,\n \"language\": \"python\",\n \"needs_streaming\": False,\n \"needs_durable_state\": False,\n \"time_to_first_demo_days\": 10,\n}\n\nresearch_agent = {\n \"team_size\": 6,\n \"language\": \"python\",\n \"needs_streaming\": False,\n \"needs_durable_state\": True,\n \"time_to_first_demo_days\": 90,\n}\n\ndef recommend_framework(project):\n if project[\"team_size\"] <= 2 and project[\"time_to_first_demo_days\"] <= 14:\n return \"framework-free\"\n if project[\"needs_streaming\"] and project[\"language\"] in (\"typescript\", \"both\"):\n return \"vercel-ai-sdk\"\n if project[\"needs_durable_state\"] and project[\"language\"] in (\"python\", \"both\"):\n return \"langgraph\"\n return \"crewai\"\n\nprint(f\"Solo MVP: {recommend_framework(solo_mvp)}\")\nprint(f\"Research agent: {recommend_framework(research_agent)}\")\n","hiddenTests":[]},{"type":"checkpoint","id":"agent-loops/why-every-framework-is-thirty-lines/s09-09-checkpoint","xp":8,"hint":[],"personalize":false,"phase":"check","estSeconds":240,"concept":"framework-stack-audit","prompt":"Final drill. Audit a real-shaped agent stack. Write\n`audit_stack(projects)` that takes a list of project profiles and\nreturns a dict with two fields:\n\n- `recommendations`: dict mapping each project's `name` to its\n recommendation string (`\"framework-free\"`, `\"vercel-ai-sdk\"`,\n `\"langgraph\"`, or `\"crewai\"`).\n- `framework_free_count`: int — how many projects stay\n framework-free.\n\nUse the SAME rule order as step 7 (first match wins):\n\n1. `team_size <= 2` AND `time_to_first_demo_days <= 14` →\n `\"framework-free\"`\n2. `needs_streaming` AND `language` in (`\"typescript\"`, `\"both\"`) →\n `\"vercel-ai-sdk\"`\n3. `needs_durable_state` AND `language` in (`\"python\"`, `\"both\"`) →\n `\"langgraph\"`\n4. else → `\"crewai\"`\n\nFive projects run. Expected output:\n```\nSolo MVP -> framework-free\nChat widget -> vercel-ai-sdk\nResearch agent -> langgraph\nNewsroom sim -> crewai\nPair scripting tool -> framework-free\nframework-free count: 2\n```\n","starter":"$42","grader":{"kind":"stdout-equality","expected":"Solo MVP -> framework-free\nChat widget -> vercel-ai-sdk\nResearch agent -> langgraph\nNewsroom sim -> crewai\nPair scripting tool -> framework-free\nframework-free count: 2\n","normalize":"collapse-trailing-newline"},"solution":"$43","passThreshold":0.66}],"xpTotal":22},{"slug":"the-industry-map","title":"The industry map — where your agent loop sits in the five-layer stack","estMinutes":8,"prerequisites":["why-every-framework-is-thirty-lines"],"status":"live","steps":[{"type":"read","id":"agent-loops/the-industry-map/s01-01-the-five-layers","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":210,"concept":"agentic-stack-layers","body":"$44","cta":"Got it","runnable":true},{"type":"mc","id":"agent-loops/the-industry-map/s02-02-which-layer-failed","xp":2,"hint":[{"level":1,"body":"The model behaved correctly and the tool ran correctly. The missing piece is a fixed, deterministic rule — 'bookings over $1,000 need a human' — that fires no matter how the model reasons. Which layer holds the rules?","cost":0}],"personalize":false,"phase":"build","estSeconds":80,"concept":"diagnosing-the-failed-layer","prompt":"The agent loop ran exactly as written. The model reasoned correctly\nabout the data it was given. Using the five-layer map, which layer\nwas missing — the layer that would have caught this *before* the\ncharge went through?\n","options":[{"id":"intelligence","label":"Intelligence — the model should have known $4,200 isn't 'cheap'","explain":"A smarter model might have flagged it — but you can't rely on that, and the airline API itself was wrong. The fix that works every time is a deterministic rule, not a better guess. That rule lives in a different layer."},{"id":"action","label":"Action — the book_flight() tool call was the failure","explain":"The Action layer did its job: the tool call executed correctly. That is exactly the problem — a perfect Action layer with no policy above it still ships the disaster."},{"id":"governance","label":"Governance — no policy rule required sign-off for a high-value booking"},{"id":"orchestration","label":"Orchestration — the request was routed to the wrong tool","explain":"Routing was fine: book_flight() is the correct tool for a booking request. Orchestration failures look like the wrong thing running, or running in the wrong place. Here the right thing ran — nothing was allowed to stop it."}],"answerIds":["governance"],"shuffle":false,"code":"# A new incident — not one from the reading. Diagnose it.\nincident = {\n \"agent\": \"travel-booking assistant\",\n \"request\": \"book me the cheapest flight to Denver\",\n \"what_happened\": [\n \"model searched flights and correctly ranked them by price\",\n \"cheapest result was a $4,200 fare the airline API had mislabeled 'economy'\",\n \"agent called book_flight() on it — the tool call succeeded\",\n \"no rule required human sign-off for bookings over $1,000\",\n ],\n \"result\": \"a $4,200 surprise charge; the loop never malfunctioned\",\n}\nprint(incident[\"agent\"], \"—\", incident[\"request\"])\nfor line in incident[\"what_happened\"]:\n print(\" -\", line)\nprint(\"RESULT:\", incident[\"result\"])\n","runnable":true},{"type":"read","id":"agent-loops/the-industry-map/s03-03-the-shape-of-the-stack","xp":1,"hint":[],"personalize":false,"phase":"check","estSeconds":200,"concept":"where-value-accrues","body":"$45","cta":"Got it","runnable":true}],"xpTotal":4}],"liveLessonCount":6,"liveStepCount":48,"xpTotal":154}},"chapter":"$1c:2:props:tree:detail","lesson":"$1c:2:props:tree:detail:lessons:1","step":"$1c:2:props:tree:detail:lessons:1:steps:1","stepIndex":1,"nav":{"tocNext":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":2},"tocPrev":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":0},"nextByPath":{"developer":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":2},"marketer":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":2},"designer":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":2},"customer-service":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":2},"copywriter":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":2},"data-analyst":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":2},"project-manager":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":2},"hr-specialist":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":2},"operations":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":2},"lawyer":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":2}},"prevByPath":{"developer":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":0},"marketer":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":0},"designer":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":0},"customer-service":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":0},"copywriter":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":0},"data-analyst":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":0},"project-manager":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":0},"hr-specialist":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":0},"operations":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":0},"lawyer":{"chapterSlug":"agent-loops","lessonSlug":"multi-step-tools","stepIndex":0}}}}]]

Multi-step tools — when one tool isn't enough — step 2 of 9