# When `assert` runs out of road, send in another model Lesson 01 ranked four eval patterns by strictness: exact match, contains, shape, judge. The first three are mechanical — string compare, regex, schema validate. The fourth is *another model* reading your model's output and grading it. That's LLM-as-judge. You reach for it when the answer is correct in many forms ("explain photosynthesis to a 5-year-old," "translate this email," "draft a support reply") and string equality is useless. The 2023 paper that made this respectable, *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena* (Zheng et al., NeurIPS 2023, arxiv.org/abs/2306.05685), showed that GPT-4-as-judge agreed with human raters on **over 80%** of comparisons — the same level humans agree with each other. That's the green light the field needed. It's also the pattern that ships the most subtle bugs. ## Two shapes you'll see in the wild | Shape | Question asked | Output | |---|---|---| | **Pairwise** | "Given outputs A and B, which is better?" | `A` / `B` / `tie` | | **Rubric** | "Score this output against this rubric." | `pass` / `fail` (+ critique) | Pairwise is what Chatbot Arena uses. Cheap relatively (you're not generating a number, just a vote), more consistent than absolute ratings — humans are better at "this one over that one" than "is this a 7 or an 8." Cost: O(n²) comparisons, no absolute quality floor. Rubric is what you want for CI and regression evals. One number per case, plug into pytest, fail the build if the rate drops. Hamel Husain's blunt rule from his eval methodology guide: **use binary pass/fail, not 1–5 Likert**. Likert scores drift, the difference between a 3 and a 4 is undefined, and averaging them produces false precision. Binary forces you to write the rubric until it's unambiguous. (hamel.dev/blog/posts/llm-judge/) ## What everyone calls it | Framework | Primitive | |---|---| | **LangSmith** | `evaluators` — `LLMEvaluator`, custom rubric configs | | **OpenAI Evals** | `graders` — `score_model` / `label_model` | | **Inspect AI** (UK AISI) | `scorers` — `model_graded_qa()`, `model_graded_fact()` | | **Promptfoo** | `assert: llm-rubric`, `assert: factuality` | | **DeepEval** | `GEval` (Liu et al. *G-Eval* paper, arxiv.org/abs/2303.16634) | | **Anthropic SDK** | No native primitive — use `tools=[...]` to force structured verdict (the "tool-use trick") | Same primitive everywhere: send the input + output + rubric, get a structured verdict back. You'll write the framework-free version in step 8. ## The trap that defines this lesson LLM-as-judge has four documented biases — position, length, self-preference, style. They're not edge cases; they're the default. A judge will pick the *first* output you show it more often than chance. It will pick the *longer* output regardless of whether length matters. It will rate outputs from its own model family higher. Step 3 names them with citations and step 7 fixes the biggest one in code. ## What you'll build A `judge_rubric(question, answer, rubric)` that returns a binary verdict + critique. Then a fix for the Likert-score trap. Then a fix for position bias (the "run both orders, count consistent wins" mitigation from Zheng et al.). Then a real pairwise judge that detects its own bias and votes `tie` when it disagrees with itself.2c:Tee8,# The four biases that make this pattern hard LLM judges are not neutral graders. They have measurable, named biases that show up in production every time. The first three were identified in the 2023 MT-Bench paper that legitimized the pattern. The fourth has emerged more recently and is, surprisingly, the biggest. ## Position bias The judge prefers whichever response is shown first. Zheng et al. (2023, arxiv.org/abs/2306.05685) measured this directly: > "GPT-4 consistency on position-swapped pairs jumps from 65.0% to > 77.5% with few-shot prompting." That means **35% of close pairs flipped** when you swapped which output came first. With prompting tricks you can get it down to ~20%. You never get it to zero. **Mitigation: run both orders.** Run `judge(A, B)` *and* `judge(B, A)`. Only count cases where the verdict was *consistent* in both orders. Cases where the judge flipped get scored as `tie` — because effectively the judge is guessing. ## Length / verbosity bias The judge prefers longer answers even when shorter would be correct. The MT-Bench paper demonstrates this with a "repetitive list attack" — pad a response with redundant bullet points and watch the judge prefer it. This is why Hamel Husain ranks judges by their resistance to verbosity attacks: a good judge reads the rubric and cares about correctness, not word count. **Mitigation: write the rubric to forbid verbosity.** "Reward correct, concise answers — penalize redundancy." Judges follow explicit rubrics surprisingly well; they invent the wrong rubric when you don't supply one. ## Self-preference bias The judge favors outputs from its own model family. GPT-4 rates GPT-4 outputs higher than they deserve; Claude rates Claude outputs similarly. The G-Eval paper (Liu et al. 2023, arxiv.org/abs/2303.16634) flagged the same pattern: > "LLM-based evaluators having a bias towards the LLM-generated > texts." Recent work (arxiv.org/abs/2410.21819) confirmed self-preference is real even when models can't see the source. The bug pattern in production: you train a system, the same model judges itself, pass rate looks great. Then you swap to a different judge, pass rate drops 30+ points. **Mitigation: use a held-out judge model.** When the system uses GPT-4o for generation, use Claude or Gemini for the judge. Different weights, different biases — the *combination* is more honest than either alone. ## Style bias (the one nobody warned you about) Judges over-weight confident, well-formatted prose. A 2026 preprint (Soumik, arxiv.org/abs/2604.23178) measured **effect sizes of 0.76–0.92 for style bias**, vs ≤0.04 for position bias — roughly 20× larger. (Caveat: that's a recent single-author preprint, so treat the exact numbers as provisional; the peer-reviewed neighbor, Park et al.'s OffsetBias at EMNLP 2024, catalogues the same bias family without those effect sizes.) Either way the direction is clear: a polished but wrong answer beats a correct but plain one. **Mitigation: rubric explicitness, again.** "Score on factual correctness; ignore tone and formatting." It helps. It doesn't fix the problem. Style bias is the open research problem in this pattern, and the production move is to *combine* LLM-as-judge with mechanical checks (string match, schema validate, citation check) rather than rely on the judge alone. ## What this means for your code The biases aren't optional considerations. They're the default behavior. Steps 4–7 will surface them in code: predict that a judge picks the long output regardless of order, then build the both-orders mitigation that catches its own bias and reports `tie`. Rule of thumb that makes this whole pattern usable: **never trust a single judge call on a single ordering**. Two calls minimum, both orders, count only consistent wins.2d:T51e,def judge_rubric(question, answer, rubric): if "Paris" in answer: return {"passed": True, "critique": "Names Paris."} return {"passed": False, "critique": "Does not name Paris."} cases = [ {"question": "capital of France?", "answer": "Paris.", "rubric": "Reward correct answer."}, {"question": "capital of France?", "answer": "Lyon, I think.", "rubric": "Reward correct answer."}, {"question": "capital of France?", "answer": "Paris, in northern France.", "rubric": "Reward correct, concise answer."}, {"question": "capital of France?", "answer": "Some city in the south, maybe.", "rubric": "Reward correct answer."}, ] def run_judge_suite(cases): passed_count = 0 for c in cases: v = judge_rubric(c["question"], c["answer"], c["rubric"]) if v["passed"]: passed_count += 1 total = len(cases) failed_count = total - passed_count pass_rate = round(passed_count / total, 2) if total else 0.0 return { "total": total, "passed": passed_count, "failed": failed_count, "pass_rate": pass_rate, } result = run_judge_suite(cases) print(f"total={result['total']} passed={result['passed']} failed={result['failed']} pass_rate={result['pass_rate']}") 2e:T475,def fake_judge(question, first, second): return "A" if len(first) > len(second) else "B" def pairwise(question, output_a, output_b): # round 1: output_a in slot A, output_b in slot B. slot_winner_1 = fake_judge(question, output_a, output_b) round_1_winner = "a" if slot_winner_1 == "A" else "b" # round 2: output_b in slot A, output_a in slot B. slot_winner_2 = fake_judge(question, output_b, output_a) round_2_winner = "b" if slot_winner_2 == "A" else "a" if round_1_winner == round_2_winner: return round_1_winner return "tie" cases = [ ("Pick the better answer.", "A long detailed correct answer here.", "Short."), ("Pick the better answer.", "Same exact length here", "Same exact length here"), ("Pick the better answer.", "Brief.", "A long detailed correct answer here."), ] for i, (q, a, b) in enumerate(cases, start=1): result = pairwise(q, a, b) label = "a wins (consistent)" if result == "a" else \ "b wins (consistent)" if result == "b" else \ "tie (position-biased — judge disagrees with itself)" print(f"case {i}: {label}") 2f:T1073,# The vibes era: 2022 to 2024 For about eighteen months after ChatGPT shipped, the entire LLM industry ran on a methodology that, in retrospect, was a methodology only by accident. It had a name — **prompt engineering** — and a job title to match. LinkedIn filled with "Prompt Engineer" titles attached to six-figure salaries. Twitter filled with thread after thread of "the 17 magic phrases that 10x ChatGPT." A cottage industry of "prompt marketplaces" appeared, selling jailbreaks and templates for $9.99. The underlying loop looked like this: 1. Write a prompt. 2. Run it on three or four inputs you happen to remember. 3. If the output looks good, ship it. 4. When a customer complains, go back to step 1. That's it. That was the methodology. Nobody called it "vibes" at the time — at the time it just felt like the work. But it was vibes. The quality bar was "does this feel right when I read it." The regression detection was "did anyone complain yet." The release gate was "the founder said go." ## Why the vibes era worked at first It worked for the same reason any unscalable practice works at small scale: the surface area was small. The first wave of LLM products were demos and prototypes. A demo only has to work for the demo. A prototype only has to work for the screenshot you put in the deck. If your prompt fails on input #47, you'll never know, because input #47 was never going to happen in front of an investor. It also worked because the models themselves were a moving target in a way that hid product quality issues. Every six months a new model dropped that was strictly better than the last. If your prompt had a bug, you could often just wait three months and let the next model patch over it. That's not engineering. That's hoping. But it felt like progress, and the chart of model capability over time made everyone feel like geniuses. ## Why the vibes era failed for products The moment you go from demo to product, three things break: **Vibes don't survive a model swap.** Every prompt engineer of the 2023 era has the same scar: the day OpenAI deprecated `gpt-3.5-turbo-0301` and their carefully tuned prompt started returning garbage. Vibes are local to a model version. The minute the model changes — and models change every quarter — your "battle-tested" prompt is no longer battle-tested against anything. Without an eval suite, you don't know *how* it broke. You only know that @@pd-path:path.domain.evalsFlowing@@. **Vibes don't catch regressions.** The most expensive bug in LLM products is the silent one: your prompt got "improved" last Tuesday to fix a customer complaint, and now it's worse on a different input nobody noticed. There is no way to detect this without a regression suite. None. The vibes era had no answer for this problem — the answer was "hope the next complaint surfaces it quickly." **Vibes can't be reviewed.** A PR that says "I changed the prompt from X to Y, it feels better to me" is not reviewable. The reviewer has no basis to push back. They either trust you or they don't. This killed the engineering culture inside AI teams. Senior ICs could not do their job — could not catch bad changes, could not enforce a quality bar — because there was nothing to point at. Process collapsed into "whoever yells the loudest wins the prompt fight." ## The cultural tell The clearest sign that prompt engineering was about to collapse as a discipline was the way prompt engineers talked about their work. They described it the way magicians describe magic — as art, as feel, as something that couldn't quite be explained. "You have to *know* what the model wants." "Some prompts just *have it*." "I can tell within a sentence whether a prompt will work." This is the language of a field that hasn't been measured yet. Every discipline goes through this phase before it becomes engineering. The chefs before recipes. The pilots before instruments. The doctors before germ theory. The phase ends when somebody figures out how to measure the thing — at which point all the people who described it as art either retool or get washed out. For LLMs, that moment came in 2024. It has a name. We'll cover it in the next step.30:T1378,# The eval turn: 2024 onward If the prompt-engineering era had a single obituary, it was a blog post Hamel Husain published in March 2024 titled **"Your AI Product Needs Evals"** (hamel.dev/blog/posts/evals/). Hamel had spent the prior eighteen months consulting for companies that had shipped LLM features and were now drowning in production failures. The post was not a thought piece. It was the field notes of someone who had watched a dozen teams discover, independently, that the methodology they'd been using had no answer to the question *"is the new prompt better than the old one?"* The thesis was simple and, by mid-2024, controversial: > If you do not have an eval suite, you do not have an AI product. > You have a demo that happens to be in production. Hamel's framing landed because it named the thing every senior IC at every AI company was already feeling. The vibes loop was not sustainable. The next prompt change was always a coin flip. @@pd-path:path.domain.evalsStream@@ The eval turn was the realization that the LLM industry had to grow up — fast — or the products would not survive contact with the second year of customer use. ## The supporting cast Hamel's post wasn't alone. The same months produced a small library of essays and papers that defined the new discipline: **Eugene Yan** had been writing about evals from the ML engineering side for almost a year — most famously the "Task-Specific LLM Evals that Do & Don't Work" piece (eugeneyan.com/writing/evals/), which catalogued what was actually working in production at scale. Yan's contribution was the *taxonomy* — reference-free vs reference-based, direct grading vs pairwise, code-graded vs model-graded — that gave the discipline a shared vocabulary. **Shreya Shankar** (Berkeley PhD; CMU faculty candidate 2026) published the paper that gave the field its academic spine: **"Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences"** (Shankar et al., UIST 2024, arxiv.org/abs/2404.12272). The paper formalized the trust problem at the heart of LLM-as-judge — if you're using a model to grade a model, who grades the grader? — and produced EvalGen, the tool that lets you iteratively align an LLM judge to human preferences. After this paper, "I'll just use LLM-as-judge" stopped being a one-liner answer and started being a process. **Braintrust** (braintrust.dev) was founded in late 2023 and shipped its product around the same window as Hamel's post. It was the first of the eval-native dev tools — not a generic logging platform, not a LangChain plugin, but a tool whose entire reason to exist was running eval suites against prompt changes. Within six months, every serious AI infra company either had an eval product (LangSmith, Galileo, Patronus) or was building one (OpenAI's evals SDK, the Inspect AI framework from the UK AISI). **Anthropic** put the period on the era in December 2024 with the publication of **"Building Effective Agents"** (anthropic.com/engineering/building-effective-agents). The essay is worth rereading after you've finished this chapter. It is a how-to guide for shipping agentic systems written by the company shipping the most-deployed agentic system on the market, and the word "eval" or "evaluation" appears in nearly every section. The mantra is explicit: > the most successful implementations weren't using complex > frameworks or specialized libraries. Instead, they were building > with simple, composable patterns. And the eval thread runs through the whole post — measure the outputs at every step, or you're flying blind. Translation: evals are not the last thing you do. Evals are the *first* thing you do. The prompt comes after. ## What changed in the industry By mid-2025, the cultural shift inside AI engineering was complete: - **Job postings changed.** "Prompt Engineer" stopped appearing as a title. "AI / LLM Engineer" started carrying an explicit eval-suite requirement in the JD. By late 2025, "can you architect an eval harness for our use case" was a standard interview loop. - **Tooling changed.** Every serious LLM framework now ships with evals as a first-class primitive (see lesson 02 step 1 for the table). The frameworks that didn't add evals got bypassed. - **Internal practice changed.** Eval-mature teams started running their eval suite *on every prompt change*. Then *on every model version they tested*. Then *on every CI run*. Then *as a release gate*. The discipline tightened in stages. Hamel's post named the field. Yan's posts gave it vocabulary. Shankar's paper gave it academic rigor. Braintrust gave it tooling. Anthropic's essay gave it the official seal. By the end of 2024, the question stopped being "should I write evals?" and started being "why don't you have them yet?" The next step is interactive. You're going to play that question forward — to a model-swap scenario — and see which kind of team survives.31:T1152,# Three case studies: who shipped, who walked back Theory only takes you so far. The eval turn is a real thing because specific companies, with specific products, learned specific lessons the hard way. Three worth knowing. ## Cursor: the eval harness that ate the IDE Cursor is the AI-first code editor practitioners keep pointing at when they talk about eval-heavy product development. The exact shape of Cursor's internal harness is private, so treat the public lesson as a product pattern, not a sourced teardown: mature AI products need regression suites that run against prompt and model changes before users see the breakage. The interesting pattern is not "they have evals." By 2025, nearly everyone claims that. The mature pattern is treating the eval suite as the *primary artifact*. Prompts, harnesses, and model bindings can all change. The eval suite persists because it encodes accumulated knowledge of "what the product is supposed to do." The eval suite is the specification. The prompt is implementation detail. This is the bar. If your eval suite cannot serve as a spec, you do not have an eval suite. You have a smoke test. ## Klarna: the walk-back In February 2024, Klarna announced — proudly — that its OpenAI-powered customer service bot had replaced the work of 700 full-time agents. The press loved it. By mid-2025, Klarna was quietly hiring human agents back. Sebastian Siemiatkowski (CEO) acknowledged in a *Bloomberg* interview that AI-only support had hurt service quality and the company was rebuilding a human-in-the-loop tier. There are a half-dozen lessons in the Klarna walk-back, but the one relevant to this chapter is this: Klarna's AI bot did not get *worse* over time. The models, in fact, got better. What happened was that Klarna's eval coverage was demo-grade — they had launched on the inputs they'd tested, and the long tail of real customer interactions hit failure modes their internal evals had never flagged. The bot was confidently wrong on things it should have escalated, confidently bland on things customers needed empathy on, and confidently helpful on things that needed a refund-policy lookup it didn't have. Without eval coverage on those edge categories, the team couldn't even *see* the problem until customer satisfaction metrics fell off a cliff. Klarna didn't fail because they used AI. They failed because they shipped AI without an eval suite that could measure the failure modes that actually mattered to customers. The walk-back wasn't a referendum on AI customer service. It was a referendum on shipping-before-measuring. ## Anthropic: "eval-driven development" becomes a mantra Anthropic — the company that publishes "Building Effective Agents" — has been explicit about eval-driven development as *practice*, not just a slogan. The pattern is consistent across their public guidance on agent design: 1. Start with a small, hand-built eval set (50 cases is plenty). 2. Build the simplest agent that passes 80% of the set. 3. Look at the failures. Find the common failure modes. 4. Either fix the agent or, more often, *expand the eval set* with cases that capture the failure mode. 5. Iterate. The mantra is: **evals come first, prompts come second, framework choice comes third**. If you don't know what you're trying to measure, no amount of framework cleverness will save you. ## The pattern across all three The pattern across Cursor, Klarna, and Anthropic is the same — and it's the central, slightly counterintuitive thesis of the eval era: > **The teams with the heaviest eval discipline ship faster, not > slower.** This sounds wrong on the first hearing. Adding tests slows you down, doesn't it? In the LLM context, no — and the reason is that the *alternative* to eval discipline is not "no overhead, just speed." The alternative is *the silent regression*: the change you shipped that worked on three inputs and broke seventeen, that you didn't notice for six weeks, that you then spent two months debugging because you had no way to isolate when it started. Eval-mature teams trade five minutes of CI time for not having that six-week debugging cycle. The net is enormous velocity. The teams without evals look fast from the outside in week one. They are dead in the water by month six. In the next step we'll codify this into a question you can apply to any team you join or start: are they eval-first, or prompt-first?32:T503,# four team workflows for shipping a new AI feature. # only one of them is eval-first. which? workflows = { "A": { "name": "FlowA", "steps": [ "1. write a prompt", "2. try it on a few inputs", "3. ship if it looks good", "4. write evals 'eventually' when there's time", ], }, "B": { "name": "FlowB", "steps": [ "1. write 30 input/expected pairs that define what success means", "2. run baseline (no prompt) — confirm 0% pass", "3. write the simplest prompt that passes the suite", "4. only ship when the suite passes", ], }, "C": { "name": "FlowC", "steps": [ "1. write a prompt", "2. ship it", "3. wait for customer complaints", "4. add an eval case per complaint", ], }, "D": { "name": "FlowD", "steps": [ "1. write a prompt", "2. ask GPT-4 to grade its own output", "3. if GPT-4 says it's good, ship", "4. no ground-truth cases on disk", ], }, } for k, v in workflows.items(): print(f"{k}: {v['name']}") for s in v['steps']: print(f" {s}") 33:T1399,# Why "eval engineer" eats "prompt engineer" The labor-market argument is the cleanest way to understand why the field has moved on. It has nothing to do with what's cool and everything to do with what's *durable*. ## Durable artifacts vs ephemeral ones A prompt is an ephemeral artifact. It is tuned for a specific model, a specific use case, often a specific customer segment. When the model changes — and the model changes every quarter — the prompt has to be retuned. When the use case shifts, the prompt has to be retuned. When the product team adds a new feature, the prompt has to be retuned. The half-life of a production prompt in 2025 is measured in weeks, sometimes days. An **eval suite** is a durable artifact. It encodes "what does this feature need to do for our customers" in a form that survives every model swap, every prompt rewrite, every framework migration. The eval suite a team wrote against GPT-4 in 2024 still runs against GPT-5 in 2026, against Claude Opus 5 in 2027, against whatever ships next. It is the *specification* of the product. The prompt is the *implementation*. This is the same distinction that makes test suites durable in traditional software engineering. A C codebase from 1995 might be unreadable today, but the test suite — if there is one — still tells you what the code was supposed to do. The implementation gets rewritten in Rust. The test suite endures. ## What this means for who gets hired A "prompt engineer" was a person who knew which incantations made GPT-3.5 sit up and beg. That skill aged badly. The incantations that worked on GPT-3.5 are useless on GPT-5. The "trick" of prompt engineering — knowing the model's quirks — is exactly the thing that becomes worthless every six months. An **eval engineer** — or "AI Engineer" or "LLM Engineer," titles vary — is a person who can: - Look at a fuzzy business problem and articulate it as a measurable outcome. - Build a test set that covers the actual failure modes. - Choose the right judge for each case (exact match, schema, LLM-as-judge, human review). - Set up CI that runs the suite on every prompt change. - Triage failures and decide which are bugs vs which are eval gaps. - Maintain the suite as the product and the customer base evolve. That skillset compounds. The eval suite a senior IC builds in year one is still earning compound returns for the company in year five. The cost to replace that person is enormous, because the eval suite encodes years of accumulated product knowledge. A prompt? Anyone can rewrite a prompt. A prompt engineer is fungible on the timescale of one model release. This is the labor-market argument. It is not "AI eats programmers." It is "the durable skill in AI eats the ephemeral skill in AI." The people writing eval suites today are doing what test engineers discovered in the early 2000s: building the artifact that keeps paying off long after the prompt-of-the-month is gone. ## Tie to the AI-native thesis This connects directly to the AI-native discussion from @@pd-path:path.ref:agent-harnesses@@. That chapter argued that companies have "process debt" — the unspoken-in-someone's-head workflows that bottleneck AI adoption. The eval-suite equivalent is **eval debt**: the gap between "we think our AI feature works" and "we can prove it works on the inputs that matter to our customers." Every AI team has eval debt. The eval-mature ones have a number on it (we have 200 cases, we add 5 per week, we run CI on every prompt change). The eval-debt-laden ones don't even know how big the problem is, because they have nothing to measure. The strategic implication for your career is the same one the AI-native thesis applied to companies: **the durable career assets in AI are the ones that survive a model swap.** That means: - Knowing how to build eval suites (durable; survives every release) - Knowing how to design agent harnesses (durable; survives prompt rewrites) - Knowing how to do evidence-based product work in AI (durable; survives the hype cycle) Not durable: knowing the magic phrase that makes GPT-4 produce better JSON than GPT-4-turbo. By the time you've memorized it, it's already a fossil. ## The synthesis The 2022-2024 era rewarded people who could make a model do a thing once. The 2024-onward era rewards people who can prove a model will do a thing reliably across the inputs the business cares about. That is a different skill, a different role, and a different career path. Writers and teams like Hamel Husain, Eugene Yan, Shreya Shankar, Anthropic, and Braintrust pushed the field toward this shape from different angles: production stories, ML practice, research, public agent guidance, and tooling. The field has chosen. The prompt engineer of 2023 is a 2024 fossil. The eval engineer of 2025 is the senior IC of 2030. In the next step, you'll build the audit you can run on any team — yours, your future employer's, your startup's — to figure out which era they're operating in.34:T43a,eval_mature_co = { "name": "EvalMatureCo", "has_test_set": True, "has_judge_prompt": True, "ci_runs_evals": True, "tracks_eval_history": True, "eval_count_per_feature": 50, } vibe_co = { "name": "VibeCo", "has_test_set": False, "has_judge_prompt": False, "ci_runs_evals": False, "tracks_eval_history": False, "eval_count_per_feature": 0, } def eval_readiness(team): score = 0 if team.get("has_test_set"): score += 25 if team.get("has_judge_prompt"): score += 15 if team.get("ci_runs_evals"): score += 25 if team.get("tracks_eval_history"): score += 15 if team.get("eval_count_per_feature", 0) >= 20: score += 20 if score >= 80: verdict = "eval-mature" elif score >= 50: verdict = "eval-aware" elif score >= 20: verdict = "eval-curious" else: verdict = "vibes era" return {"score": score, "verdict": verdict} print(f"EvalMatureCo: {eval_readiness(eval_mature_co)}") print(f"VibeCo: {eval_readiness(vibe_co)}") 35:T40d,Final drill. Synthesize the chapter's history into one function: `triage_teams(teams)` that takes a list of team profiles and returns a dict with two fields: - `verdicts`: a dict mapping each team's `name` to its verdict string (same four tiers as step 07) - `most_at_risk`: the `name` of the team with the LOWEST readiness score (the team most likely to break on the next model launch) Score each team using the same rules as step 07: - `has_test_set` True: +25 - `has_judge_prompt` True: +15 - `ci_runs_evals` True: +25 - `tracks_eval_history` True: +15 - `eval_count_per_feature` >= 20: +20 Verdict tiers: >=80 `"eval-mature"`, >=50 `"eval-aware"`, >=20 `"eval-curious"`, else `"vibes era"`. On ties for lowest score, return the FIRST tied team (Python's `min` with `key=` preserves stable order). Five teams run. Expected output: ``` verdicts: {'CursorClone': 'eval-mature', 'SeriesBAI': 'eval-mature', 'SeedAI': 'eval-curious', 'ConsultingShop': 'eval-curious', 'PromptShopCo': 'vibes era'} most at risk: PromptShopCo ``` 36:T68e,teams = [ {"name": "CursorClone", "has_test_set": True, "has_judge_prompt": True, "ci_runs_evals": True, "tracks_eval_history": True, "eval_count_per_feature": 80}, {"name": "SeriesBAI", "has_test_set": True, "has_judge_prompt": True, "ci_runs_evals": True, "tracks_eval_history": False, "eval_count_per_feature": 30}, {"name": "SeedAI", "has_test_set": True, "has_judge_prompt": False, "ci_runs_evals": False, "tracks_eval_history": False, "eval_count_per_feature": 10}, {"name": "ConsultingShop", "has_test_set": True, "has_judge_prompt": True, "ci_runs_evals": False, "tracks_eval_history": False, "eval_count_per_feature": 5}, {"name": "PromptShopCo", "has_test_set": False, "has_judge_prompt": False, "ci_runs_evals": False, "tracks_eval_history": False, "eval_count_per_feature": 0}, ] def score_team(t): s = 0 if t.get("has_test_set"): s += 25 if t.get("has_judge_prompt"): s += 15 if t.get("ci_runs_evals"): s += 25 if t.get("tracks_eval_history"): s += 15 if t.get("eval_count_per_feature", 0) >= 20: s += 20 return s def verdict_for(score): if score >= 80: return "eval-mature" if score >= 50: return "eval-aware" if score >= 20: return "eval-curious" return "vibes era" def triage_teams(teams): verdicts = {t["name"]: verdict_for(score_team(t)) for t in teams} most_at_risk = min(teams, key=score_team)["name"] return {"verdicts": verdicts, "most_at_risk": most_at_risk} result = triage_teams(teams) print(f"verdicts: {result['verdicts']}") print(f"most at risk: {result['most_at_risk']}") 37:Tc5a,# The last door before your users You've written evals (lesson 01), hired a judge and audited its biases (lesson 02). One question is left: who decides whether the new model actually ships? If the answer is "whoever feels good about it on Friday," everything upstream was decoration. A **validation gate** is that decision, written down as code. It is exactly three things: 1. **A threshold** — a number, chosen before you saw the candidate's score. Chosen after, it's not a threshold; it's a justification. 2. **An eval set** — fixed, versioned, frozen. If the cases change every run, the score isn't comparable to last week's and the threshold is measuring nothing. 3. **A decision rule** — what happens on failure. "Block the deploy" is a rule. "Post in Slack and see" is not. Run the editor. The candidate scores 0.83 against a 0.85 floor, and the gate says `BLOCK` — no meeting required. ## Absolute gates and regression gates An **absolute gate** compares the candidate to a fixed bar: accuracy at least 0.85, zero schema violations, refusal rate under 2%. It answers *"is this good enough to face users?"* A **regression gate** compares the candidate to the *last shipped model* on the same frozen set: new score must be at least old score minus a small tolerance. It answers *"is this at least as good as what users already have?"* — which is the question your users are silently asking every time you deploy. You want both. A candidate can clear a 0.80 floor while quietly scoring four points below the model it replaces. The absolute gate smiles; the regression gate blocks. You'll build exactly that case in step 3. ## Overrides happen. Unrecorded overrides are the sin. Sometimes shipping past a red gate is the right call — the regression is confined to a locale you're sunsetting, the fix ships tomorrow, the outage is worse than the dip. Fine. But the override is a decision by a *person*, and it gets recorded: **who overrode, why, and which gates were red at the time.** Same receipts culture as the judge lesson — a verdict without a critique is noise, and an override without a name and a reason is just a gate someone deleted quietly. Six months later, when someone asks why v9 shipped below the bar, the deploy log answers instead of nobody remembering. ## The theater gate Here's the failure mode that makes this lesson worth its slot: teams add a gate to satisfy a checklist, set the threshold at a level the worst model they've ever trained would clear, and enjoy years of green checkmarks. **A gate that cannot fail is not protecting the deploy — it's decorating it.** The test is brutal and simple: when did this gate last block anything? If the answer is "never, and it never could," you've built a ribbon to cut, not a door that locks. Step 5 has you fix one that computes its threshold from the candidate's own score — always green, forever, by construction. Six steps ahead: spot the theater gate in a lineup, predict an absolute-passes-regression-fails split, wire the regression comparison, un-rig the rigged gate, build the gate runner, and finish with an override log that names names.1c:[["$","$L24",null,{"payload":{"catalog":"A gate that can't fail isn't a gate (step 2/7) · test the agent before you trust it · promptdojo","byPath":{"developer":"A gate that can't fail isn't a gate (step 2/7) · test the agent before you trust it · promptdojo","marketer":"A gate that can't fail isn't a gate (step 2/7) · test the campaign before you ship it · promptdojo","customer-service":"A gate that can't fail isn't a gate (step 2/7) · test the reply before you send it · promptdojo","copywriter":"A gate that can't fail isn't a gate (step 2/7) · test the copy before you publish it · promptdojo","data-analyst":"A gate that can't fail isn't a gate (step 2/7) · test the analysis before you cite it · promptdojo","project-manager":"A gate that can't fail isn't a gate (step 2/7) · test the build before you ship it · promptdojo","hr-specialist":"A gate that can't fail isn't a gate (step 2/7) · test the people draft before you send it · promptdojo","lawyer":"A gate that can't fail isn't a gate (step 2/7) · test the draft before you file it · promptdojo"}}}],["$","script",null,{"type":"application/ld+json","dangerouslySetInnerHTML":{"__html":"$25"}}],["$","$L26",null,{"tree":{"toc":{"version":"0.1.0","generatedAt":"1970-01-01T00:00:00.000Z","chapters":[{"number":0,"slug":"before-you-build","title":"from worker to builder","blurb":"stop doing repeated work by hand. this chapter is the map: the builder loop, old workplace deliverables turned into site artifacts, and the reusable tool card. if you are starting here, those moves are the lesson. if you already ran a studio, you already used the loop.","status":"live","lessonCount":10,"stepCount":14,"liveLessonCount":10,"liveStepCount":14,"estMinutes":74,"xpTotal":36},{"number":1,"slug":"read-the-script","title":"read the script — your first code-reading lab","blurb":"ai hands you small scripts. this lab covers the three reads that matter: what the names hold, whether the function actually returns, and which branch of the if-chain really fires. if you are starting here, this labels the reads. if you already ran a studio, you already wrote these.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":21,"xpTotal":78},{"number":2,"slug":"read-the-data-shapes","title":"read the data shapes — the json ai just dumped","blurb":"every api response, config file, and llm reply is lists and dicts nested in a tree. read the shape on sight, walk it with a loop, and catch the wrong-layer bug ai loves to ship.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":23,"xpTotal":80},{"number":3,"slug":"read-the-failure","title":"read the failure — cursor wrote this and crashed","blurb":"when python crashes, it tells you exactly what happened and where. most non-engineers panic at the wall of text — and ai 'fixes' the crash by hiding it. learn to read the failure, then catch only what's worth catching.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":24,"xpTotal":83},{"number":4,"slug":"read-the-pipeline","title":"read the pipeline — files in, api calls out","blurb":"every real script reads a file or calls an api — usually both. learn the with-block, the status-code families, and the retry rules: the seatbelts ai forgets to put on.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":26,"xpTotal":83},{"number":7,"slug":"mutation-and-state","title":"mutation — why your code mysteriously breaks","blurb":"when a list inside a function changes the list outside the function, that's mutation. ai does this constantly without flagging it, and it's the bug class that takes the longest to find.","status":"live","lessonCount":2,"stepCount":9,"liveLessonCount":2,"liveStepCount":9,"estMinutes":10,"xpTotal":32},{"number":13,"slug":"llm-apis","title":"llm apis — talking to claude and openai","blurb":"every ai feature you ship eventually calls a model api. learn the messages pattern, how to read the response, and the four lines ai writes every single time.","status":"live","lessonCount":4,"stepCount":27,"liveLessonCount":4,"liveStepCount":27,"estMinutes":45,"xpTotal":87},{"number":14,"slug":"structured-output","title":"structured output — making the model return json","blurb":"free-form text breaks every pipeline. learn the schema-first pattern ai uses to get reliable json back, validate it with pydantic, and catch the model's lies before they hit prod.","status":"live","lessonCount":3,"stepCount":25,"liveLessonCount":3,"liveStepCount":25,"estMinutes":38,"xpTotal":80},{"number":15,"slug":"mcp","title":"mcp — the model context protocol","blurb":"mcp is the new standard for plugging tools and data sources into ai agents. learn what an mcp server actually is, how claude code lists tools, and why this is replacing one-off integrations everywhere.","status":"live","lessonCount":3,"stepCount":26,"liveLessonCount":3,"liveStepCount":26,"estMinutes":40,"xpTotal":85},{"number":16,"slug":"agent-loops","title":"agent loops — tool_use and the request/tool/respond cycle","blurb":"an agent isn't magic. it's a while loop. learn the actual cycle claude code, cursor, and every other agent uses: model returns tool_use, you run the tool, you send the result back, repeat until end_turn.","status":"live","lessonCount":6,"stepCount":48,"liveLessonCount":6,"liveStepCount":48,"estMinutes":71,"xpTotal":154},{"number":17,"slug":"git-and-github","title":"git and github cli — the ai builder's actual workflow","blurb":"cursor and claude code commit on your behalf. reading those commits — and undoing the bad ones — is your job. learn the three-state model, the commands you'll run every day, and what `gh` does that `git` can't.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":36,"xpTotal":79},{"number":18,"slug":"secrets-and-env","title":"secrets — .env, api keys, and what not to commit","blurb":"ai ships keys to github all the time. learn the .env pattern, why os.getenv is non-negotiable, what to do when a key leaks, and the gitignore lines you need on day one.","status":"live","lessonCount":2,"stepCount":16,"liveLessonCount":2,"liveStepCount":16,"estMinutes":16,"xpTotal":58},{"number":19,"slug":"prompting","title":"prompting ai coding tools","blurb":"an AI coding tool writes and edits with you — Cursor is one brand, Claude Code is another. the difference between a one-shot session and a four-hour spiral is almost always the first prompt.","status":"live","lessonCount":5,"stepCount":36,"liveLessonCount":5,"liveStepCount":36,"estMinutes":59,"xpTotal":118},{"number":20,"slug":"agent-traces","title":"reading agent traces and telemetry","blurb":"when an agent fails, the trace tells you exactly where. learn to read tool calls, tool results, and stop reasons — the json breadcrumbs every agent leaves behind.","status":"live","lessonCount":3,"stepCount":25,"liveLessonCount":3,"liveStepCount":25,"estMinutes":30,"xpTotal":88},{"number":21,"slug":"evals","title":"test the agent before you trust it","blurb":"same input, check the output, pass or fail. don't ship on a demo that felt right.","status":"live","lessonCount":5,"stepCount":34,"liveLessonCount":5,"liveStepCount":34,"estMinutes":57,"xpTotal":112},{"number":22,"slug":"context-and-retrieval","title":"context and retrieval — feeding the model real data","blurb":"rag without the overengineering. chunking, embeddings, vector search, and the small set of patterns that make a model answer from your data instead of its training set.","status":"live","lessonCount":5,"stepCount":44,"liveLessonCount":5,"liveStepCount":44,"estMinutes":75,"xpTotal":144},{"number":23,"slug":"production-tradeoffs","title":"production tradeoffs — cost, latency, quality","blurb":"the three numbers every shipped llm feature lives or dies by. token math, caching, streaming, batching, and the small set of decisions that move the product more than a model swap ever will.","status":"live","lessonCount":4,"stepCount":33,"liveLessonCount":4,"liveStepCount":33,"estMinutes":57,"xpTotal":111},{"number":24,"slug":"debugging-output","title":"debugging broken ai output","blurb":"when the model lies to your customer. the methodology for narrowing down what went wrong, the four most-common breakage classes, and the discipline that separates 'we shipped a fix' from 'we blamed the model and shrugged'.","status":"live","lessonCount":4,"stepCount":34,"liveLessonCount":4,"liveStepCount":34,"estMinutes":61,"xpTotal":112},{"number":25,"slug":"capstone","title":"mid-path build — ship the CLI agent","blurb":"wire it all together. the prompt, the call, the validation, the trace, the eval, the MCP tool. a path checkpoint, not the end of the course: the smallest end-to-end llm feature you could ship to a real user. retrieval and prompt-cache cost work live in later chapters — extend this build with them when you scale past the demo input set.","status":"live","lessonCount":7,"stepCount":62,"liveLessonCount":7,"liveStepCount":62,"estMinutes":100,"xpTotal":219},{"number":26,"slug":"agent-harnesses","title":"agent harnesses — the layer between you and the raw api","blurb":"claude code, cursor, aider, codex cli — they're all the same four layers wrapped around the same model api. learn what those layers are, what each adds, and what you'd build yourself if you had to. when the harness changes the world, the rollback lives in the repo — a runbook a junior can execute at 3am.","status":"live","lessonCount":5,"stepCount":37,"liveLessonCount":5,"liveStepCount":37,"estMinutes":69,"xpTotal":119},{"number":27,"slug":"ai-image-generation","title":"ai image generation — from prompt to production asset","blurb":"the 2026 image model landscape, the prompts that work, and the pipeline that turns one good idea into a hundred ready-to-ship images. nano banana 2 for volume, nano banana pro for fidelity, flux, midjourney, ideogram, gpt-image-2. when each wins and what they cost.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":60,"xpTotal":67},{"number":28,"slug":"ai-video-generation","title":"ai video generation — omni, veo 3.1, sora, and the second wave","blurb":"video is the hardest content type to generate, the most expensive, and the most strategically interesting. learn the 2026 model lineup, the camera-control patterns that separate slop from craft, and the cost math that decides whether your idea is viable.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":50,"xpTotal":61},{"number":29,"slug":"programmatic-design","title":"programmatic design — code-driven video and the new design pipeline","blurb":"ai generates raw assets; code stitches them into something shippable. hyperframes, remotion, claude design — when each tool wins, how they combine, and the data-driven workflows that turn one template into a hundred videos.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":46,"xpTotal":66},{"number":30,"slug":"harness-engineering","title":"harness engineering — the discipline behind the model","blurb":"every coding agent is a model plus a harness. the model is bought; the harness is engineered. learn the craft: how to ratchet rules from failures, fight context rot, design long-horizon loops, wire hooks as enforcement, and read the haas shift that's reshaping what you build vs buy.","status":"live","lessonCount":6,"stepCount":52,"liveLessonCount":6,"liveStepCount":52,"estMinutes":126,"xpTotal":143},{"number":31,"slug":"intro-to-terminal","title":"intro to terminal — the text way to drive your computer","blurb":"the terminal is a text box: type a name, press enter, the computer does it. by the end you can move around your files, make folders, and read files without the mouse. every later tool chapter assumes this.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":26,"xpTotal":50},{"number":32,"slug":"intro-to-claude-cli","title":"intro to claude cli — the door to real building","blurb":"you've used claude in a chat window. the claude cli is the same model with its hands on your actual files. this chapter installs it, signs you in, and runs your first real command. by the end you've watched an ai read, plan, and change things on your machine, and you know when to reach for the cli instead of the chat box.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":32,"xpTotal":48},{"number":33,"slug":"intro-to-codex-cli","title":"intro to openai codex cli — a second tool in the kit","blurb":"you know the claude cli. the openai codex cli does the same job, an ai working in your terminal on your real files, with a different company behind it. this chapter installs it, signs you in, and runs your first command. most of what you already know carries straight over, so this chapter is mostly about what is different and when to reach for which.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":32,"xpTotal":45},{"number":34,"slug":"team-skills","title":"claude skills for teams — playbooks your whole team shares","blurb":"a claude skill is a packaged set of instructions — your team's playbook — that claude loads when it's relevant, so nobody has to re-explain it. this chapter is for people who manage teams. it covers what a skill is, how a team shares and provisions skills, real examples for hr, legal, and ops work, when a skill beats a one-off prompt, and the governance you need before any skill touches real work.","status":"live","lessonCount":4,"stepCount":28,"liveLessonCount":4,"liveStepCount":28,"estMinutes":41,"xpTotal":60},{"number":35,"slug":"dataframes-numpy-pandas","title":"dataframes with numpy and pandas","blurb":"tables are the working surface of applied ml. learn rows, columns, missing values, joins, aggregates, and the dataframe habits ai-generated notebooks assume.","status":"live","lessonCount":6,"stepCount":42,"liveLessonCount":6,"liveStepCount":42,"estMinutes":43,"xpTotal":156},{"number":36,"slug":"sql-for-ml-datasets","title":"sql for ml datasets","blurb":"most training data starts in a database. learn the select, join, filter, aggregate, and leakage traps that decide whether a model is learning signal or nonsense.","status":"live","lessonCount":4,"stepCount":28,"liveLessonCount":4,"liveStepCount":28,"estMinutes":28,"xpTotal":104},{"number":37,"slug":"dataset-pipelines","title":"dataset formats, ingestion, and validation pipelines","blurb":"a dataset is a product surface. build ingestion, validation, partitions, manifests, and checkpoints so the next run is not a mystery.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":38,"slug":"ml-math-and-stats","title":"ml math and statistics that actually show up","blurb":"vectors, probability, distributions, correlation, and uncertainty are not trivia. they are how you read model behavior without worshipping it.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":39,"slug":"supervised-learning-workflows","title":"supervised learning workflows","blurb":"labels, splits, baselines, training, prediction, and evaluation. the supervised workflow is the first complete model loop.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":40,"xpTotal":130},{"number":40,"slug":"unsupervised-learning-and-embeddings","title":"unsupervised learning, embeddings, and recommenders","blurb":"not every useful model has labels. cluster, compare, retrieve, and recommend by turning examples into useful neighborhoods.","status":"live","lessonCount":4,"stepCount":28,"liveLessonCount":4,"liveStepCount":28,"estMinutes":32,"xpTotal":104},{"number":41,"slug":"metrics-and-error-analysis","title":"metrics, slices, and error analysis","blurb":"accuracy is a blunt instrument. learn confusion matrices, precision, recall, thresholds, slices, and failure notes so model quality has evidence.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":42,"slug":"pytorch-tensors-and-autograd","title":"pytorch tensors and autograd","blurb":"read tensor code without flinching. tensors, shapes, broadcasting, gradients, and autograd are the grammar of modern deep learning scripts.","status":"live","lessonCount":4,"stepCount":28,"liveLessonCount":4,"liveStepCount":28,"estMinutes":30,"xpTotal":104},{"number":43,"slug":"training-loops-and-optimizers","title":"training loops, backprop, optimizers, and schedulers","blurb":"the training loop is where models change. learn loss, gradients, optimizer steps, schedules, checkpoints, and the bugs ai ships there.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":44,"slug":"deep-learning-architectures","title":"cnns, transformers, and useful llm internals","blurb":"architecture literacy for builders: convolution, attention, tokens, decoding, kv cache, quantization, and what those choices do to cost and behavior.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":45,"slug":"feature-experiments-registries","title":"feature pipelines, experiment tracking, and registries","blurb":"features, runs, configs, artifacts, and registries are how ml work becomes repeatable instead of a lucky notebook.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":46,"slug":"model-serving-and-mlops","title":"model serving, ci/cd, and mlops","blurb":"serving a model means handling inputs, versions, routes, batch jobs, ci gates, rollback, and production failures deliberately.","status":"live","lessonCount":6,"stepCount":42,"liveLessonCount":6,"liveStepCount":42,"estMinutes":44,"xpTotal":156},{"number":47,"slug":"monitoring-cloud-portfolio","title":"monitoring, drift, cloud scale, and portfolio launch","blurb":"the last mile: logs, drift, alerts, retraining decisions, cloud cost, gpu constraints, architecture docs, demos, and role stories.","status":"live","lessonCount":6,"stepCount":42,"liveLessonCount":6,"liveStepCount":42,"estMinutes":44,"xpTotal":156},{"number":48,"slug":"graphic-design-studio","title":"graphic design studio — prompt, kit, and fifty on-brand assets","blurb":"how a working designer runs image models: prompts written like creative briefs, a brand bible that travels as structured data — with the design tokens that carry it onto screens — and the seeds, reference locks, and QA filters that make fifty generated assets look like one studio shipped them.","status":"live","lessonCount":3,"stepCount":27,"liveLessonCount":3,"liveStepCount":27,"estMinutes":73,"xpTotal":82},{"number":49,"slug":"campaign-studio","title":"campaign studio — one campaign, end to end, with the gate on","blurb":"brief to voice-locked copy variants to batch plan to calendar to the numbers pass. the whole campaign as data your scripts can check — and an approval gate nothing skips. built for the person who owns the send button.","status":"live","lessonCount":3,"stepCount":22,"liveLessonCount":3,"liveStepCount":22,"estMinutes":59,"xpTotal":69},{"number":50,"slug":"support-studio","title":"support studio — the reply drafter you can trust","blurb":"how a working support agent runs AI: a policy pack the drafter can't talk around, a banned-promises linter that holds every draft, golden eval sets that make each macro earn autonomy one intent at a time, and the escalation line where money, law, and anger stay human.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":61,"xpTotal":71},{"number":51,"slug":"copy-studio","title":"copy studio — voice from evidence, variants at scale, claims that clear","blurb":"a client's voice measured from their own published copy, variant batches that pass the voice gate or die with a named reason, and a claim check that blocks the ship button while any fact is unsourced. corrections-log culture, as code.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":66,"xpTotal":72},{"number":52,"slug":"analytics-studio","title":"analytics studio — messy export to defensible memo","blurb":"how a working analyst runs AI: a cleaning log that accounts for every dropped row, sample-first checks that catch the plausible-but-wrong aggregation, a memo where every number carries the address of the cell that made it, SR 11-7-style challenge against the AI's recommendation, and the rerun test — same data, same code, same memo, or it isn't done.","status":"live","lessonCount":3,"stepCount":22,"liveLessonCount":3,"liveStepCount":22,"estMinutes":61,"xpTotal":74},{"number":53,"slug":"delivery-studio","title":"delivery studio — direct the build without becoming the bottleneck","blurb":"how a working delivery lead runs AI builders: specs with acceptance checks written before the build, a review that runs the checks instead of admiring the demo, status roll-ups computed from acceptance records instead of standup vibes, the not-ready call made on evidence you can read out loud to a steering committee, and an hours-and-dollars receipt — before/after on a named function, finance's loaded rate, a named owner, the accepted thing that shipped — small enough to defend in a staff meeting.","status":"live","lessonCount":4,"stepCount":32,"liveLessonCount":4,"liveStepCount":32,"estMinutes":74,"xpTotal":96},{"number":54,"slug":"people-studio","title":"people studio — draft fast, decide human, prove everything","blurb":"how a working hr specialist runs AI: a data firewall that keeps employee records out of prompts by code instead of willpower, recruiting briefs where every claim cites a resume line, policy drafts that cannot publish without a named legal reviewer, and the adverse-action paper trail that survives a discrimination claim.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":56,"xpTotal":72},{"number":55,"slug":"ops-studio","title":"ops studio — tribal knowledge to running automation","blurb":"how a working ops lead gets the process out of one person's head and into automation that can be trusted: SOPs captured as versioned data with the invented steps hunted down, recurring checklists with an explicit auto-run vs named-approval line, exception queues instead of silent failures, a receipt against the export for every number in the weekly report, and a PII proxy that masks member identifiers before the model and restores them only for the named human who sends.","status":"live","lessonCount":4,"stepCount":32,"liveLessonCount":4,"liveStepCount":32,"estMinutes":80,"xpTotal":96},{"number":56,"slug":"legal-studio","title":"legal studio — the cite-check gate and the log that survives","blurb":"how a working lawyer runs AI: a citation-verification drill where every authority gets opened at the source (the fabricated ones always look right), clause extraction that's worthless without a pinpoint, privilege rules written as a routing table instead of remembered under deadline, and the review log you'd want in the record when opposing counsel moves for sanctions.","status":"live","lessonCount":3,"stepCount":22,"liveLessonCount":3,"liveStepCount":22,"estMinutes":60,"xpTotal":69},{"number":57,"slug":"agent-studio","title":"agent studio — ship one agent feature before you stand up","blurb":"one sitting, one agent feature, end to end: a tool schema that makes the dangerous call a validation error, a loop that meters its own spend, golden cases wired before the router works, a trace you debug from instead of vibes, and the ratchet that turns tonight's incident into tomorrow's eval. practice at production tempo.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":66,"xpTotal":73}]},"accessToc":{"chapters":[{"slug":"before-you-build","lessons":[{"slug":"from-worker-to-builder","steps":[{"id":"before-you-build/from-worker-to-builder/s01-01-intro"}]},{"slug":"upgrade-dull-artifacts-into-interactive-sites","steps":[{"id":"before-you-build/upgrade-dull-artifacts-into-interactive-sites/s01-01-intro"},{"id":"before-you-build/upgrade-dull-artifacts-into-interactive-sites/s02-02-build-brief"},{"id":"before-you-build/upgrade-dull-artifacts-into-interactive-sites/s03-03-your-artifact-brief"}]},{"slug":"why-building-matters-now","steps":[{"id":"before-you-build/why-building-matters-now/s01-01-intro"}]},{"slug":"the-builder-loop","steps":[{"id":"before-you-build/the-builder-loop/s01-01-intro"}]},{"slug":"your-first-builder-brief","steps":[{"id":"before-you-build/your-first-builder-brief/s01-01-intro"}]},{"slug":"what-the-ai-is-doing","steps":[{"id":"before-you-build/what-the-ai-is-doing/s01-01-intro"}]},{"slug":"build-your-first-reusable-tool","steps":[{"id":"before-you-build/build-your-first-reusable-tool/s01-01-intro"}]},{"slug":"from-prompt-to-project","steps":[{"id":"before-you-build/from-prompt-to-project/s01-01-intro"}]},{"slug":"why-code-appears-later","steps":[{"id":"before-you-build/why-code-appears-later/s01-01-intro"}]},{"slug":"use-it-again","steps":[{"id":"before-you-build/use-it-again/s01-01-intro"},{"id":"before-you-build/use-it-again/s02-02-which-fix-compounds"},{"id":"before-you-build/use-it-again/s03-03-save-version-two"}]}]},{"slug":"read-the-script","lessons":[{"slug":"names-and-values","steps":[{"id":"read-the-script/names-and-values/s01-01-intro"},{"id":"read-the-script/names-and-values/s02-02-spot-the-type"},{"id":"read-the-script/names-and-values/s03-03-assignment"},{"id":"read-the-script/names-and-values/s04-04-predict-the-score"},{"id":"read-the-script/names-and-values/s05-05-fill-the-name"},{"id":"read-the-script/names-and-values/s06-06-fix-the-typo"},{"id":"read-the-script/names-and-values/s07-07-write-greeting"},{"id":"read-the-script/names-and-values/s08-08-checkpoint"}]},{"slug":"functions-and-the-hallucinated-return","steps":[{"id":"read-the-script/functions-and-the-hallucinated-return/s01-01-intro"},{"id":"read-the-script/functions-and-the-hallucinated-return/s02-02-which-returns"},{"id":"read-the-script/functions-and-the-hallucinated-return/s03-03-return-vs-print"},{"id":"read-the-script/functions-and-the-hallucinated-return/s04-04-predict-the-result"},{"id":"read-the-script/functions-and-the-hallucinated-return/s05-05-fix-the-missing-return"},{"id":"read-the-script/functions-and-the-hallucinated-return/s06-06-fix-print-vs-return"},{"id":"read-the-script/functions-and-the-hallucinated-return/s07-07-write-square"},{"id":"read-the-script/functions-and-the-hallucinated-return/s08-08-checkpoint"}]},{"slug":"decisions-if-elif","steps":[{"id":"read-the-script/decisions-if-elif/s01-01-intro"},{"id":"read-the-script/decisions-if-elif/s02-02-truthiness-quiz"},{"id":"read-the-script/decisions-if-elif/s03-03-eq-vs-is"},{"id":"read-the-script/decisions-if-elif/s04-04-predict-the-empty-list"},{"id":"read-the-script/decisions-if-elif/s05-05-fix-the-zero-bug"},{"id":"read-the-script/decisions-if-elif/s06-06-fix-the-elif-order"},{"id":"read-the-script/decisions-if-elif/s07-07-checkpoint"},{"id":"read-the-script/decisions-if-elif/s08-08-going-deeper"}]}]},{"slug":"read-the-data-shapes","lessons":[{"slug":"the-shapes-ai-dumps-on-you","steps":[{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s01-01-intro"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s02-02-which-is-which"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s03-03-nested-paths"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s04-04-predict-the-deep-key"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s05-05-fill-the-index"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s06-06-fix-the-keyerror"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s07-07-fix-the-list-vs-dict"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s08-08-checkpoint"}]},{"slug":"walking-the-shape","steps":[{"id":"read-the-data-shapes/walking-the-shape/s01-01-intro"},{"id":"read-the-data-shapes/walking-the-shape/s02-02-which-loop-shape"},{"id":"read-the-data-shapes/walking-the-shape/s03-03-walking-the-shape"},{"id":"read-the-data-shapes/walking-the-shape/s04-04-predict-the-list-loop"},{"id":"read-the-data-shapes/walking-the-shape/s05-05-fill-the-loop-variable"},{"id":"read-the-data-shapes/walking-the-shape/s06-06-fix-the-dict-loop"},{"id":"read-the-data-shapes/walking-the-shape/s07-07-write-sum-loop"},{"id":"read-the-data-shapes/walking-the-shape/s08-08-checkpoint"}]},{"slug":"comprehensions-and-the-wrong-layer-bug","steps":[{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s01-01-intro"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s02-02-which-comp-shape"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s03-03-with-filter"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s04-04-predict-the-comp"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s05-05-fix-the-wrong-layer"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s06-06-write-comp"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s07-07-checkpoint"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s08-08-going-deeper"}]}]},{"slug":"read-the-failure","lessons":[{"slug":"read-bottom-up","steps":[{"id":"read-the-failure/read-bottom-up/s01-01-anatomy"},{"id":"read-the-failure/read-bottom-up/s02-02-which-error"},{"id":"read-the-failure/read-bottom-up/s03-03-the-five-fingerprints"},{"id":"read-the-failure/read-bottom-up/s04-04-predict-the-class"},{"id":"read-the-failure/read-bottom-up/s05-05-fix-nameerror"},{"id":"read-the-failure/read-bottom-up/s06-06-fix-attributeerror"},{"id":"read-the-failure/read-bottom-up/s07-07-write-defensive"},{"id":"read-the-failure/read-bottom-up/s08-08-checkpoint"}]},{"slug":"try-except-what-ai-misuses","steps":[{"id":"read-the-failure/try-except-what-ai-misuses/s01-01-intro"},{"id":"read-the-failure/try-except-what-ai-misuses/s02-02-which-block-runs"},{"id":"read-the-failure/try-except-what-ai-misuses/s03-03-naming-the-error"},{"id":"read-the-failure/try-except-what-ai-misuses/s04-04-predict-the-branch"},{"id":"read-the-failure/try-except-what-ai-misuses/s05-05-fill-the-keyword"},{"id":"read-the-failure/try-except-what-ai-misuses/s06-06-fix-the-bare-except"},{"id":"read-the-failure/try-except-what-ai-misuses/s07-07-write-safe-lookup"},{"id":"read-the-failure/try-except-what-ai-misuses/s08-08-checkpoint"}]},{"slug":"fail-loudly","steps":[{"id":"read-the-failure/fail-loudly/s01-01-intro"},{"id":"read-the-failure/fail-loudly/s02-02-which-raise-fires"},{"id":"read-the-failure/fail-loudly/s03-03-predict-the-raise"},{"id":"read-the-failure/fail-loudly/s04-04-fill-the-raise-keyword"},{"id":"read-the-failure/fail-loudly/s05-05-fix-the-silent-swallow"},{"id":"read-the-failure/fail-loudly/s06-06-write-validate-age"},{"id":"read-the-failure/fail-loudly/s07-07-checkpoint"},{"id":"read-the-failure/fail-loudly/s08-08-going-deeper"}]}]},{"slug":"read-the-pipeline","lessons":[{"slug":"files-without-fear","steps":[{"id":"read-the-pipeline/files-without-fear/s01-01-intro"},{"id":"read-the-pipeline/files-without-fear/s02-02-which-mode"},{"id":"read-the-pipeline/files-without-fear/s03-03-pathlib-and-manifests"},{"id":"read-the-pipeline/files-without-fear/s04-04-predict-the-suffix"},{"id":"read-the-pipeline/files-without-fear/s05-05-fill-the-mode"},{"id":"read-the-pipeline/files-without-fear/s06-06-fix-the-wrong-mode"},{"id":"read-the-pipeline/files-without-fear/s07-07-write-manifest-entry"},{"id":"read-the-pipeline/files-without-fear/s08-08-checkpoint"}]},{"slug":"talking-to-apis","steps":[{"id":"read-the-pipeline/talking-to-apis/s01-01-intro"},{"id":"read-the-pipeline/talking-to-apis/s02-02-which-status-family"},{"id":"read-the-pipeline/talking-to-apis/s03-03-parsing-the-json"},{"id":"read-the-pipeline/talking-to-apis/s04-04-predict-the-key"},{"id":"read-the-pipeline/talking-to-apis/s05-05-fill-the-method"},{"id":"read-the-pipeline/talking-to-apis/s06-06-fix-the-status-check"},{"id":"read-the-pipeline/talking-to-apis/s07-07-write-classify-status"},{"id":"read-the-pipeline/talking-to-apis/s08-08-checkpoint"}]},{"slug":"when-the-pipeline-breaks","steps":[{"id":"read-the-pipeline/when-the-pipeline-breaks/s01-01-intro"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s02-02-honor-the-retry-after"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s03-03-predict-the-status-handling"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s04-04-fill-the-raise-for-status"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s05-05-fix-the-missing-check"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s06-06-fix-the-broad-retry"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s07-07-checkpoint"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s08-08-going-deeper"}]}]},{"slug":"mutation-and-state","lessons":[{"slug":"why-it-breaks","steps":[{"id":"mutation-and-state/why-it-breaks/s01-01-intro"},{"id":"mutation-and-state/why-it-breaks/s02-02-which-mutates"},{"id":"mutation-and-state/why-it-breaks/s03-03-shared-reference"},{"id":"mutation-and-state/why-it-breaks/s04-04-fix-the-shared-list"},{"id":"mutation-and-state/why-it-breaks/s05-05-checkpoint"}]},{"slug":"copy-vs-reference","steps":[{"id":"mutation-and-state/copy-vs-reference/s01-01-deep-copy"},{"id":"mutation-and-state/copy-vs-reference/s02-02-predict-the-shallow"},{"id":"mutation-and-state/copy-vs-reference/s03-03-fix-the-shallow-bug"},{"id":"mutation-and-state/copy-vs-reference/s04-04-checkpoint"}]}]},{"slug":"llm-apis","lessons":[{"slug":"before-the-api-what-the-model-is-doing","steps":[{"id":"llm-apis/before-the-api-what-the-model-is-doing/s01-01-intro"}]},{"slug":"the-messages-pattern","steps":[{"id":"llm-apis/the-messages-pattern/s01-01-intro"},{"id":"llm-apis/the-messages-pattern/s02-02-which-role-is-yours"},{"id":"llm-apis/the-messages-pattern/s03-03-reading-the-response"},{"id":"llm-apis/the-messages-pattern/s04-04-predict-the-text"},{"id":"llm-apis/the-messages-pattern/s05-05-fill-the-role"},{"id":"llm-apis/the-messages-pattern/s06-06-fix-the-message-shape"},{"id":"llm-apis/the-messages-pattern/s07-07-fix-the-response-access"},{"id":"llm-apis/the-messages-pattern/s08-08-write-ask-claude"},{"id":"llm-apis/the-messages-pattern/s09-09-checkpoint"}]},{"slug":"reading-the-response","steps":[{"id":"llm-apis/reading-the-response/s01-01-intro"},{"id":"llm-apis/reading-the-response/s02-02-which-block-do-you-read"},{"id":"llm-apis/reading-the-response/s03-03-the-five-stop-reasons"},{"id":"llm-apis/reading-the-response/s04-04-predict-the-text"},{"id":"llm-apis/reading-the-response/s05-05-fill-the-text-extract"},{"id":"llm-apis/reading-the-response/s06-06-fix-the-content-zero"},{"id":"llm-apis/reading-the-response/s07-07-fix-the-stop-assumption"},{"id":"llm-apis/reading-the-response/s08-08-write-the-extractor"},{"id":"llm-apis/reading-the-response/s09-09-checkpoint"}]},{"slug":"the-model-picker","steps":[{"id":"llm-apis/the-model-picker/s01-01-why-model-choice-is-a-product-decision"},{"id":"llm-apis/the-model-picker/s02-02-haiku-territory"},{"id":"llm-apis/the-model-picker/s03-03-which-model-here"},{"id":"llm-apis/the-model-picker/s04-04-sonnet-territory"},{"id":"llm-apis/the-model-picker/s05-05-opus-and-the-thinking-budget"},{"id":"llm-apis/the-model-picker/s06-06-pick-the-model"},{"id":"llm-apis/the-model-picker/s07-07-write-the-model-routing-memo"},{"id":"llm-apis/the-model-picker/s08-08-checkpoint"}]}]},{"slug":"structured-output","lessons":[{"slug":"getting-json-back","steps":[{"id":"structured-output/getting-json-back/s01-01-intro"},{"id":"structured-output/getting-json-back/s02-02-why-validate"},{"id":"structured-output/getting-json-back/s03-03-pydantic-models"},{"id":"structured-output/getting-json-back/s04-04-predict-the-validated"},{"id":"structured-output/getting-json-back/s05-05-fill-the-field-type"},{"id":"structured-output/getting-json-back/s06-06-fix-the-required-field"},{"id":"structured-output/getting-json-back/s07-07-fix-the-type-mismatch"},{"id":"structured-output/getting-json-back/s08-08-write-parse-extraction"},{"id":"structured-output/getting-json-back/s09-09-checkpoint"}]},{"slug":"schemas-eat-prompts","steps":[{"id":"structured-output/schemas-eat-prompts/s01-01-the-trust-boundary"},{"id":"structured-output/schemas-eat-prompts/s02-02-three-shipped-breakages"},{"id":"structured-output/schemas-eat-prompts/s03-03-which-boundary-broke"},{"id":"structured-output/schemas-eat-prompts/s04-04-schema-as-prompt-replacement"},{"id":"structured-output/schemas-eat-prompts/s05-05-where-validation-belongs"},{"id":"structured-output/schemas-eat-prompts/s06-06-the-pydantic-migration-as-war-story"},{"id":"structured-output/schemas-eat-prompts/s07-07-write-the-boundary-audit"},{"id":"structured-output/schemas-eat-prompts/s08-08-checkpoint"}]},{"slug":"dataset-schema-validation","steps":[{"id":"structured-output/dataset-schema-validation/s01-01-validate-at-the-boundary"},{"id":"structured-output/dataset-schema-validation/s02-02-the-silent-drop"},{"id":"structured-output/dataset-schema-validation/s03-03-predict-the-missing-fields"},{"id":"structured-output/dataset-schema-validation/s04-04-fill-the-membership-test"},{"id":"structured-output/dataset-schema-validation/s05-05-fix-the-range-check"},{"id":"structured-output/dataset-schema-validation/s06-06-write-validate-batch"},{"id":"structured-output/dataset-schema-validation/s07-07-claims-with-receipts"},{"id":"structured-output/dataset-schema-validation/s08-08-checkpoint"}]}]},{"slug":"mcp","lessons":[{"slug":"what-mcp-is","steps":[{"id":"mcp/what-mcp-is/s01-01-intro"},{"id":"mcp/what-mcp-is/s02-02-which-side-is-the-server"},{"id":"mcp/what-mcp-is/s03-03-the-tool-list"},{"id":"mcp/what-mcp-is/s04-04-predict-the-tool-result"},{"id":"mcp/what-mcp-is/s05-05-fill-the-tool-name"},{"id":"mcp/what-mcp-is/s06-06-fix-the-missing-argument"},{"id":"mcp/what-mcp-is/s07-07-fix-the-result-shape"},{"id":"mcp/what-mcp-is/s08-08-write-call-tool"},{"id":"mcp/what-mcp-is/s09-09-checkpoint"}]},{"slug":"writing-a-tiny-mcp-server","steps":[{"id":"mcp/writing-a-tiny-mcp-server/s01-01-intro"},{"id":"mcp/writing-a-tiny-mcp-server/s02-02-which-message-is-call-tool"},{"id":"mcp/writing-a-tiny-mcp-server/s03-03-the-tool-registry"},{"id":"mcp/writing-a-tiny-mcp-server/s04-04-predict-the-list-tools-output"},{"id":"mcp/writing-a-tiny-mcp-server/s05-05-fill-the-input-schema"},{"id":"mcp/writing-a-tiny-mcp-server/s06-06-fix-the-missing-error-response"},{"id":"mcp/writing-a-tiny-mcp-server/s07-07-fix-the-validation-gap"},{"id":"mcp/writing-a-tiny-mcp-server/s08-08-write-the-call-tool-dispatcher"},{"id":"mcp/writing-a-tiny-mcp-server/s09-09-checkpoint"}]},{"slug":"why-mcp-won","steps":[{"id":"mcp/why-mcp-won/s01-01-the-graveyard-of-tool-protocols"},{"id":"mcp/why-mcp-won/s02-02-what-openai-plugins-got-wrong"},{"id":"mcp/why-mcp-won/s03-03-which-failure-mode"},{"id":"mcp/why-mcp-won/s04-04-the-mcp-bet"},{"id":"mcp/why-mcp-won/s05-05-the-network-effect"},{"id":"mcp/why-mcp-won/s06-06-spot-the-mcp-wedge"},{"id":"mcp/why-mcp-won/s07-07-write-the-protocol-postmortem"},{"id":"mcp/why-mcp-won/s08-08-checkpoint"}]}]},{"slug":"agent-loops","lessons":[{"slug":"the-loop","steps":[{"id":"agent-loops/the-loop/s01-01-intro"},{"id":"agent-loops/the-loop/s02-02-which-stop-reason"},{"id":"agent-loops/the-loop/s03-03-tool-result-shape"},{"id":"agent-loops/the-loop/s04-04-predict-the-next-call"},{"id":"agent-loops/the-loop/s05-05-fill-the-stop-condition"},{"id":"agent-loops/the-loop/s06-06-fix-the-tool-id"},{"id":"agent-loops/the-loop/s07-07-fix-the-loop-exit"},{"id":"agent-loops/the-loop/s08-08-write-the-loop"},{"id":"agent-loops/the-loop/s09-09-checkpoint"}]},{"slug":"multi-step-tools","steps":[{"id":"agent-loops/multi-step-tools/s01-01-intro"},{"id":"agent-loops/multi-step-tools/s02-02-which-tool-runs"},{"id":"agent-loops/multi-step-tools/s03-03-tool-registry"},{"id":"agent-loops/multi-step-tools/s04-04-predict-the-chain"},{"id":"agent-loops/multi-step-tools/s05-05-fill-the-dispatch"},{"id":"agent-loops/multi-step-tools/s06-06-fix-the-missing-tool"},{"id":"agent-loops/multi-step-tools/s07-07-fix-the-parallel-ids"},{"id":"agent-loops/multi-step-tools/s08-08-write-multi-tool-agent"},{"id":"agent-loops/multi-step-tools/s09-09-checkpoint"}]},{"slug":"routing","steps":[{"id":"agent-loops/routing/s01-01-intro"},{"id":"agent-loops/routing/s02-02-when-to-route"},{"id":"agent-loops/routing/s03-03-classifier-output"},{"id":"agent-loops/routing/s04-04-predict-the-route"},{"id":"agent-loops/routing/s05-05-fill-the-dispatch"},{"id":"agent-loops/routing/s06-06-fix-the-fallback"},{"id":"agent-loops/routing/s07-07-fix-the-loose-classifier"},{"id":"agent-loops/routing/s08-08-write-the-router"},{"id":"agent-loops/routing/s09-09-checkpoint"}]},{"slug":"evaluator-optimizer","steps":[{"id":"agent-loops/evaluator-optimizer/s01-01-intro"},{"id":"agent-loops/evaluator-optimizer/s02-02-when-it-earns-its-keep"},{"id":"agent-loops/evaluator-optimizer/s03-03-judge-output-shape"},{"id":"agent-loops/evaluator-optimizer/s04-04-predict-the-iterations"},{"id":"agent-loops/evaluator-optimizer/s05-05-fill-the-exit"},{"id":"agent-loops/evaluator-optimizer/s06-06-fix-the-judge-parsing"},{"id":"agent-loops/evaluator-optimizer/s07-07-fix-the-missing-cap"},{"id":"agent-loops/evaluator-optimizer/s08-08-write-the-evaluator-optimizer"},{"id":"agent-loops/evaluator-optimizer/s09-09-checkpoint"}]},{"slug":"why-every-framework-is-thirty-lines","steps":[{"id":"agent-loops/why-every-framework-is-thirty-lines/s01-01-the-frameworks-converged"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s02-02-langgraph-walkthrough"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s03-03-the-work-is-a-graph"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s04-04-which-framework-fits"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s05-05-vercel-ai-sdk-walkthrough"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s06-06-the-five-canonical-patterns"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s07-07-when-to-stay-framework-free"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s08-08-write-the-tradeoff-memo"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s09-09-checkpoint"}]},{"slug":"the-industry-map","steps":[{"id":"agent-loops/the-industry-map/s01-01-the-five-layers"},{"id":"agent-loops/the-industry-map/s02-02-which-layer-failed"},{"id":"agent-loops/the-industry-map/s03-03-the-shape-of-the-stack"}]}]},{"slug":"git-and-github","lessons":[{"slug":"the-three-states","steps":[{"id":"git-and-github/the-three-states/s01-01-intro"},{"id":"git-and-github/the-three-states/s02-02-which-command-stages"},{"id":"git-and-github/the-three-states/s03-03-status-and-diff"},{"id":"git-and-github/the-three-states/s04-04-predict-the-status"},{"id":"git-and-github/the-three-states/s05-05-fill-the-command"},{"id":"git-and-github/the-three-states/s06-06-fix-the-staged-files"},{"id":"git-and-github/the-three-states/s07-07-fix-the-pr-command"},{"id":"git-and-github/the-three-states/s08-08-write-status-summary"},{"id":"git-and-github/the-three-states/s09-09-checkpoint"}]},{"slug":"three-git-disasters-ai-shipped","steps":[{"id":"git-and-github/three-git-disasters-ai-shipped/s01-01-why-postmortems-teach-better"},{"id":"git-and-github/three-git-disasters-ai-shipped/s02-02-uber-2016-credentials-in-repo"},{"id":"git-and-github/three-git-disasters-ai-shipped/s03-03-which-control-would-have-caught-it"},{"id":"git-and-github/three-git-disasters-ai-shipped/s04-04-the-samsung-paste"},{"id":"git-and-github/three-git-disasters-ai-shipped/s05-05-the-cursor-git-add-dot-default"},{"id":"git-and-github/three-git-disasters-ai-shipped/s06-06-spot-the-failure-mode"},{"id":"git-and-github/three-git-disasters-ai-shipped/s07-07-write-your-pre-commit-checklist"},{"id":"git-and-github/three-git-disasters-ai-shipped/s08-08-checkpoint"}]},{"slug":"github-actions-for-model-checks","steps":[{"id":"git-and-github/github-actions-for-model-checks/s01-01-intro"},{"id":"git-and-github/github-actions-for-model-checks/s02-02-green-but-wrong"},{"id":"git-and-github/github-actions-for-model-checks/s03-03-predict-the-run"},{"id":"git-and-github/github-actions-for-model-checks/s04-04-fill-the-trigger"},{"id":"git-and-github/github-actions-for-model-checks/s05-05-fix-the-eval-gate"},{"id":"git-and-github/github-actions-for-model-checks/s06-06-write-run-job"},{"id":"git-and-github/github-actions-for-model-checks/s07-07-checkpoint"}]}]},{"slug":"secrets-and-env","lessons":[{"slug":"keeping-keys-safe","steps":[{"id":"secrets-and-env/keeping-keys-safe/s01-01-intro"},{"id":"secrets-and-env/keeping-keys-safe/s02-02-which-line-leaks"},{"id":"secrets-and-env/keeping-keys-safe/s03-03-loading-env-vars"},{"id":"secrets-and-env/keeping-keys-safe/s04-04-predict-the-key"},{"id":"secrets-and-env/keeping-keys-safe/s05-05-fill-the-getenv"},{"id":"secrets-and-env/keeping-keys-safe/s06-06-fix-the-hardcoded-key"},{"id":"secrets-and-env/keeping-keys-safe/s07-07-fix-the-missing-default"},{"id":"secrets-and-env/keeping-keys-safe/s08-08-write-load-config"},{"id":"secrets-and-env/keeping-keys-safe/s09-09-checkpoint"}]},{"slug":"ml-service-secrets","steps":[{"id":"secrets-and-env/ml-service-secrets/s01-01-intro"},{"id":"secrets-and-env/ml-service-secrets/s02-02-which-log-line-leaks"},{"id":"secrets-and-env/ml-service-secrets/s03-03-predict-the-scan"},{"id":"secrets-and-env/ml-service-secrets/s04-04-fill-the-lookup"},{"id":"secrets-and-env/ml-service-secrets/s05-05-fix-the-logged-key"},{"id":"secrets-and-env/ml-service-secrets/s06-06-write-scan-logs"},{"id":"secrets-and-env/ml-service-secrets/s07-07-checkpoint"}]}]},{"slug":"prompting","lessons":[{"slug":"builder-briefs-before-prompts","steps":[{"id":"prompting/builder-briefs-before-prompts/s01-01-intro"}]},{"slug":"the-prompt-craft","steps":[{"id":"prompting/the-prompt-craft/s01-01-intro"},{"id":"prompting/the-prompt-craft/s02-02-which-prompt-is-better"},{"id":"prompting/the-prompt-craft/s03-03-context-rot"},{"id":"prompting/the-prompt-craft/s04-04-predict-the-output"},{"id":"prompting/the-prompt-craft/s05-05-fill-the-constraint"},{"id":"prompting/the-prompt-craft/s06-06-fix-the-vague-prompt"},{"id":"prompting/the-prompt-craft/s07-07-fix-the-bloated-context"},{"id":"prompting/the-prompt-craft/s08-08-write-the-prompt"},{"id":"prompting/the-prompt-craft/s09-09-checkpoint"}]},{"slug":"few-shot-and-reasoning","steps":[{"id":"prompting/few-shot-and-reasoning/s01-01-intro"},{"id":"prompting/few-shot-and-reasoning/s02-02-when-few-shot-helps"},{"id":"prompting/few-shot-and-reasoning/s03-03-the-cot-trap"},{"id":"prompting/few-shot-and-reasoning/s04-04-predict-the-format-lock"},{"id":"prompting/few-shot-and-reasoning/s05-05-fill-the-example-format"},{"id":"prompting/few-shot-and-reasoning/s06-06-fix-the-cot-on-reasoning"},{"id":"prompting/few-shot-and-reasoning/s07-07-fix-the-format-mismatch"},{"id":"prompting/few-shot-and-reasoning/s08-08-write-the-prompt-builder"},{"id":"prompting/few-shot-and-reasoning/s09-09-checkpoint"}]},{"slug":"agent-config-files","steps":[{"id":"prompting/agent-config-files/s01-01-intro"},{"id":"prompting/agent-config-files/s02-02-which-belongs-in-the-file"},{"id":"prompting/agent-config-files/s03-03-anatomy-of-a-good-file"},{"id":"prompting/agent-config-files/s04-04-predict-which-rule-fires"},{"id":"prompting/agent-config-files/s05-05-fill-the-rule-block"},{"id":"prompting/agent-config-files/s06-06-fix-the-too-vague"},{"id":"prompting/agent-config-files/s07-07-fix-the-stale-pointer"},{"id":"prompting/agent-config-files/s08-08-write-a-claude-md"},{"id":"prompting/agent-config-files/s09-09-checkpoint"}]},{"slug":"what-aged-in-prompting","steps":[{"id":"prompting/what-aged-in-prompting/s01-01-prompting-as-a-moving-target"},{"id":"prompting/what-aged-in-prompting/s02-02-cot-then-vs-now"},{"id":"prompting/what-aged-in-prompting/s03-03-which-still-works"},{"id":"prompting/what-aged-in-prompting/s04-04-the-system-prompt-arms-race"},{"id":"prompting/what-aged-in-prompting/s05-05-spot-the-stale-technique"},{"id":"prompting/what-aged-in-prompting/s06-06-the-prompt-engineer-debate"},{"id":"prompting/what-aged-in-prompting/s07-07-write-the-cheatsheet-update"},{"id":"prompting/what-aged-in-prompting/s08-08-checkpoint"}]}]},{"slug":"agent-traces","lessons":[{"slug":"reading-the-trace","steps":[{"id":"agent-traces/reading-the-trace/s01-01-intro"},{"id":"agent-traces/reading-the-trace/s02-02-which-turn-failed"},{"id":"agent-traces/reading-the-trace/s03-03-the-stop-reasons"},{"id":"agent-traces/reading-the-trace/s04-04-predict-the-stop"},{"id":"agent-traces/reading-the-trace/s05-05-fill-the-tool-name"},{"id":"agent-traces/reading-the-trace/s06-06-fix-the-loop"},{"id":"agent-traces/reading-the-trace/s07-07-fix-the-arg-shape"},{"id":"agent-traces/reading-the-trace/s08-08-write-the-summarizer"},{"id":"agent-traces/reading-the-trace/s09-09-checkpoint"}]},{"slug":"trace-driven-debugging","steps":[{"id":"agent-traces/trace-driven-debugging/s01-01-intro"},{"id":"agent-traces/trace-driven-debugging/s02-02-which-pattern-is-this"},{"id":"agent-traces/trace-driven-debugging/s03-03-the-loop-signature"},{"id":"agent-traces/trace-driven-debugging/s04-04-predict-the-call-count"},{"id":"agent-traces/trace-driven-debugging/s05-05-fill-the-detector"},{"id":"agent-traces/trace-driven-debugging/s06-06-fix-the-tool-routing"},{"id":"agent-traces/trace-driven-debugging/s07-07-fix-the-prompt-bloat"},{"id":"agent-traces/trace-driven-debugging/s08-08-write-the-trace-summary"},{"id":"agent-traces/trace-driven-debugging/s09-09-checkpoint"}]},{"slug":"prediction-ids-and-inference-logs","steps":[{"id":"agent-traces/prediction-ids-and-inference-logs/s01-01-intro"},{"id":"agent-traces/prediction-ids-and-inference-logs/s02-02-pick-the-log-line"},{"id":"agent-traces/prediction-ids-and-inference-logs/s03-03-predict-the-ids"},{"id":"agent-traces/prediction-ids-and-inference-logs/s04-04-fill-the-join-key"},{"id":"agent-traces/prediction-ids-and-inference-logs/s05-05-fix-the-join"},{"id":"agent-traces/prediction-ids-and-inference-logs/s06-06-write-the-logger"},{"id":"agent-traces/prediction-ids-and-inference-logs/s07-07-checkpoint"}]}]},{"slug":"evals","lessons":[{"slug":"check-before-you-trust","steps":[{"id":"evals/check-before-you-trust/s01-01-intro"}]},{"slug":"writing-evals","steps":[{"id":"evals/writing-evals/s01-01-intro"},{"id":"evals/writing-evals/s02-02-which-eval-passes"},{"id":"evals/writing-evals/s03-03-eval-patterns"},{"id":"evals/writing-evals/s04-04-predict-the-pass-rate"},{"id":"evals/writing-evals/s05-05-fill-the-assertion"},{"id":"evals/writing-evals/s06-06-fix-the-flaky-eval"},{"id":"evals/writing-evals/s07-07-fix-the-overstrict-eval"},{"id":"evals/writing-evals/s08-08-write-the-suite"},{"id":"evals/writing-evals/s09-09-checkpoint"}]},{"slug":"llm-as-judge","steps":[{"id":"evals/llm-as-judge/s01-01-intro"},{"id":"evals/llm-as-judge/s02-02-pairwise-or-rubric"},{"id":"evals/llm-as-judge/s03-03-the-four-biases"},{"id":"evals/llm-as-judge/s04-04-predict-the-bias"},{"id":"evals/llm-as-judge/s05-05-fill-the-rubric-call"},{"id":"evals/llm-as-judge/s06-06-fix-the-likert-trap"},{"id":"evals/llm-as-judge/s07-07-fix-the-position-bias"},{"id":"evals/llm-as-judge/s08-08-write-the-rubric-judge"},{"id":"evals/llm-as-judge/s09-09-checkpoint"}]},{"slug":"the-rise-of-evals-as-a-discipline","steps":[{"id":"evals/the-rise-of-evals-as-a-discipline/s01-01-the-prompt-engineering-era"},{"id":"evals/the-rise-of-evals-as-a-discipline/s02-02-the-eval-turn"},{"id":"evals/the-rise-of-evals-as-a-discipline/s03-03-which-company-survives-a-model-swap"},{"id":"evals/the-rise-of-evals-as-a-discipline/s04-04-case-studies"},{"id":"evals/the-rise-of-evals-as-a-discipline/s05-05-eval-first-vs-prompt-first"},{"id":"evals/the-rise-of-evals-as-a-discipline/s06-06-why-the-eval-engineer-eats-the-prompt-engineer"},{"id":"evals/the-rise-of-evals-as-a-discipline/s07-07-write-the-eval-readiness-audit"},{"id":"evals/the-rise-of-evals-as-a-discipline/s08-08-checkpoint"}]},{"slug":"model-validation-gates","steps":[{"id":"evals/model-validation-gates/s01-01-intro"},{"id":"evals/model-validation-gates/s02-02-spot-the-theater"},{"id":"evals/model-validation-gates/s03-03-predict-the-decision"},{"id":"evals/model-validation-gates/s04-04-fill-the-regression-check"},{"id":"evals/model-validation-gates/s05-05-fix-the-theater-gate"},{"id":"evals/model-validation-gates/s06-06-write-the-gate-runner"},{"id":"evals/model-validation-gates/s07-07-checkpoint"}]}]},{"slug":"context-and-retrieval","lessons":[{"slug":"chunking-that-respects-structure","steps":[{"id":"context-and-retrieval/chunking-that-respects-structure/s01-01-intro"},{"id":"context-and-retrieval/chunking-that-respects-structure/s02-02-which-content-shape-breaks"},{"id":"context-and-retrieval/chunking-that-respects-structure/s03-03-the-recursive-splitter"},{"id":"context-and-retrieval/chunking-that-respects-structure/s04-04-predict-the-shred"},{"id":"context-and-retrieval/chunking-that-respects-structure/s05-05-fill-the-separator-list"},{"id":"context-and-retrieval/chunking-that-respects-structure/s06-06-fix-the-mid-sentence-cut"},{"id":"context-and-retrieval/chunking-that-respects-structure/s07-07-fix-the-missing-overlap"},{"id":"context-and-retrieval/chunking-that-respects-structure/s08-08-write-the-recursive-splitter"},{"id":"context-and-retrieval/chunking-that-respects-structure/s09-09-checkpoint"}]},{"slug":"embedding-that-fits-the-budget","steps":[{"id":"context-and-retrieval/embedding-that-fits-the-budget/s01-01-intro"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s02-02-which-model-fits"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s03-03-the-vector-space"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s04-04-predict-the-closest"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s05-05-fill-the-cosine-formula"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s06-06-fix-the-wrong-embedding-model"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s07-07-fix-the-missing-normalization"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s08-08-write-the-similarity-ranker"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s09-09-checkpoint"}]},{"slug":"retrieval-that-finds-the-right-thing","steps":[{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s01-01-intro"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s02-02-which-result-is-good"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s03-03-the-distance-threshold"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s04-04-predict-the-top-k"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s05-05-fill-the-cutoff"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s06-06-fix-the-no-threshold"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s07-07-fix-the-dup-results"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s08-08-write-the-ranker"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s09-09-checkpoint"}]},{"slug":"rag-vs-long-context-vs-fine-tune","steps":[{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s01-01-the-three-way-fork"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s02-02-rag-when-you-need-freshness"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s03-03-which-fork-for-this-scenario"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s04-04-long-context-when-the-corpus-fits"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s05-05-fine-tune-when-style-or-format-is-the-product"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s06-06-which-fork-dies-first"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s07-07-graph-vector-or-both"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s08-08-the-rubric"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s09-09-write-the-fork-decision"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s10-10-checkpoint"}]},{"slug":"retrieval-metrics-and-vector-db-shape","steps":[{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s01-01-the-eval-set"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s02-02-filter-or-search-first"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s03-03-predict-recall-precision"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s04-04-fill-the-reciprocal"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s05-05-fix-the-late-filter"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s06-06-write-recall-at-k"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s07-07-checkpoint"}]}]},{"slug":"production-tradeoffs","lessons":[{"slug":"prompt-caching-correctly","steps":[{"id":"production-tradeoffs/prompt-caching-correctly/s01-01-intro"},{"id":"production-tradeoffs/prompt-caching-correctly/s02-02-which-piece-caches"},{"id":"production-tradeoffs/prompt-caching-correctly/s03-03-the-cache-breakpoints"},{"id":"production-tradeoffs/prompt-caching-correctly/s04-04-predict-the-savings"},{"id":"production-tradeoffs/prompt-caching-correctly/s05-05-fill-the-cache-control"},{"id":"production-tradeoffs/prompt-caching-correctly/s06-06-fix-the-cache-buster"},{"id":"production-tradeoffs/prompt-caching-correctly/s07-07-fix-the-ttl-default"},{"id":"production-tradeoffs/prompt-caching-correctly/s08-08-write-the-cost-estimator"},{"id":"production-tradeoffs/prompt-caching-correctly/s09-09-checkpoint"}]},{"slug":"read-the-token-bill","steps":[{"id":"production-tradeoffs/read-the-token-bill/s01-01-intro"},{"id":"production-tradeoffs/read-the-token-bill/s02-02-which-prompt-costs-more"},{"id":"production-tradeoffs/read-the-token-bill/s03-03-the-shape-of-a-bill"},{"id":"production-tradeoffs/read-the-token-bill/s04-04-predict-the-monthly-spend"},{"id":"production-tradeoffs/read-the-token-bill/s05-05-fill-the-cost-formula"},{"id":"production-tradeoffs/read-the-token-bill/s06-06-fix-the-pricing-ratio-error"},{"id":"production-tradeoffs/read-the-token-bill/s07-07-fix-the-units-error"},{"id":"production-tradeoffs/read-the-token-bill/s08-08-write-the-cost-estimator"},{"id":"production-tradeoffs/read-the-token-bill/s09-09-checkpoint"}]},{"slug":"the-model-price-war","steps":[{"id":"production-tradeoffs/the-model-price-war/s01-01-the-price-curve"},{"id":"production-tradeoffs/the-model-price-war/s02-02-three-bets-and-how-they-aged"},{"id":"production-tradeoffs/the-model-price-war/s03-03-which-startup-survived"},{"id":"production-tradeoffs/the-model-price-war/s04-04-the-incumbent-trap"},{"id":"production-tradeoffs/the-model-price-war/s05-05-features-now-viable"},{"id":"production-tradeoffs/the-model-price-war/s06-06-pricing-for-a-war-you-cant-predict"},{"id":"production-tradeoffs/the-model-price-war/s07-07-write-the-cost-model"},{"id":"production-tradeoffs/the-model-price-war/s08-08-checkpoint"}]},{"slug":"batch-realtime-costs","steps":[{"id":"production-tradeoffs/batch-realtime-costs/s01-01-who-is-waiting"},{"id":"production-tradeoffs/batch-realtime-costs/s02-02-which-lane"},{"id":"production-tradeoffs/batch-realtime-costs/s03-03-predict-the-bill"},{"id":"production-tradeoffs/batch-realtime-costs/s04-04-fill-the-miss-rate"},{"id":"production-tradeoffs/batch-realtime-costs/s05-05-fix-the-free-discount"},{"id":"production-tradeoffs/batch-realtime-costs/s06-06-write-monthly-bill"},{"id":"production-tradeoffs/batch-realtime-costs/s07-07-checkpoint"}]}]},{"slug":"debugging-output","lessons":[{"slug":"read-the-trace-not-the-chat","steps":[{"id":"debugging-output/read-the-trace-not-the-chat/s01-01-intro"},{"id":"debugging-output/read-the-trace-not-the-chat/s02-02-where-to-look-first"},{"id":"debugging-output/read-the-trace-not-the-chat/s03-03-the-four-failure-classes"},{"id":"debugging-output/read-the-trace-not-the-chat/s04-04-predict-the-bad-turn"},{"id":"debugging-output/read-the-trace-not-the-chat/s05-05-fill-the-find-bad-turn"},{"id":"debugging-output/read-the-trace-not-the-chat/s06-06-fix-the-fix-the-symptom"},{"id":"debugging-output/read-the-trace-not-the-chat/s07-07-fix-the-blame-the-model"},{"id":"debugging-output/read-the-trace-not-the-chat/s08-08-write-the-trace-summary"},{"id":"debugging-output/read-the-trace-not-the-chat/s09-09-checkpoint"}]},{"slug":"the-four-breakage-classes","steps":[{"id":"debugging-output/the-four-breakage-classes/s01-01-intro"},{"id":"debugging-output/the-four-breakage-classes/s02-02-classify-the-refund-bug"},{"id":"debugging-output/the-four-breakage-classes/s03-03-the-trace-anatomy"},{"id":"debugging-output/the-four-breakage-classes/s04-04-predict-the-class"},{"id":"debugging-output/the-four-breakage-classes/s05-05-fill-the-class-name"},{"id":"debugging-output/the-four-breakage-classes/s06-06-fix-the-misclassification"},{"id":"debugging-output/the-four-breakage-classes/s07-07-fix-the-parse-mangle"},{"id":"debugging-output/the-four-breakage-classes/s08-08-write-the-classifier"},{"id":"debugging-output/the-four-breakage-classes/s09-09-checkpoint"}]},{"slug":"five-real-postmortems","steps":[{"id":"debugging-output/five-real-postmortems/s01-01-the-postmortem-template"},{"id":"debugging-output/five-real-postmortems/s02-02-air-canada-refund-lawsuit"},{"id":"debugging-output/five-real-postmortems/s03-03-dpd-and-nyc-mycity"},{"id":"debugging-output/five-real-postmortems/s04-04-classify-the-five"},{"id":"debugging-output/five-real-postmortems/s05-05-recruiter-json-mangle"},{"id":"debugging-output/five-real-postmortems/s06-06-glean-style-retrieval-miss"},{"id":"debugging-output/five-real-postmortems/s07-07-which-fix-catches-the-most"},{"id":"debugging-output/five-real-postmortems/s08-08-write-the-postmortem"},{"id":"debugging-output/five-real-postmortems/s09-09-checkpoint"}]},{"slug":"prediction-failure-triage","steps":[{"id":"debugging-output/prediction-failure-triage/s01-01-reproduce-classify-route"},{"id":"debugging-output/prediction-failure-triage/s02-02-first-move"},{"id":"debugging-output/prediction-failure-triage/s03-03-predict-the-route"},{"id":"debugging-output/prediction-failure-triage/s04-04-fill-the-owners"},{"id":"debugging-output/prediction-failure-triage/s05-05-fix-the-paraphrase-repro"},{"id":"debugging-output/prediction-failure-triage/s06-06-write-the-router"},{"id":"debugging-output/prediction-failure-triage/s07-07-checkpoint"}]}]},{"slug":"capstone","lessons":[{"slug":"pick-a-project-that-ships","steps":[{"id":"capstone/pick-a-project-that-ships/s01-01-the-builders-graveyard"},{"id":"capstone/pick-a-project-that-ships/s02-02-which-project-survives-contact-with-reality"},{"id":"capstone/pick-a-project-that-ships/s03-03-the-three-shapes-that-ship"},{"id":"capstone/pick-a-project-that-ships/s04-04-the-wedge-checklist"},{"id":"capstone/pick-a-project-that-ships/s05-05-spot-the-missing-wedge"},{"id":"capstone/pick-a-project-that-ships/s06-06-scope-down-the-capstone"},{"id":"capstone/pick-a-project-that-ships/s07-07-checkpoint"}]},{"slug":"build-a-cli-agent","steps":[{"id":"capstone/build-a-cli-agent/s01-01-intro"},{"id":"capstone/build-a-cli-agent/s02-02-which-piece-is-missing"},{"id":"capstone/build-a-cli-agent/s03-03-the-tool-loop"},{"id":"capstone/build-a-cli-agent/s04-04-predict-the-final-answer"},{"id":"capstone/build-a-cli-agent/s05-05-fill-the-stop-check"},{"id":"capstone/build-a-cli-agent/s06-06-fix-the-tool-dispatch"},{"id":"capstone/build-a-cli-agent/s07-07-fix-the-message-append"},{"id":"capstone/build-a-cli-agent/s08-08-the-logging-pattern"},{"id":"capstone/build-a-cli-agent/s09-09-write-the-tool-runner"},{"id":"capstone/build-a-cli-agent/s10-10-fix-the-loop-exit"},{"id":"capstone/build-a-cli-agent/s11-11-write-the-agent"},{"id":"capstone/build-a-cli-agent/s12-12-checkpoint"}]},{"slug":"wire-the-real-model","steps":[{"id":"capstone/wire-the-real-model/s01-01-intro"},{"id":"capstone/wire-the-real-model/s02-02-where-the-key-lives"},{"id":"capstone/wire-the-real-model/s03-03-the-sdk-response-shape"},{"id":"capstone/wire-the-real-model/s04-04-predict-the-block-type"},{"id":"capstone/wire-the-real-model/s05-05-fill-the-key-load"},{"id":"capstone/wire-the-real-model/s06-06-fix-the-hardcoded-key"},{"id":"capstone/wire-the-real-model/s07-07-fix-the-block-access"},{"id":"capstone/wire-the-real-model/s08-08-write-the-shape-adapter"},{"id":"capstone/wire-the-real-model/s09-09-checkpoint"}]},{"slug":"validate-tool-inputs","steps":[{"id":"capstone/validate-tool-inputs/s01-01-intro"},{"id":"capstone/validate-tool-inputs/s02-02-which-input-needs-validation"},{"id":"capstone/validate-tool-inputs/s03-03-the-validator-surface"},{"id":"capstone/validate-tool-inputs/s04-04-predict-the-error"},{"id":"capstone/validate-tool-inputs/s05-05-fill-the-validate-call"},{"id":"capstone/validate-tool-inputs/s06-06-fix-the-untyped-tool"},{"id":"capstone/validate-tool-inputs/s07-07-fix-the-error-leak"},{"id":"capstone/validate-tool-inputs/s08-08-write-the-validator"},{"id":"capstone/validate-tool-inputs/s09-09-checkpoint"}]},{"slug":"add-evals-and-traces","steps":[{"id":"capstone/add-evals-and-traces/s01-01-intro"},{"id":"capstone/add-evals-and-traces/s02-02-which-eval-fits"},{"id":"capstone/add-evals-and-traces/s03-03-the-trace-shape"},{"id":"capstone/add-evals-and-traces/s04-04-predict-the-pass-rate"},{"id":"capstone/add-evals-and-traces/s05-05-fill-the-trace-append"},{"id":"capstone/add-evals-and-traces/s06-06-fix-the-trace-mutation"},{"id":"capstone/add-evals-and-traces/s07-07-fix-the-tautological-eval"},{"id":"capstone/add-evals-and-traces/s08-08-write-the-eval-runner"},{"id":"capstone/add-evals-and-traces/s09-09-checkpoint"}]},{"slug":"wire-an-mcp-tool","steps":[{"id":"capstone/wire-an-mcp-tool/s01-01-intro"},{"id":"capstone/wire-an-mcp-tool/s02-02-when-to-use-mcp"},{"id":"capstone/wire-an-mcp-tool/s03-03-the-mcp-protocol"},{"id":"capstone/wire-an-mcp-tool/s04-04-predict-the-tool-list"},{"id":"capstone/wire-an-mcp-tool/s05-05-fill-the-tools-call"},{"id":"capstone/wire-an-mcp-tool/s06-06-fix-the-result-shape"},{"id":"capstone/wire-an-mcp-tool/s07-07-fix-the-iserror-flag"},{"id":"capstone/wire-an-mcp-tool/s08-08-write-the-mcp-bridge"},{"id":"capstone/wire-an-mcp-tool/s09-09-checkpoint"}]},{"slug":"portfolio-ml-system-option","steps":[{"id":"capstone/portfolio-ml-system-option/s01-01-the-other-capstone"},{"id":"capstone/portfolio-ml-system-option/s02-02-what-reviewers-open"},{"id":"capstone/portfolio-ml-system-option/s03-03-predict-the-gate"},{"id":"capstone/portfolio-ml-system-option/s04-04-fill-the-gate"},{"id":"capstone/portfolio-ml-system-option/s05-05-fix-the-train-accuracy-gate"},{"id":"capstone/portfolio-ml-system-option/s06-06-write-the-scope-check"},{"id":"capstone/portfolio-ml-system-option/s07-07-checkpoint"}]}]},{"slug":"agent-harnesses","lessons":[{"slug":"project-workspaces-not-product-names","steps":[{"id":"agent-harnesses/project-workspaces-not-product-names/s01-01-intro"}]},{"slug":"what-a-harness-is","steps":[{"id":"agent-harnesses/what-a-harness-is/s01-01-intro"},{"id":"agent-harnesses/what-a-harness-is/s02-02-which-is-a-harness"},{"id":"agent-harnesses/what-a-harness-is/s03-03-the-four-layers"},{"id":"agent-harnesses/what-a-harness-is/s04-04-predict-the-layer"},{"id":"agent-harnesses/what-a-harness-is/s05-05-fill-the-dispatch"},{"id":"agent-harnesses/what-a-harness-is/s06-06-fix-the-no-retry"},{"id":"agent-harnesses/what-a-harness-is/s07-07-fix-the-hardcoded-registry"},{"id":"agent-harnesses/what-a-harness-is/s08-08-write-mini-harness"},{"id":"agent-harnesses/what-a-harness-is/s09-09-checkpoint"}]},{"slug":"architecting-an-ai-native-workflow","steps":[{"id":"agent-harnesses/architecting-an-ai-native-workflow/s01-01-intro"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s02-02-which-workflow-fits"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s03-03-mapping-the-workflow"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s04-04-predict-the-readiness-score"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s05-05-fill-the-required-fields"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s06-06-fix-the-implicit-policy"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s07-07-fix-the-wrong-metric"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s08-08-write-the-workflow-validator"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s09-09-checkpoint"}]},{"slug":"five-industries-walked-through","steps":[{"id":"agent-harnesses/five-industries-walked-through/s01-01-the-case-study-method"},{"id":"agent-harnesses/five-industries-walked-through/s02-02-home-services"},{"id":"agent-harnesses/five-industries-walked-through/s03-03-which-task-belongs-to-the-agent"},{"id":"agent-harnesses/five-industries-walked-through/s04-04-insurance-brokerage"},{"id":"agent-harnesses/five-industries-walked-through/s05-05-recruiting-firm"},{"id":"agent-harnesses/five-industries-walked-through/s06-06-which-industry-wedges-fastest"},{"id":"agent-harnesses/five-industries-walked-through/s07-07-the-incumbent-trap"},{"id":"agent-harnesses/five-industries-walked-through/s08-08-write-the-process-debt-audit"},{"id":"agent-harnesses/five-industries-walked-through/s09-09-checkpoint"}]},{"slug":"incident-runbook-from-the-repo","steps":[{"id":"agent-harnesses/incident-runbook-from-the-repo/s01-01-intro"},{"id":"agent-harnesses/incident-runbook-from-the-repo/s02-02-which-is-a-runbook"},{"id":"agent-harnesses/incident-runbook-from-the-repo/s03-03-detect-decide-writeback"},{"id":"agent-harnesses/incident-runbook-from-the-repo/s04-04-predict-the-decision"},{"id":"agent-harnesses/incident-runbook-from-the-repo/s05-05-fill-the-decision-rule"},{"id":"agent-harnesses/incident-runbook-from-the-repo/s06-06-fix-the-wiki-page"},{"id":"agent-harnesses/incident-runbook-from-the-repo/s07-07-fix-the-unnamed-human"},{"id":"agent-harnesses/incident-runbook-from-the-repo/s08-08-write-the-runbook"},{"id":"agent-harnesses/incident-runbook-from-the-repo/s09-09-checkpoint"}]}]},{"slug":"ai-image-generation","lessons":[{"slug":"the-image-model-landscape","steps":[{"id":"ai-image-generation/the-image-model-landscape/s01-01-the-six-families"},{"id":"ai-image-generation/the-image-model-landscape/s02-02-flux-vs-midjourney"},{"id":"ai-image-generation/the-image-model-landscape/s03-03-pick-the-model"},{"id":"ai-image-generation/the-image-model-landscape/s04-04-nano-banana-and-the-batch-economics"},{"id":"ai-image-generation/the-image-model-landscape/s05-05-text-in-image"},{"id":"ai-image-generation/the-image-model-landscape/s06-06-which-model-for-text"},{"id":"ai-image-generation/the-image-model-landscape/s07-07-the-decision-tree"},{"id":"ai-image-generation/the-image-model-landscape/s08-08-write-pick-image-model"},{"id":"ai-image-generation/the-image-model-landscape/s09-09-checkpoint"}]},{"slug":"prompting-for-real-output","steps":[{"id":"ai-image-generation/prompting-for-real-output/s01-01-why-the-girl-with-red-hair-fails"},{"id":"ai-image-generation/prompting-for-real-output/s02-02-the-eight-knobs"},{"id":"ai-image-generation/prompting-for-real-output/s03-03-fix-the-vague-prompt"},{"id":"ai-image-generation/prompting-for-real-output/s04-04-controlnet-and-reference"},{"id":"ai-image-generation/prompting-for-real-output/s05-05-which-knob-is-missing"},{"id":"ai-image-generation/prompting-for-real-output/s06-06-failure-modes"},{"id":"ai-image-generation/prompting-for-real-output/s07-07-write-score-image-prompt"},{"id":"ai-image-generation/prompting-for-real-output/s08-08-checkpoint"}]},{"slug":"the-image-pipeline","steps":[{"id":"ai-image-generation/the-image-pipeline/s01-01-the-six-stages"},{"id":"ai-image-generation/the-image-pipeline/s02-02-batch-and-filter"},{"id":"ai-image-generation/the-image-pipeline/s03-03-which-stage-is-broken"},{"id":"ai-image-generation/the-image-pipeline/s04-04-format-and-platform"},{"id":"ai-image-generation/the-image-pipeline/s05-05-fill-the-pipeline"},{"id":"ai-image-generation/the-image-pipeline/s06-06-write-cost-for-batch"},{"id":"ai-image-generation/the-image-pipeline/s07-07-checkpoint"}]}]},{"slug":"ai-video-generation","lessons":[{"slug":"the-video-model-lineup","steps":[{"id":"ai-video-generation/the-video-model-lineup/s01-01-the-landscape"},{"id":"ai-video-generation/the-video-model-lineup/s02-02-sora-and-veo"},{"id":"ai-video-generation/the-video-model-lineup/s03-03-higgsfield-and-camera-control"},{"id":"ai-video-generation/the-video-model-lineup/s04-04-the-second-wave"},{"id":"ai-video-generation/the-video-model-lineup/s05-05-text-vs-image-vs-video"},{"id":"ai-video-generation/the-video-model-lineup/s06-06-which-lane"},{"id":"ai-video-generation/the-video-model-lineup/s07-07-pick-the-model"},{"id":"ai-video-generation/the-video-model-lineup/s08-08-write-pick-video-model"},{"id":"ai-video-generation/the-video-model-lineup/s09-09-checkpoint"}]},{"slug":"camera-control-and-motion","steps":[{"id":"ai-video-generation/camera-control-and-motion/s01-01-the-static-camera-problem"},{"id":"ai-video-generation/camera-control-and-motion/s02-02-shot-vocabulary"},{"id":"ai-video-generation/camera-control-and-motion/s03-03-which-shot-is-which"},{"id":"ai-video-generation/camera-control-and-motion/s04-04-camera-moves"},{"id":"ai-video-generation/camera-control-and-motion/s05-05-the-i2v-keyframe-pattern"},{"id":"ai-video-generation/camera-control-and-motion/s06-06-spot-the-generic-prompt"},{"id":"ai-video-generation/camera-control-and-motion/s07-07-write-audit-shot-list"},{"id":"ai-video-generation/camera-control-and-motion/s08-08-checkpoint"}]},{"slug":"the-cost-math","steps":[{"id":"ai-video-generation/the-cost-math/s01-01-the-unit-economics"},{"id":"ai-video-generation/the-cost-math/s02-02-the-retake-rate-reality"},{"id":"ai-video-generation/the-cost-math/s03-03-which-tier-for-this-budget"},{"id":"ai-video-generation/the-cost-math/s04-04-the-60-second-promo"},{"id":"ai-video-generation/the-cost-math/s05-05-write-cost-per-minute"},{"id":"ai-video-generation/the-cost-math/s06-06-checkpoint"}]}]},{"slug":"programmatic-design","lessons":[{"slug":"why-programmatic","steps":[{"id":"programmatic-design/why-programmatic/s01-01-intro"},{"id":"programmatic-design/why-programmatic/s02-02-the-three-jobs"},{"id":"programmatic-design/why-programmatic/s03-03-which-task-is-code"},{"id":"programmatic-design/why-programmatic/s04-04-parametric-vs-one-shot"},{"id":"programmatic-design/why-programmatic/s05-05-the-classifier"},{"id":"programmatic-design/why-programmatic/s06-06-write-classify-video-task"},{"id":"programmatic-design/why-programmatic/s07-07-checkpoint"}]},{"slug":"hyperframes-and-remotion","steps":[{"id":"programmatic-design/hyperframes-and-remotion/s01-01-the-two-tools"},{"id":"programmatic-design/hyperframes-and-remotion/s02-02-hyperframes-tour"},{"id":"programmatic-design/hyperframes-and-remotion/s03-03-hyperframes-cli"},{"id":"programmatic-design/hyperframes-and-remotion/s04-04-remotion-tour"},{"id":"programmatic-design/hyperframes-and-remotion/s05-05-the-decision-tree"},{"id":"programmatic-design/hyperframes-and-remotion/s06-06-pick-the-tool"},{"id":"programmatic-design/hyperframes-and-remotion/s07-07-claude-design-html"},{"id":"programmatic-design/hyperframes-and-remotion/s08-08-write-pick-tool"},{"id":"programmatic-design/hyperframes-and-remotion/s09-09-checkpoint"}]},{"slug":"the-ai-native-design-pipeline","steps":[{"id":"programmatic-design/the-ai-native-design-pipeline/s01-01-the-seven-steps"},{"id":"programmatic-design/the-ai-native-design-pipeline/s02-02-where-ai-fits-each-step"},{"id":"programmatic-design/the-ai-native-design-pipeline/s03-03-which-model-which-step"},{"id":"programmatic-design/the-ai-native-design-pipeline/s04-04-website-to-hyperframes"},{"id":"programmatic-design/the-ai-native-design-pipeline/s05-05-the-cost-model"},{"id":"programmatic-design/the-ai-native-design-pipeline/s06-06-score-the-pipeline"},{"id":"programmatic-design/the-ai-native-design-pipeline/s07-07-write-score-pipeline"},{"id":"programmatic-design/the-ai-native-design-pipeline/s08-08-checkpoint"}]}]},{"slug":"harness-engineering","lessons":[{"slug":"the-harness-engineering-mindset","steps":[{"id":"harness-engineering/the-harness-engineering-mindset/s01-01-the-model-is-only-one-input"},{"id":"harness-engineering/the-harness-engineering-mindset/s02-02-the-six-pieces-of-a-harness"},{"id":"harness-engineering/the-harness-engineering-mindset/s03-03-which-piece-is-missing"},{"id":"harness-engineering/the-harness-engineering-mindset/s04-04-the-skill-issue-reframe"},{"id":"harness-engineering/the-harness-engineering-mindset/s05-05-pick-the-fix"},{"id":"harness-engineering/the-harness-engineering-mindset/s06-06-the-harness-gap"},{"id":"harness-engineering/the-harness-engineering-mindset/s07-07-write-the-harness-inventory"},{"id":"harness-engineering/the-harness-engineering-mindset/s08-08-checkpoint"}]},{"slug":"the-ratchet","steps":[{"id":"harness-engineering/the-ratchet/s01-01-failures-are-signals-not-flukes"},{"id":"harness-engineering/the-ratchet/s02-02-every-line-traces-to-a-failure"},{"id":"harness-engineering/the-ratchet/s03-03-which-rule-earned-its-line"},{"id":"harness-engineering/the-ratchet/s04-04-working-backwards-from-behavior"},{"id":"harness-engineering/the-ratchet/s05-05-the-ratchet-log"},{"id":"harness-engineering/the-ratchet/s06-06-spot-the-cruft"},{"id":"harness-engineering/the-ratchet/s07-07-write-the-ratchet-update"},{"id":"harness-engineering/the-ratchet/s08-08-audit-the-ratchet"},{"id":"harness-engineering/the-ratchet/s09-09-checkpoint"}]},{"slug":"context-engineering","steps":[{"id":"harness-engineering/context-engineering/s01-01-the-context-rot"},{"id":"harness-engineering/context-engineering/s02-02-compaction"},{"id":"harness-engineering/context-engineering/s03-03-tool-call-offloading"},{"id":"harness-engineering/context-engineering/s04-04-progressive-disclosure"},{"id":"harness-engineering/context-engineering/s05-05-which-technique-fits"},{"id":"harness-engineering/context-engineering/s06-06-the-memory-hierarchy"},{"id":"harness-engineering/context-engineering/s07-07-classify-the-data"},{"id":"harness-engineering/context-engineering/s08-08-write-the-context-budget"},{"id":"harness-engineering/context-engineering/s09-09-checkpoint"}]},{"slug":"long-horizon-execution","steps":[{"id":"harness-engineering/long-horizon-execution/s01-01-the-40-step-problem"},{"id":"harness-engineering/long-horizon-execution/s02-02-loops-intercepting-early-stops"},{"id":"harness-engineering/long-horizon-execution/s03-03-planning-as-a-separate-step"},{"id":"harness-engineering/long-horizon-execution/s04-04-splits-generation-vs-evaluation"},{"id":"harness-engineering/long-horizon-execution/s05-05-which-pattern-fixes-the-failure"},{"id":"harness-engineering/long-horizon-execution/s06-06-hooks-as-the-enforcement-layer"},{"id":"harness-engineering/long-horizon-execution/s07-07-the-type-check-backpressure-pattern"},{"id":"harness-engineering/long-horizon-execution/s08-08-write-the-hook-router"},{"id":"harness-engineering/long-horizon-execution/s09-09-write-the-completion-guard"},{"id":"harness-engineering/long-horizon-execution/s10-10-checkpoint"}]},{"slug":"the-haas-shift","steps":[{"id":"harness-engineering/the-haas-shift/s01-01-llm-apis-to-harness-apis"},{"id":"harness-engineering/the-haas-shift/s02-02-the-convergence"},{"id":"harness-engineering/the-haas-shift/s03-03-which-haas-fits"},{"id":"harness-engineering/the-haas-shift/s04-04-harnesses-dont-shrink-they-move"},{"id":"harness-engineering/the-haas-shift/s05-05-the-overfitting-feedback-loop"},{"id":"harness-engineering/the-haas-shift/s06-06-mcp-security-as-prompt-injection"},{"id":"harness-engineering/the-haas-shift/s07-07-which-component-is-debt"},{"id":"harness-engineering/the-haas-shift/s08-08-write-the-build-vs-buy"},{"id":"harness-engineering/the-haas-shift/s09-09-checkpoint"}]},{"slug":"pipeline-boundaries","steps":[{"id":"harness-engineering/pipeline-boundaries/s01-01-the-boundary-is-an-artifact"},{"id":"harness-engineering/pipeline-boundaries/s02-02-spot-the-risky-handoff"},{"id":"harness-engineering/pipeline-boundaries/s03-03-predict-the-gate"},{"id":"harness-engineering/pipeline-boundaries/s04-04-fill-the-gate"},{"id":"harness-engineering/pipeline-boundaries/s05-05-fix-the-overtrust"},{"id":"harness-engineering/pipeline-boundaries/s06-06-write-the-advance-check"},{"id":"harness-engineering/pipeline-boundaries/s07-07-checkpoint"}]}]},{"slug":"intro-to-terminal","lessons":[{"slug":"what-the-terminal-is","steps":[{"id":"intro-to-terminal/what-the-terminal-is/s01-01-not-a-cockpit"},{"id":"intro-to-terminal/what-the-terminal-is/s02-02-what-it-actually-is"},{"id":"intro-to-terminal/what-the-terminal-is/s03-03-open-it"},{"id":"intro-to-terminal/what-the-terminal-is/s04-04-which-app"},{"id":"intro-to-terminal/what-the-terminal-is/s05-05-the-prompt"},{"id":"intro-to-terminal/what-the-terminal-is/s06-06-what-a-command-is"},{"id":"intro-to-terminal/what-the-terminal-is/s07-07-checkpoint"}]},{"slug":"moving-around","steps":[{"id":"intro-to-terminal/moving-around/s01-01-the-filesystem-is-a-tree"},{"id":"intro-to-terminal/moving-around/s02-02-pwd-and-ls"},{"id":"intro-to-terminal/moving-around/s03-03-where-am-i"},{"id":"intro-to-terminal/moving-around/s04-04-read-this-listing"},{"id":"intro-to-terminal/moving-around/s05-05-cd"},{"id":"intro-to-terminal/moving-around/s06-06-follow-the-moves"},{"id":"intro-to-terminal/moving-around/s07-07-pick-the-command"},{"id":"intro-to-terminal/moving-around/s08-08-checkpoint"}]},{"slug":"making-things","steps":[{"id":"intro-to-terminal/making-things/s01-01-mkdir-and-touch"},{"id":"intro-to-terminal/making-things/s02-02-folder-or-file"},{"id":"intro-to-terminal/making-things/s03-03-cat-and-echo"},{"id":"intro-to-terminal/making-things/s04-04-reading-a-file"},{"id":"intro-to-terminal/making-things/s05-05-rm-carefully"},{"id":"intro-to-terminal/making-things/s06-06-the-no-undo-rule"},{"id":"intro-to-terminal/making-things/s07-07-string-it-together"},{"id":"intro-to-terminal/making-things/s08-08-checkpoint"}]}]},{"slug":"intro-to-claude-cli","lessons":[{"slug":"what-the-claude-cli-is","steps":[{"id":"intro-to-claude-cli/what-the-claude-cli-is/s01-01-the-glass-is-gone"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s02-02-chat-vs-cli"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s03-03-which-tool"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s04-04-the-agent-loop"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s05-05-read-the-loop"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s06-06-permission-and-limits"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s07-07-checkpoint"}]},{"slug":"install-and-sign-in","steps":[{"id":"intro-to-claude-cli/install-and-sign-in/s01-01-two-ways-to-install"},{"id":"intro-to-claude-cli/install-and-sign-in/s02-02-run-the-installer"},{"id":"intro-to-claude-cli/install-and-sign-in/s03-03-install-check"},{"id":"intro-to-claude-cli/install-and-sign-in/s04-04-sign-in"},{"id":"intro-to-claude-cli/install-and-sign-in/s05-05-what-sign-in-did"},{"id":"intro-to-claude-cli/install-and-sign-in/s06-06-auth-check"},{"id":"intro-to-claude-cli/install-and-sign-in/s07-07-account-not-api-key"},{"id":"intro-to-claude-cli/install-and-sign-in/s08-08-checkpoint"}]},{"slug":"your-first-session","steps":[{"id":"intro-to-claude-cli/your-first-session/s01-01-start-in-a-folder"},{"id":"intro-to-claude-cli/your-first-session/s02-02-your-first-instruction"},{"id":"intro-to-claude-cli/your-first-session/s03-03-watch-the-loop"},{"id":"intro-to-claude-cli/your-first-session/s04-04-read-what-it-did"},{"id":"intro-to-claude-cli/your-first-session/s05-05-one-shot-or-interactive"},{"id":"intro-to-claude-cli/your-first-session/s06-06-chat-or-cli-decision"},{"id":"intro-to-claude-cli/your-first-session/s07-07-where-this-goes"},{"id":"intro-to-claude-cli/your-first-session/s08-08-checkpoint"}]}]},{"slug":"intro-to-codex-cli","lessons":[{"slug":"another-tool-same-idea","steps":[{"id":"intro-to-codex-cli/another-tool-same-idea/s01-01-why-a-second-tool"},{"id":"intro-to-codex-cli/another-tool-same-idea/s02-02-what-carries-over"},{"id":"intro-to-codex-cli/another-tool-same-idea/s03-03-what-carries-over-check"},{"id":"intro-to-codex-cli/another-tool-same-idea/s04-04-what-is-different"},{"id":"intro-to-codex-cli/another-tool-same-idea/s05-05-spot-the-difference"},{"id":"intro-to-codex-cli/another-tool-same-idea/s06-06-when-to-reach-for-which"},{"id":"intro-to-codex-cli/another-tool-same-idea/s07-07-checkpoint"}]},{"slug":"install-and-sign-in","steps":[{"id":"intro-to-codex-cli/install-and-sign-in/s01-01-the-install-paths"},{"id":"intro-to-codex-cli/install-and-sign-in/s02-02-installing-it"},{"id":"intro-to-codex-cli/install-and-sign-in/s03-03-verify-it"},{"id":"intro-to-codex-cli/install-and-sign-in/s04-04-install-check"},{"id":"intro-to-codex-cli/install-and-sign-in/s05-05-sign-in"},{"id":"intro-to-codex-cli/install-and-sign-in/s06-06-account-vs-api-key"},{"id":"intro-to-codex-cli/install-and-sign-in/s07-07-auth-check"},{"id":"intro-to-codex-cli/install-and-sign-in/s08-08-checkpoint"}]},{"slug":"your-first-session","steps":[{"id":"intro-to-codex-cli/your-first-session/s01-01-start-and-instruct"},{"id":"intro-to-codex-cli/your-first-session/s02-02-the-loop-again"},{"id":"intro-to-codex-cli/your-first-session/s03-03-read-what-it-did"},{"id":"intro-to-codex-cli/your-first-session/s04-04-usage-patterns"},{"id":"intro-to-codex-cli/your-first-session/s05-05-usage-scenario"},{"id":"intro-to-codex-cli/your-first-session/s06-06-running-both"},{"id":"intro-to-codex-cli/your-first-session/s07-07-where-this-goes"},{"id":"intro-to-codex-cli/your-first-session/s08-08-checkpoint"}]}]},{"slug":"team-skills","lessons":[{"slug":"what-a-skill-is","steps":[{"id":"team-skills/what-a-skill-is/s01-01-the-re-explaining-problem"},{"id":"team-skills/what-a-skill-is/s02-02-a-skill-defined"},{"id":"team-skills/what-a-skill-is/s03-03-a-real-skill"},{"id":"team-skills/what-a-skill-is/s04-04-what-a-skill-is-check"},{"id":"team-skills/what-a-skill-is/s05-05-skill-vs-prompt-vs-agent"},{"id":"team-skills/what-a-skill-is/s06-06-skill-or-agent"},{"id":"team-skills/what-a-skill-is/s07-07-checkpoint"}]},{"slug":"skills-as-a-team-thing","steps":[{"id":"team-skills/skills-as-a-team-thing/s01-01-the-team-problem"},{"id":"team-skills/skills-as-a-team-thing/s02-02-org-wide-skills"},{"id":"team-skills/skills-as-a-team-thing/s03-03-sharing-and-the-directory"},{"id":"team-skills/skills-as-a-team-thing/s04-04-deployment-check"},{"id":"team-skills/skills-as-a-team-thing/s05-05-collaborating-through-skills"},{"id":"team-skills/skills-as-a-team-thing/s06-06-sharing-scenario"},{"id":"team-skills/skills-as-a-team-thing/s07-07-checkpoint"}]},{"slug":"role-playbooks","steps":[{"id":"team-skills/role-playbooks/s01-01-hr-skills"},{"id":"team-skills/role-playbooks/s02-02-legal-skills"},{"id":"team-skills/role-playbooks/s03-03-operations-skills"},{"id":"team-skills/role-playbooks/s04-04-match-the-skill"},{"id":"team-skills/role-playbooks/s05-05-skill-or-one-off"},{"id":"team-skills/role-playbooks/s06-06-skill-or-one-off-check"},{"id":"team-skills/role-playbooks/s07-07-checkpoint"}]},{"slug":"governance-and-trust","steps":[{"id":"team-skills/governance-and-trust/s01-01-a-skill-can-do-real-things"},{"id":"team-skills/governance-and-trust/s02-02-treat-it-like-software"},{"id":"team-skills/governance-and-trust/s03-03-evaluate-before-you-deploy"},{"id":"team-skills/governance-and-trust/s04-04-access-control"},{"id":"team-skills/governance-and-trust/s05-05-governance-scenario"},{"id":"team-skills/governance-and-trust/s06-06-a-rollout-checklist"},{"id":"team-skills/governance-and-trust/s07-07-checkpoint"}]}]},{"slug":"dataframes-numpy-pandas","lessons":[{"slug":"ml-package-map","steps":[{"id":"dataframes-numpy-pandas/ml-package-map/s01-01-intro"},{"id":"dataframes-numpy-pandas/ml-package-map/s02-02-match-the-tool"},{"id":"dataframes-numpy-pandas/ml-package-map/s03-03-predict-sys-modules"},{"id":"dataframes-numpy-pandas/ml-package-map/s04-04-fill-the-registry-key"},{"id":"dataframes-numpy-pandas/ml-package-map/s05-05-fix-the-swapped-map"},{"id":"dataframes-numpy-pandas/ml-package-map/s06-06-write-find-unused"},{"id":"dataframes-numpy-pandas/ml-package-map/s07-07-checkpoint"}]},{"slug":"array-and-series-shapes","steps":[{"id":"dataframes-numpy-pandas/array-and-series-shapes/s01-01-intro"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s02-02-choose-the-risk"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s03-03-predict-the-checks"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s04-04-fill-the-return"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s05-05-fix-the-overtrust"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s06-06-write-count-ready"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s07-07-checkpoint"}]},{"slug":"dataframe-selection-and-cleaning","steps":[{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s01-01-intro"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s02-02-choose-the-risk"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s03-03-predict-the-checks"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s04-04-fill-the-return"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s05-05-fix-the-overtrust"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s06-06-write-count-ready"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s07-07-checkpoint"}]},{"slug":"missing-values-and-types","steps":[{"id":"dataframes-numpy-pandas/missing-values-and-types/s01-01-intro"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s02-02-choose-the-risk"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s03-03-predict-the-checks"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s04-04-fill-the-return"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s05-05-fix-the-overtrust"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s06-06-write-count-ready"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s07-07-checkpoint"}]},{"slug":"groupby-joins-and-features","steps":[{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s01-01-intro"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s02-02-choose-the-risk"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s03-03-predict-the-checks"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s04-04-fill-the-return"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s05-05-fix-the-overtrust"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s06-06-write-count-ready"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s07-07-checkpoint"}]},{"slug":"mission-api-to-dataframe","steps":[{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s01-01-intro"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s02-02-choose-the-risk"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s03-03-predict-the-checks"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s04-04-fill-the-return"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s05-05-fix-the-overtrust"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s06-06-write-count-ready"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s07-07-checkpoint"}]}]},{"slug":"sql-for-ml-datasets","lessons":[{"slug":"select-filter-aggregate","steps":[{"id":"sql-for-ml-datasets/select-filter-aggregate/s01-01-intro"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s02-02-choose-the-risk"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s03-03-predict-the-checks"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s04-04-fill-the-return"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s05-05-fix-the-overtrust"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s06-06-write-count-ready"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s07-07-checkpoint"}]},{"slug":"joins-and-label-windows","steps":[{"id":"sql-for-ml-datasets/joins-and-label-windows/s01-01-intro"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s02-02-choose-the-risk"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s03-03-predict-the-checks"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s04-04-fill-the-return"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s05-05-fix-the-overtrust"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s06-06-write-count-ready"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s07-07-checkpoint"}]},{"slug":"sql-quality-checks","steps":[{"id":"sql-for-ml-datasets/sql-quality-checks/s01-01-intro"},{"id":"sql-for-ml-datasets/sql-quality-checks/s02-02-choose-the-risk"},{"id":"sql-for-ml-datasets/sql-quality-checks/s03-03-predict-the-checks"},{"id":"sql-for-ml-datasets/sql-quality-checks/s04-04-fill-the-return"},{"id":"sql-for-ml-datasets/sql-quality-checks/s05-05-fix-the-overtrust"},{"id":"sql-for-ml-datasets/sql-quality-checks/s06-06-write-count-ready"},{"id":"sql-for-ml-datasets/sql-quality-checks/s07-07-checkpoint"}]},{"slug":"mission-sql-feature-query-lab","steps":[{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s01-01-intro"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s02-02-choose-the-risk"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s03-03-predict-the-checks"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s04-04-fill-the-return"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s05-05-fix-the-overtrust"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s06-06-write-count-ready"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s07-07-checkpoint"}]}]},{"slug":"dataset-pipelines","lessons":[{"slug":"csv-jsonl-parquet-tradeoffs","steps":[{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s01-01-intro"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s02-02-choose-the-risk"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s03-03-predict-the-checks"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s04-04-fill-the-return"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s05-05-fix-the-overtrust"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s06-06-write-count-ready"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s07-07-checkpoint"}]},{"slug":"pagination-retries-checkpoints","steps":[{"id":"dataset-pipelines/pagination-retries-checkpoints/s01-01-intro"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s02-02-choose-the-risk"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s03-03-predict-the-checks"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s04-04-fill-the-return"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s05-05-fix-the-overtrust"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s06-06-write-count-ready"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s07-07-checkpoint"}]},{"slug":"schema-drift-and-contracts","steps":[{"id":"dataset-pipelines/schema-drift-and-contracts/s01-01-intro"},{"id":"dataset-pipelines/schema-drift-and-contracts/s02-02-choose-the-risk"},{"id":"dataset-pipelines/schema-drift-and-contracts/s03-03-predict-the-checks"},{"id":"dataset-pipelines/schema-drift-and-contracts/s04-04-fill-the-return"},{"id":"dataset-pipelines/schema-drift-and-contracts/s05-05-fix-the-overtrust"},{"id":"dataset-pipelines/schema-drift-and-contracts/s06-06-write-count-ready"},{"id":"dataset-pipelines/schema-drift-and-contracts/s07-07-checkpoint"}]},{"slug":"pipeline-orchestration-shape","steps":[{"id":"dataset-pipelines/pipeline-orchestration-shape/s01-01-intro"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s02-02-choose-the-risk"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s03-03-predict-the-checks"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s04-04-fill-the-return"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s05-05-fix-the-overtrust"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s06-06-write-count-ready"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s07-07-checkpoint"}]},{"slug":"mission-api-to-dataset-pipeline","steps":[{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s01-01-intro"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s02-02-choose-the-risk"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s03-03-predict-the-checks"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s04-04-fill-the-return"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s05-05-fix-the-overtrust"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s06-06-write-count-ready"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s07-07-checkpoint"}]}]},{"slug":"ml-math-and-stats","lessons":[{"slug":"vectors-matrices-dot-products","steps":[{"id":"ml-math-and-stats/vectors-matrices-dot-products/s01-01-intro"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s02-02-choose-the-risk"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s03-03-predict-the-checks"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s04-04-fill-the-return"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s05-05-fix-the-overtrust"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s06-06-write-count-ready"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s07-07-checkpoint"}]},{"slug":"probability-and-baselines","steps":[{"id":"ml-math-and-stats/probability-and-baselines/s01-01-intro"},{"id":"ml-math-and-stats/probability-and-baselines/s02-02-choose-the-risk"},{"id":"ml-math-and-stats/probability-and-baselines/s03-03-predict-the-checks"},{"id":"ml-math-and-stats/probability-and-baselines/s04-04-fill-the-return"},{"id":"ml-math-and-stats/probability-and-baselines/s05-05-fix-the-overtrust"},{"id":"ml-math-and-stats/probability-and-baselines/s06-06-write-count-ready"},{"id":"ml-math-and-stats/probability-and-baselines/s07-07-checkpoint"}]},{"slug":"distributions-sampling-variance","steps":[{"id":"ml-math-and-stats/distributions-sampling-variance/s01-01-intro"},{"id":"ml-math-and-stats/distributions-sampling-variance/s02-02-choose-the-risk"},{"id":"ml-math-and-stats/distributions-sampling-variance/s03-03-predict-the-checks"},{"id":"ml-math-and-stats/distributions-sampling-variance/s04-04-fill-the-return"},{"id":"ml-math-and-stats/distributions-sampling-variance/s05-05-fix-the-overtrust"},{"id":"ml-math-and-stats/distributions-sampling-variance/s06-06-write-variance"},{"id":"ml-math-and-stats/distributions-sampling-variance/s07-07-checkpoint"}]},{"slug":"bias-variance-and-leakage","steps":[{"id":"ml-math-and-stats/bias-variance-and-leakage/s01-01-intro"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s02-02-choose-the-risk"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s03-03-predict-the-checks"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s04-04-fill-the-return"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s05-05-fix-the-overtrust"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s06-06-write-diagnose"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s07-07-checkpoint"}]},{"slug":"mission-statistical-sanity-check","steps":[{"id":"ml-math-and-stats/mission-statistical-sanity-check/s01-01-intro"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s02-02-choose-the-risk"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s03-03-predict-the-checks"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s04-04-fill-the-return"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s05-05-fix-the-overtrust"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s06-06-write-count-ready"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s07-07-checkpoint"}]}]},{"slug":"supervised-learning-workflows","lessons":[{"slug":"labels-features-splits","steps":[{"id":"supervised-learning-workflows/labels-features-splits/s01-01-intro"},{"id":"supervised-learning-workflows/labels-features-splits/s02-02-choose-the-risk"},{"id":"supervised-learning-workflows/labels-features-splits/s03-03-predict-the-checks"},{"id":"supervised-learning-workflows/labels-features-splits/s04-04-fill-the-return"},{"id":"supervised-learning-workflows/labels-features-splits/s05-05-fix-the-overtrust"},{"id":"supervised-learning-workflows/labels-features-splits/s06-06-write-split-audit"},{"id":"supervised-learning-workflows/labels-features-splits/s07-07-checkpoint"}]},{"slug":"baselines-before-fancy-models","steps":[{"id":"supervised-learning-workflows/baselines-before-fancy-models/s01-01-intro"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s02-02-choose-the-risk"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s03-03-predict-the-checks"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s04-04-fill-the-return"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s05-05-fix-the-overtrust"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s06-06-write-baseline-comparison"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s07-07-checkpoint"}]},{"slug":"training-and-prediction-loop","steps":[{"id":"supervised-learning-workflows/training-and-prediction-loop/s01-01-intro"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s02-02-choose-the-risk"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s03-03-predict-the-checks"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s04-04-fill-the-return"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s05-05-fix-the-overtrust"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s06-06-write-train"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s07-07-checkpoint"}]},{"slug":"overfitting-and-regularization","steps":[{"id":"supervised-learning-workflows/overfitting-and-regularization/s01-01-intro"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s02-02-choose-the-risk"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s03-03-predict-the-checks"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s04-04-fill-the-return"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s05-05-fix-the-overtrust"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s06-06-write-overfit-audit"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s07-07-checkpoint"}]},{"slug":"mission-baseline-model-showdown","steps":[{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s01-01-intro"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s02-02-choose-the-risk"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s03-03-predict-the-checks"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s04-04-fill-the-return"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s05-05-fix-the-overtrust"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s06-06-write-classifier-brief"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s07-07-checkpoint"}]}]},{"slug":"unsupervised-learning-and-embeddings","lessons":[{"slug":"clustering-without-labels","steps":[{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s01-01-intro"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s02-02-choose-the-risk"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s03-03-predict-the-checks"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s04-04-fill-the-return"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s05-05-fix-the-overtrust"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s06-06-write-cluster-summary"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s07-07-checkpoint"}]},{"slug":"dimensionality-and-distance","steps":[{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s01-01-intro"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s02-02-choose-the-risk"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s03-03-predict-the-checks"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s04-04-fill-the-return"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s05-05-fix-the-overtrust"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s06-06-write-euclidean"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s07-07-checkpoint"}]},{"slug":"embeddings-as-features","steps":[{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s01-01-intro"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s02-02-choose-the-risk"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s03-03-predict-the-checks"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s04-04-fill-the-return"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s05-05-fix-the-overtrust"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s06-06-write-retrieval-receipt"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s07-07-checkpoint"}]},{"slug":"mission-tiny-recommender","steps":[{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s01-01-intro"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s02-02-choose-the-risk"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s03-03-predict-the-checks"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s04-04-fill-the-return"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s05-05-fix-the-overtrust"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s06-06-write-recommendation-card"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s07-07-checkpoint"}]}]},{"slug":"metrics-and-error-analysis","lessons":[{"slug":"confusion-matrix","steps":[{"id":"metrics-and-error-analysis/confusion-matrix/s01-01-intro"},{"id":"metrics-and-error-analysis/confusion-matrix/s02-02-choose-the-risk"},{"id":"metrics-and-error-analysis/confusion-matrix/s03-03-predict-the-checks"},{"id":"metrics-and-error-analysis/confusion-matrix/s04-04-fill-the-return"},{"id":"metrics-and-error-analysis/confusion-matrix/s05-05-fix-the-overtrust"},{"id":"metrics-and-error-analysis/confusion-matrix/s06-06-write-count-ready"},{"id":"metrics-and-error-analysis/confusion-matrix/s07-07-checkpoint"}]},{"slug":"precision-recall-thresholds","steps":[{"id":"metrics-and-error-analysis/precision-recall-thresholds/s01-01-intro"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s02-02-choose-the-risk"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s03-03-predict-the-checks"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s04-04-fill-the-return"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s05-05-fix-the-overtrust"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s06-06-write-count-ready"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s07-07-checkpoint"}]},{"slug":"slice-based-error-analysis","steps":[{"id":"metrics-and-error-analysis/slice-based-error-analysis/s01-01-intro"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s02-02-choose-the-risk"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s03-03-predict-the-checks"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s04-04-fill-the-return"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s05-05-fix-the-overtrust"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s06-06-write-count-ready"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s07-07-checkpoint"}]},{"slug":"regression-metrics-and-residuals","steps":[{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s01-01-intro"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s02-02-choose-the-risk"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s03-03-predict-the-checks"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s04-04-fill-the-return"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s05-05-fix-the-overtrust"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s06-06-write-count-ready"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s07-07-checkpoint"}]},{"slug":"mission-confusion-matrix-triage","steps":[{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s01-01-intro"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s02-02-choose-the-risk"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s03-03-predict-the-checks"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s04-04-fill-the-return"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s05-05-fix-the-overtrust"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s06-06-write-count-ready"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s07-07-checkpoint"}]}]},{"slug":"pytorch-tensors-and-autograd","lessons":[{"slug":"tensors-and-shapes","steps":[{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s01-01-intro"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s02-02-choose-the-risk"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s03-03-predict-the-checks"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s04-04-fill-the-return"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s05-05-fix-the-overtrust"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s06-06-write-count-ready"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s07-07-checkpoint"}]},{"slug":"broadcasting-and-vectorization","steps":[{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s01-01-intro"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s02-02-choose-the-risk"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s03-03-predict-the-checks"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s04-04-fill-the-return"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s05-05-fix-the-overtrust"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s06-06-write-count-ready"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s07-07-checkpoint"}]},{"slug":"autograd-mental-model","steps":[{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s01-01-intro"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s02-02-choose-the-risk"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s03-03-predict-the-checks"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s04-04-fill-the-return"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s05-05-fix-the-overtrust"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s06-06-write-count-ready"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s07-07-checkpoint"}]},{"slug":"mission-tensor-playground","steps":[{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s01-01-intro"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s02-02-choose-the-risk"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s03-03-predict-the-checks"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s04-04-fill-the-return"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s05-05-fix-the-overtrust"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s06-06-write-count-ready"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s07-07-checkpoint"}]}]},{"slug":"training-loops-and-optimizers","lessons":[{"slug":"loss-forward-backward-step","steps":[{"id":"training-loops-and-optimizers/loss-forward-backward-step/s01-01-intro"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s02-02-choose-the-risk"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s03-03-predict-the-checks"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s04-04-fill-the-return"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s05-05-fix-the-overtrust"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s06-06-write-count-ready"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s07-07-checkpoint"}]},{"slug":"gradient-descent-by-hand","steps":[{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s01-01-intro"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s02-02-choose-the-risk"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s03-03-predict-the-checks"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s04-04-fill-the-return"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s05-05-fix-the-overtrust"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s06-06-write-count-ready"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s07-07-checkpoint"}]},{"slug":"optimizers-and-schedulers","steps":[{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s01-01-intro"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s02-02-choose-the-risk"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s03-03-predict-the-checks"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s04-04-fill-the-return"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s05-05-fix-the-overtrust"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s06-06-write-count-ready"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s07-07-checkpoint"}]},{"slug":"checkpoints-and-reproducibility","steps":[{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s01-01-intro"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s02-02-choose-the-risk"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s03-03-predict-the-checks"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s04-04-fill-the-return"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s05-05-fix-the-overtrust"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s06-06-write-count-ready"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s07-07-checkpoint"}]},{"slug":"mission-overfit-then-recover","steps":[{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s01-01-intro"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s02-02-choose-the-risk"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s03-03-predict-the-checks"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s04-04-fill-the-return"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s05-05-fix-the-overtrust"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s06-06-write-count-ready"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s07-07-checkpoint"}]}]},{"slug":"deep-learning-architectures","lessons":[{"slug":"cnns-and-local-patterns","steps":[{"id":"deep-learning-architectures/cnns-and-local-patterns/s01-01-intro"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s02-02-choose-the-risk"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s03-03-predict-the-checks"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s04-04-fill-the-return"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s05-05-fix-the-overtrust"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s06-06-write-count-ready"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s07-07-checkpoint"}]},{"slug":"tokenizers-and-context-budget","steps":[{"id":"deep-learning-architectures/tokenizers-and-context-budget/s01-01-intro"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s02-02-choose-the-risk"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s03-03-predict-the-checks"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s04-04-fill-the-return"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s05-05-fix-the-overtrust"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s06-06-write-count-ready"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s07-07-checkpoint"}]},{"slug":"attention-and-transformer-blocks","steps":[{"id":"deep-learning-architectures/attention-and-transformer-blocks/s01-01-intro"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s02-02-choose-the-risk"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s03-03-predict-the-checks"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s04-04-fill-the-return"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s05-05-fix-the-overtrust"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s06-06-write-count-ready"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s07-07-checkpoint"}]},{"slug":"decoding-kv-cache-quantization","steps":[{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s01-01-intro"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s02-02-choose-the-risk"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s03-03-predict-the-checks"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s04-04-fill-the-return"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s05-05-fix-the-overtrust"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s06-06-write-count-ready"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s07-07-checkpoint"}]},{"slug":"mission-architecture-tradeoff-note","steps":[{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s01-01-intro"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s02-02-choose-the-risk"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s03-03-predict-the-checks"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s04-04-fill-the-return"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s05-05-fix-the-overtrust"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s06-06-write-count-ready"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s07-07-checkpoint"}]}]},{"slug":"feature-experiments-registries","lessons":[{"slug":"feature-pipeline-contracts","steps":[{"id":"feature-experiments-registries/feature-pipeline-contracts/s01-01-intro"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s02-02-choose-the-risk"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s03-03-predict-the-checks"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s04-04-fill-the-return"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s05-05-fix-the-overtrust"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s06-06-write-count-ready"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s07-07-checkpoint"}]},{"slug":"train-inference-skew","steps":[{"id":"feature-experiments-registries/train-inference-skew/s01-01-intro"},{"id":"feature-experiments-registries/train-inference-skew/s02-02-choose-the-risk"},{"id":"feature-experiments-registries/train-inference-skew/s03-03-predict-the-checks"},{"id":"feature-experiments-registries/train-inference-skew/s04-04-fill-the-return"},{"id":"feature-experiments-registries/train-inference-skew/s05-05-fix-the-overtrust"},{"id":"feature-experiments-registries/train-inference-skew/s06-06-write-count-ready"},{"id":"feature-experiments-registries/train-inference-skew/s07-07-checkpoint"}]},{"slug":"experiment-tracker-lite","steps":[{"id":"feature-experiments-registries/experiment-tracker-lite/s01-01-intro"},{"id":"feature-experiments-registries/experiment-tracker-lite/s02-02-choose-the-risk"},{"id":"feature-experiments-registries/experiment-tracker-lite/s03-03-predict-the-checks"},{"id":"feature-experiments-registries/experiment-tracker-lite/s04-04-fill-the-return"},{"id":"feature-experiments-registries/experiment-tracker-lite/s05-05-fix-the-overtrust"},{"id":"feature-experiments-registries/experiment-tracker-lite/s06-06-write-count-ready"},{"id":"feature-experiments-registries/experiment-tracker-lite/s07-07-checkpoint"}]},{"slug":"model-registry-shape","steps":[{"id":"feature-experiments-registries/model-registry-shape/s01-01-intro"},{"id":"feature-experiments-registries/model-registry-shape/s02-02-choose-the-risk"},{"id":"feature-experiments-registries/model-registry-shape/s03-03-predict-the-checks"},{"id":"feature-experiments-registries/model-registry-shape/s04-04-fill-the-return"},{"id":"feature-experiments-registries/model-registry-shape/s05-05-fix-the-overtrust"},{"id":"feature-experiments-registries/model-registry-shape/s06-06-write-count-ready"},{"id":"feature-experiments-registries/model-registry-shape/s07-07-checkpoint"}]},{"slug":"mission-feature-pipeline-and-tracker","steps":[{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s01-01-intro"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s02-02-choose-the-risk"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s03-03-predict-the-checks"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s04-04-fill-the-return"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s05-05-fix-the-overtrust"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s06-06-write-count-ready"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s07-07-checkpoint"}]}]},{"slug":"model-serving-and-mlops","lessons":[{"slug":"fastapi-inference-shape","steps":[{"id":"model-serving-and-mlops/fastapi-inference-shape/s01-01-intro"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s06-06-write-count-ready"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s07-07-checkpoint"}]},{"slug":"batch-vs-realtime-serving","steps":[{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s01-01-intro"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s06-06-write-count-ready"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s07-07-checkpoint"}]},{"slug":"docker-runtime-and-config","steps":[{"id":"model-serving-and-mlops/docker-runtime-and-config/s01-01-intro"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s06-06-write-count-ready"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s07-07-checkpoint"}]},{"slug":"github-actions-validation-gates","steps":[{"id":"model-serving-and-mlops/github-actions-validation-gates/s01-01-intro"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s06-06-write-failing-checks"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s07-07-checkpoint"}]},{"slug":"continuous-training-and-rollout","steps":[{"id":"model-serving-and-mlops/continuous-training-and-rollout/s01-01-intro"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s06-06-write-count-ready"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s07-07-checkpoint"}]},{"slug":"mission-fastapi-model-server","steps":[{"id":"model-serving-and-mlops/mission-fastapi-model-server/s01-01-intro"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s06-06-write-count-ready"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s07-07-checkpoint"}]}]},{"slug":"monitoring-cloud-portfolio","lessons":[{"slug":"structured-logs-and-alerts","steps":[{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s01-01-intro"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s06-06-write-count-ready"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s07-07-checkpoint"}]},{"slug":"data-and-concept-drift","steps":[{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s01-01-intro"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s06-06-write-count-ready"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s07-07-checkpoint"}]},{"slug":"retraining-triggers","steps":[{"id":"monitoring-cloud-portfolio/retraining-triggers/s01-01-intro"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s06-06-write-count-ready"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s07-07-checkpoint"}]},{"slug":"cloud-gpu-kubernetes-cost","steps":[{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s01-01-intro"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s06-06-write-count-ready"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s07-07-checkpoint"}]},{"slug":"portfolio-architecture-docs","steps":[{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s01-01-intro"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s06-06-write-count-ready"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s07-07-checkpoint"}]},{"slug":"mission-final-portfolio-ml-system","steps":[{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s01-01-intro"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s06-06-write-decide"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s07-07-checkpoint"}]}]},{"slug":"graphic-design-studio","lessons":[{"slug":"prompting-like-an-art-director","steps":[{"id":"graphic-design-studio/prompting-like-an-art-director/s01-01-the-prompt-is-a-creative-brief"},{"id":"graphic-design-studio/prompting-like-an-art-director/s02-02-style-language"},{"id":"graphic-design-studio/prompting-like-an-art-director/s03-03-which-tool-for-the-headline"},{"id":"graphic-design-studio/prompting-like-an-art-director/s04-04-predict-the-scaffold"},{"id":"graphic-design-studio/prompting-like-an-art-director/s05-05-negative-space-and-the-iteration-loop"},{"id":"graphic-design-studio/prompting-like-an-art-director/s06-06-fix-the-mood-word-prompt"},{"id":"graphic-design-studio/prompting-like-an-art-director/s07-07-write-prompt-linter"},{"id":"graphic-design-studio/prompting-like-an-art-director/s08-08-checkpoint"}]},{"slug":"the-design-kit","steps":[{"id":"graphic-design-studio/the-design-kit/s01-01-the-brand-bible-as-data"},{"id":"graphic-design-studio/the-design-kit/s02-02-fill-the-palette-line"},{"id":"graphic-design-studio/the-design-kit/s03-03-banned-looks"},{"id":"graphic-design-studio/the-design-kit/s04-04-predict-the-negatives"},{"id":"graphic-design-studio/the-design-kit/s05-05-which-field-failed"},{"id":"graphic-design-studio/the-design-kit/s06-06-fix-the-crossed-kit"},{"id":"graphic-design-studio/the-design-kit/s07-07-write-contrast-check"},{"id":"graphic-design-studio/the-design-kit/s08-08-tokens-for-the-interface"},{"id":"graphic-design-studio/the-design-kit/s09-09-write-token-audit"},{"id":"graphic-design-studio/the-design-kit/s10-10-checkpoint"}]},{"slug":"fifty-assets-one-brand","steps":[{"id":"graphic-design-studio/fifty-assets-one-brand/s01-01-the-consistency-problem"},{"id":"graphic-design-studio/fifty-assets-one-brand/s02-02-seeds-the-same-dice-twice"},{"id":"graphic-design-studio/fifty-assets-one-brand/s03-03-why-does-the-mascot-drift"},{"id":"graphic-design-studio/fifty-assets-one-brand/s04-04-predict-the-batch-plan"},{"id":"graphic-design-studio/fifty-assets-one-brand/s05-05-the-qa-filter"},{"id":"graphic-design-studio/fifty-assets-one-brand/s06-06-defend-the-pick"},{"id":"graphic-design-studio/fifty-assets-one-brand/s07-07-fix-the-qa-threshold"},{"id":"graphic-design-studio/fifty-assets-one-brand/s08-08-write-palette-distance"},{"id":"graphic-design-studio/fifty-assets-one-brand/s09-09-checkpoint"}]}]},{"slug":"campaign-studio","lessons":[{"slug":"voice-pack-as-data","steps":[{"id":"campaign-studio/voice-pack-as-data/s01-01-the-campaign-and-the-gate"},{"id":"campaign-studio/voice-pack-as-data/s02-02-lint-the-draft"},{"id":"campaign-studio/voice-pack-as-data/s03-03-predict-the-violations"},{"id":"campaign-studio/voice-pack-as-data/s04-04-testable-or-mood"},{"id":"campaign-studio/voice-pack-as-data/s05-05-fill-the-linter"},{"id":"campaign-studio/voice-pack-as-data/s06-06-variants-as-records"},{"id":"campaign-studio/voice-pack-as-data/s07-07-write-lint-copy"},{"id":"campaign-studio/voice-pack-as-data/s08-08-checkpoint"}]},{"slug":"batch-plan-and-calendar","steps":[{"id":"campaign-studio/batch-plan-and-calendar/s01-01-slots-before-prompts"},{"id":"campaign-studio/batch-plan-and-calendar/s02-02-which-field-is-missing"},{"id":"campaign-studio/batch-plan-and-calendar/s03-03-write-batch-plan"},{"id":"campaign-studio/batch-plan-and-calendar/s04-04-the-calendar-is-data"},{"id":"campaign-studio/batch-plan-and-calendar/s05-05-predict-the-queue"},{"id":"campaign-studio/batch-plan-and-calendar/s06-06-fix-the-gate"},{"id":"campaign-studio/batch-plan-and-calendar/s07-07-checkpoint"}]},{"slug":"the-numbers-pass","steps":[{"id":"campaign-studio/the-numbers-pass/s01-01-a-report-is-a-draft-over-data"},{"id":"campaign-studio/the-numbers-pass/s02-02-predict-the-verdict"},{"id":"campaign-studio/the-numbers-pass/s03-03-write-verify-numbers"},{"id":"campaign-studio/the-numbers-pass/s04-04-denominators-and-the-ftc"},{"id":"campaign-studio/the-numbers-pass/s05-05-what-may-not-ship"},{"id":"campaign-studio/the-numbers-pass/s06-06-fill-the-policy-router"},{"id":"campaign-studio/the-numbers-pass/s07-07-checkpoint"}]}]},{"slug":"support-studio","lessons":[{"slug":"the-policy-pack","steps":[{"id":"support-studio/the-policy-pack/s01-01-the-reply-that-binds-the-company"},{"id":"support-studio/the-policy-pack/s02-02-the-pack-as-data"},{"id":"support-studio/the-policy-pack/s03-03-spot-the-banned-promise"},{"id":"support-studio/the-policy-pack/s04-04-predict-the-lint"},{"id":"support-studio/the-policy-pack/s05-05-fix-the-stale-citation"},{"id":"support-studio/the-policy-pack/s06-06-the-review-loop-evidence"},{"id":"support-studio/the-policy-pack/s07-07-write-lint-draft"},{"id":"support-studio/the-policy-pack/s08-08-checkpoint"}]},{"slug":"golden-evals-earn-autonomy","steps":[{"id":"support-studio/golden-evals-earn-autonomy/s01-01-the-klarna-arc"},{"id":"support-studio/golden-evals-earn-autonomy/s02-02-a-macro-is-three-things"},{"id":"support-studio/golden-evals-earn-autonomy/s03-03-inside-the-lane"},{"id":"support-studio/golden-evals-earn-autonomy/s04-04-predict-the-eval-run"},{"id":"support-studio/golden-evals-earn-autonomy/s05-05-fill-the-promotion-rule"},{"id":"support-studio/golden-evals-earn-autonomy/s06-06-write-run-golden-evals"},{"id":"support-studio/golden-evals-earn-autonomy/s07-07-checkpoint"}]},{"slug":"the-escalation-boundary","steps":[{"id":"support-studio/the-escalation-boundary/s01-01-drafting-is-not-spending"},{"id":"support-studio/the-escalation-boundary/s02-02-triggers-as-conditions"},{"id":"support-studio/the-escalation-boundary/s03-03-route-this-ticket"},{"id":"support-studio/the-escalation-boundary/s04-04-predict-the-routing"},{"id":"support-studio/the-escalation-boundary/s05-05-fix-the-refund-hole"},{"id":"support-studio/the-escalation-boundary/s06-06-disclosure-and-the-kill-switch"},{"id":"support-studio/the-escalation-boundary/s07-07-write-route-ticket"},{"id":"support-studio/the-escalation-boundary/s08-08-checkpoint"}]}]},{"slug":"copy-studio","lessons":[{"slug":"the-voice-fingerprint","steps":[{"id":"copy-studio/the-voice-fingerprint/s01-01-the-client-you-inherit"},{"id":"copy-studio/the-voice-fingerprint/s02-02-the-fingerprint"},{"id":"copy-studio/the-voice-fingerprint/s03-03-predict-the-ceiling"},{"id":"copy-studio/the-voice-fingerprint/s04-04-the-samples-say-otherwise"},{"id":"copy-studio/the-voice-fingerprint/s05-05-fill-derive-the-rules"},{"id":"copy-studio/the-voice-fingerprint/s06-06-the-pack-with-provenance"},{"id":"copy-studio/the-voice-fingerprint/s07-07-write-fingerprint"},{"id":"copy-studio/the-voice-fingerprint/s08-08-checkpoint"}]},{"slug":"variants-that-hold-the-line","steps":[{"id":"copy-studio/variants-that-hold-the-line/s01-01-twelve-by-thursday"},{"id":"copy-studio/variants-that-hold-the-line/s02-02-the-drift-problem"},{"id":"copy-studio/variants-that-hold-the-line/s03-03-predict-the-drift-report"},{"id":"copy-studio/variants-that-hold-the-line/s04-04-which-subject-line-ships"},{"id":"copy-studio/variants-that-hold-the-line/s05-05-fix-the-channel-check"},{"id":"copy-studio/variants-that-hold-the-line/s06-06-the-kill-log"},{"id":"copy-studio/variants-that-hold-the-line/s07-07-write-voice-gate"},{"id":"copy-studio/variants-that-hold-the-line/s08-08-checkpoint"}]},{"slug":"the-claim-check","steps":[{"id":"copy-studio/the-claim-check/s01-01-forty-one-of-seventy-seven"},{"id":"copy-studio/the-claim-check/s02-02-what-counts-as-checkable"},{"id":"copy-studio/the-claim-check/s03-03-predict-the-claim-list"},{"id":"copy-studio/the-claim-check/s04-04-what-closes-a-row"},{"id":"copy-studio/the-claim-check/s05-05-fill-the-ship-decision"},{"id":"copy-studio/the-claim-check/s06-06-the-corrections-log"},{"id":"copy-studio/the-claim-check/s07-07-write-ship-gate"},{"id":"copy-studio/the-claim-check/s08-08-checkpoint"}]}]},{"slug":"analytics-studio","lessons":[{"slug":"the-cleaning-log","steps":[{"id":"analytics-studio/the-cleaning-log/s01-01-where-did-those-rows-go"},{"id":"analytics-studio/the-cleaning-log/s02-02-the-log-as-data"},{"id":"analytics-studio/the-cleaning-log/s03-03-refuse-the-silent-step"},{"id":"analytics-studio/the-cleaning-log/s04-04-predict-the-receipts"},{"id":"analytics-studio/the-cleaning-log/s05-05-fill-the-null-decision"},{"id":"analytics-studio/the-cleaning-log/s06-06-fix-the-vanishing-rows"},{"id":"analytics-studio/the-cleaning-log/s07-07-write-the-null-ledger"},{"id":"analytics-studio/the-cleaning-log/s08-08-checkpoint"}]},{"slug":"every-number-has-an-address","steps":[{"id":"analytics-studio/every-number-has-an-address/s01-01-the-plausible-wrong-number"},{"id":"analytics-studio/every-number-has-an-address/s02-02-sample-first-then-full"},{"id":"analytics-studio/every-number-has-an-address/s03-03-which-check-catches-it"},{"id":"analytics-studio/every-number-has-an-address/s04-04-predict-the-fan-out"},{"id":"analytics-studio/every-number-has-an-address/s05-05-fill-the-trace"},{"id":"analytics-studio/every-number-has-an-address/s06-06-write-audit-the-memo"},{"id":"analytics-studio/every-number-has-an-address/s07-07-checkpoint"}]},{"slug":"challenge-then-rerun","steps":[{"id":"analytics-studio/challenge-then-rerun/s01-01-the-confident-recommendation"},{"id":"analytics-studio/challenge-then-rerun/s02-02-the-triad-as-a-checklist"},{"id":"analytics-studio/challenge-then-rerun/s03-03-name-the-leg"},{"id":"analytics-studio/challenge-then-rerun/s04-04-predict-the-rerun"},{"id":"analytics-studio/challenge-then-rerun/s05-05-fix-the-drifting-memo"},{"id":"analytics-studio/challenge-then-rerun/s06-06-write-score-the-challenge"},{"id":"analytics-studio/challenge-then-rerun/s07-07-checkpoint"}]}]},{"slug":"delivery-studio","lessons":[{"slug":"the-spec-is-the-build","steps":[{"id":"delivery-studio/the-spec-is-the-build/s01-01-the-clarifying-question-is-gone"},{"id":"delivery-studio/the-spec-is-the-build/s02-02-the-ticket-as-data"},{"id":"delivery-studio/the-spec-is-the-build/s03-03-spot-the-untestable-check"},{"id":"delivery-studio/the-spec-is-the-build/s04-04-predict-the-spec-lint"},{"id":"delivery-studio/the-spec-is-the-build/s05-05-fill-the-scope-fence"},{"id":"delivery-studio/the-spec-is-the-build/s06-06-inside-or-outside-the-frontier"},{"id":"delivery-studio/the-spec-is-the-build/s07-07-write-lint-spec"},{"id":"delivery-studio/the-spec-is-the-build/s08-08-checkpoint"}]},{"slug":"acceptance-before-the-build","steps":[{"id":"delivery-studio/acceptance-before-the-build/s01-01-the-ninety-seven-thousand-dollar-skip"},{"id":"delivery-studio/acceptance-before-the-build/s02-02-evidence-against-the-checks"},{"id":"delivery-studio/acceptance-before-the-build/s03-03-which-check-catches-it"},{"id":"delivery-studio/acceptance-before-the-build/s04-04-fix-the-vibes-review"},{"id":"delivery-studio/acceptance-before-the-build/s05-05-predict-the-review-run"},{"id":"delivery-studio/acceptance-before-the-build/s06-06-rework-is-a-list-not-a-feeling"},{"id":"delivery-studio/acceptance-before-the-build/s07-07-write-review-deliverable"},{"id":"delivery-studio/acceptance-before-the-build/s08-08-checkpoint"}]},{"slug":"status-from-artifacts","steps":[{"id":"delivery-studio/status-from-artifacts/s01-01-watermelon-status"},{"id":"delivery-studio/status-from-artifacts/s02-02-derive-the-rag"},{"id":"delivery-studio/status-from-artifacts/s03-03-spot-the-artifact"},{"id":"delivery-studio/status-from-artifacts/s04-04-fill-the-worst-of"},{"id":"delivery-studio/status-from-artifacts/s05-05-predict-the-rollup"},{"id":"delivery-studio/status-from-artifacts/s06-06-the-not-ready-call"},{"id":"delivery-studio/status-from-artifacts/s07-07-write-readiness-gate"},{"id":"delivery-studio/status-from-artifacts/s08-08-checkpoint"}]},{"slug":"the-hours-saved-receipt","steps":[{"id":"delivery-studio/the-hours-saved-receipt/s01-01-the-token-chart-is-not-a-receipt"},{"id":"delivery-studio/the-hours-saved-receipt/s02-02-the-receipt-as-data"},{"id":"delivery-studio/the-hours-saved-receipt/s03-03-which-line-survives-staff-meeting"},{"id":"delivery-studio/the-hours-saved-receipt/s04-04-fix-the-vanity-chart"},{"id":"delivery-studio/the-hours-saved-receipt/s05-05-predict-the-receipt-run"},{"id":"delivery-studio/the-hours-saved-receipt/s06-06-only-accepted-ships-count"},{"id":"delivery-studio/the-hours-saved-receipt/s07-07-write-hours-receipt"},{"id":"delivery-studio/the-hours-saved-receipt/s08-08-checkpoint"}]}]},{"slug":"people-studio","lessons":[{"slug":"the-data-firewall","steps":[{"id":"people-studio/the-data-firewall/s01-01-the-rule-that-cannot-live-in-memory"},{"id":"people-studio/the-data-firewall/s02-02-the-firewall-as-data"},{"id":"people-studio/the-data-firewall/s03-03-which-payload-passes"},{"id":"people-studio/the-data-firewall/s04-04-predict-the-scan"},{"id":"people-studio/the-data-firewall/s05-05-fix-the-leaky-gate"},{"id":"people-studio/the-data-firewall/s06-06-employee-relations-takes-nothing"},{"id":"people-studio/the-data-firewall/s07-07-write-prompt-gate"},{"id":"people-studio/the-data-firewall/s08-08-checkpoint"}]},{"slug":"briefs-with-receipts","steps":[{"id":"people-studio/briefs-with-receipts/s01-01-the-most-automated-desk"},{"id":"people-studio/briefs-with-receipts/s02-02-a-claim-with-a-receipt"},{"id":"people-studio/briefs-with-receipts/s03-03-sourced-or-inference"},{"id":"people-studio/briefs-with-receipts/s04-04-predict-the-receipt-audit"},{"id":"people-studio/briefs-with-receipts/s05-05-fill-the-lookup"},{"id":"people-studio/briefs-with-receipts/s06-06-the-screening-line"},{"id":"people-studio/briefs-with-receipts/s07-07-write-audit-brief"},{"id":"people-studio/briefs-with-receipts/s08-08-checkpoint"}]},{"slug":"the-paper-trail","steps":[{"id":"people-studio/the-paper-trail/s01-01-policy-is-legal-surface"},{"id":"people-studio/the-paper-trail/s02-02-the-boundary-as-a-state-machine"},{"id":"people-studio/the-paper-trail/s03-03-who-signs-off"},{"id":"people-studio/the-paper-trail/s04-04-predict-the-version-log"},{"id":"people-studio/the-paper-trail/s05-05-fix-the-shortcut"},{"id":"people-studio/the-paper-trail/s06-06-the-evidence-trail"},{"id":"people-studio/the-paper-trail/s07-07-write-decision-record"},{"id":"people-studio/the-paper-trail/s08-08-checkpoint"}]}]},{"slug":"ops-studio","lessons":[{"slug":"the-sop-is-data","steps":[{"id":"ops-studio/the-sop-is-data/s01-01-the-two-weeks-notice-problem"},{"id":"ops-studio/the-sop-is-data/s02-02-the-sop-as-data"},{"id":"ops-studio/the-sop-is-data/s03-03-spot-the-invented-step"},{"id":"ops-studio/the-sop-is-data/s04-04-fill-the-sop-lint"},{"id":"ops-studio/the-sop-is-data/s05-05-predict-the-diff"},{"id":"ops-studio/the-sop-is-data/s06-06-the-diff-is-the-change-control"},{"id":"ops-studio/the-sop-is-data/s07-07-write-lint-sop"},{"id":"ops-studio/the-sop-is-data/s08-08-checkpoint"}]},{"slug":"auto-run-or-named-approval","steps":[{"id":"ops-studio/auto-run-or-named-approval/s01-01-the-checklist-that-lied"},{"id":"ops-studio/auto-run-or-named-approval/s02-02-the-schedule-is-data"},{"id":"ops-studio/auto-run-or-named-approval/s03-03-which-item-may-auto-run"},{"id":"ops-studio/auto-run-or-named-approval/s04-04-fix-the-silent-auto-complete"},{"id":"ops-studio/auto-run-or-named-approval/s05-05-predict-the-exception-queue"},{"id":"ops-studio/auto-run-or-named-approval/s06-06-exceptions-are-a-queue-not-a-shrug"},{"id":"ops-studio/auto-run-or-named-approval/s07-07-write-run-checklist"},{"id":"ops-studio/auto-run-or-named-approval/s08-08-checkpoint"}]},{"slug":"numbers-with-receipts","steps":[{"id":"ops-studio/numbers-with-receipts/s01-01-the-number-that-moves-money"},{"id":"ops-studio/numbers-with-receipts/s02-02-recompute-dont-trust"},{"id":"ops-studio/numbers-with-receipts/s03-03-which-claim-has-no-receipt"},{"id":"ops-studio/numbers-with-receipts/s04-04-predict-the-verification"},{"id":"ops-studio/numbers-with-receipts/s05-05-fill-the-receipt-check"},{"id":"ops-studio/numbers-with-receipts/s06-06-the-anomaly-is-a-question"},{"id":"ops-studio/numbers-with-receipts/s07-07-write-verify-report"},{"id":"ops-studio/numbers-with-receipts/s08-08-checkpoint"}]},{"slug":"the-pii-proxy","steps":[{"id":"ops-studio/the-pii-proxy/s01-01-the-ticket-that-left-the-building"},{"id":"ops-studio/the-pii-proxy/s02-02-mask-then-unmask"},{"id":"ops-studio/the-pii-proxy/s03-03-which-payload-may-leave"},{"id":"ops-studio/the-pii-proxy/s04-04-fill-the-leftover-check"},{"id":"ops-studio/the-pii-proxy/s05-05-predict-the-two-sides"},{"id":"ops-studio/the-pii-proxy/s06-06-unmask-is-a-named-human-act"},{"id":"ops-studio/the-pii-proxy/s07-07-write-the-proxy"},{"id":"ops-studio/the-pii-proxy/s08-08-checkpoint"}]}]},{"slug":"legal-studio","lessons":[{"slug":"the-cite-check-gate","steps":[{"id":"legal-studio/the-cite-check-gate/s01-01-the-brief-that-reads-perfectly"},{"id":"legal-studio/the-cite-check-gate/s02-02-three-verdicts-as-data"},{"id":"legal-studio/the-cite-check-gate/s03-03-which-cite-needs-no-check"},{"id":"legal-studio/the-cite-check-gate/s04-04-predict-the-verification-run"},{"id":"legal-studio/the-cite-check-gate/s05-05-fix-the-verifier-that-asks-again"},{"id":"legal-studio/the-cite-check-gate/s06-06-write-verify-authorities"},{"id":"legal-studio/the-cite-check-gate/s07-07-checkpoint"}]},{"slug":"clause-extraction-with-pinpoints","steps":[{"id":"legal-studio/clause-extraction-with-pinpoints/s01-01-a-claim-without-a-pinpoint-is-a-rumor"},{"id":"legal-studio/clause-extraction-with-pinpoints/s02-02-the-playbook-as-data"},{"id":"legal-studio/clause-extraction-with-pinpoints/s03-03-which-row-is-unusable"},{"id":"legal-studio/clause-extraction-with-pinpoints/s04-04-predict-the-triage"},{"id":"legal-studio/clause-extraction-with-pinpoints/s05-05-fill-the-walk-away-route"},{"id":"legal-studio/clause-extraction-with-pinpoints/s06-06-write-triage-extraction"},{"id":"legal-studio/clause-extraction-with-pinpoints/s07-07-checkpoint"}]},{"slug":"privilege-rules-and-the-review-log","steps":[{"id":"legal-studio/privilege-rules-and-the-review-log/s01-01-privilege-is-a-routing-question"},{"id":"legal-studio/privilege-rules-and-the-review-log/s02-02-the-routing-table"},{"id":"legal-studio/privilege-rules-and-the-review-log/s03-03-which-request-is-the-waiver-risk"},{"id":"legal-studio/privilege-rules-and-the-review-log/s04-04-predict-the-router"},{"id":"legal-studio/privilege-rules-and-the-review-log/s05-05-fix-the-privilege-hole"},{"id":"legal-studio/privilege-rules-and-the-review-log/s06-06-the-log-that-survives-a-motion"},{"id":"legal-studio/privilege-rules-and-the-review-log/s07-07-write-log-ready"},{"id":"legal-studio/privilege-rules-and-the-review-log/s08-08-checkpoint"}]}]},{"slug":"agent-studio","lessons":[{"slug":"the-tool-contract","steps":[{"id":"agent-studio/the-tool-contract/s01-01-the-feature-you-ship-today"},{"id":"agent-studio/the-tool-contract/s02-02-the-schema-is-the-blast-radius"},{"id":"agent-studio/the-tool-contract/s03-03-spot-the-loose-tool"},{"id":"agent-studio/the-tool-contract/s04-04-predict-the-validation"},{"id":"agent-studio/the-tool-contract/s05-05-fix-the-yolo-rerun"},{"id":"agent-studio/the-tool-contract/s06-06-write-validate-call"},{"id":"agent-studio/the-tool-contract/s07-07-checkpoint"}]},{"slug":"the-loop-with-a-meter","steps":[{"id":"agent-studio/the-loop-with-a-meter/s01-01-a-loop-that-cannot-stop-is-an-invoice"},{"id":"agent-studio/the-loop-with-a-meter/s02-02-caps-are-part-of-the-loop"},{"id":"agent-studio/the-loop-with-a-meter/s03-03-why-did-it-stop"},{"id":"agent-studio/the-loop-with-a-meter/s04-04-predict-the-stop-reason"},{"id":"agent-studio/the-loop-with-a-meter/s05-05-fill-the-budget-check"},{"id":"agent-studio/the-loop-with-a-meter/s06-06-the-trace-is-the-receipt"},{"id":"agent-studio/the-loop-with-a-meter/s07-07-write-run-loop"},{"id":"agent-studio/the-loop-with-a-meter/s08-08-checkpoint"}]},{"slug":"evals-before-it-works","steps":[{"id":"agent-studio/evals-before-it-works/s01-01-green-before-it-runs"},{"id":"agent-studio/evals-before-it-works/s02-02-a-golden-case-is-a-past-incident"},{"id":"agent-studio/evals-before-it-works/s03-03-which-case-catches-it"},{"id":"agent-studio/evals-before-it-works/s04-04-predict-the-eval-run"},{"id":"agent-studio/evals-before-it-works/s05-05-fix-the-router-to-green"},{"id":"agent-studio/evals-before-it-works/s06-06-the-ratchet-entry"},{"id":"agent-studio/evals-before-it-works/s07-07-write-lint-the-ratchet"},{"id":"agent-studio/evals-before-it-works/s08-08-checkpoint"}]}]}]},"detail":{"number":21,"slug":"evals","title":"test the agent before you trust it","blurb":"same input, check the output, pass or fail. don't ship on a demo that felt right.","overview":"$27","status":"live","lessons":[{"slug":"check-before-you-trust","title":"Check before you trust","estMinutes":8,"prerequisites":[],"status":"live","steps":[{"type":"read","id":"evals/check-before-you-trust/s01-01-intro","xp":2,"hint":[],"personalize":false,"phase":"warmup","estSeconds":300,"concept":"check-before-you-trust","body":"$28","cta":"Got it","runnable":true}],"xpTotal":2},{"slug":"writing-evals","title":"Assertions on AI output, not vibes","estMinutes":9,"prerequisites":[],"status":"live","steps":[{"type":"read","id":"evals/writing-evals/s01-01-intro","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":110,"concept":"eval-mindset","body":"$29","cta":"Got it","code":"# the canonical eval shape — input, expected, actual, pass/fail.\ncases = [\n {\"name\": \"extract_email_simple\",\n \"input\": \"ping me at sam@example.com\",\n \"expected\": \"sam@example.com\"},\n {\"name\": \"extract_email_with_punct\",\n \"input\": \"(maya@example.com)\",\n \"expected\": \"maya@example.com\"},\n]\n\n# a tiny extractor so the eval loop is visible. run the live\n# version in your agent when you want a model to do this.\ndef fake_extract(text):\n import re\n m = re.search(r\"[\\w.+-]+@[\\w-]+\\.[\\w.-]+\", text)\n return m.group(0) if m else None\n\npassed = 0\nfor c in cases:\n actual = fake_extract(c[\"input\"])\n ok = actual == c[\"expected\"]\n passed += ok\n print(f\"{c['name']}: {'PASS' if ok else 'FAIL'}\")\nprint(f\"{passed}/{len(cases)}\")\n","runnable":true},{"type":"mc","id":"evals/writing-evals/s02-02-which-eval-passes","xp":2,"hint":[{"level":1,"body":"`actual` is a full sentence. Which check is True for any sentence that contains the word `Paris`?","cost":0},{"level":2,"body":"`\"Paris\" in actual` is the substring check — it passes whenever the answer is somewhere in the output. That's the most robust pattern for free-form AI output.","cost":0}],"personalize":false,"phase":"warmup","estSeconds":50,"concept":"assertion-styles","prompt":"Cursor wrote three evals against the same string output. Two are\nFalse, one is True. Which assertion *passes*?\n","options":[{"id":"a","label":"`e1` — exact equality with `\"Paris\"`","explain":"False — `actual` is the full sentence, not just the word `Paris`."},{"id":"b","label":"`e2` — `\"Paris\" in actual`"},{"id":"c","label":"`e3` — `actual.startswith(\"Paris\")`","explain":"False — the sentence starts with `The`, not `Paris`."},{"id":"d","label":"None of them pass.","explain":"One of them passes. Look at the substring check."}],"answerIds":["b"],"shuffle":false,"code":"actual = \"The capital of France is Paris.\"\n\n# three candidate evals, each checking a different way.\ne1 = actual == \"Paris\"\ne2 = \"Paris\" in actual\ne3 = actual.startswith(\"Paris\")\n\nprint(e1, e2, e3)\n","runnable":true},{"type":"read","id":"evals/writing-evals/s03-03-eval-patterns","xp":1,"hint":[],"personalize":false,"phase":"build","estSeconds":130,"concept":"eval-patterns","body":"$2a","cta":"Got it","code":"# four eval patterns, in order from strict to loose.\nimport json\n\n# pattern 1: exact match\ndef eval_exact(actual, expected):\n return actual == expected\n\n# pattern 2: substring\ndef eval_contains(actual, expected_substring):\n return expected_substring.lower() in actual.lower()\n\n# pattern 3: shape check on JSON\ndef eval_shape(actual_str, required_keys):\n try:\n data = json.loads(actual_str)\n except json.JSONDecodeError:\n return False\n return all(k in data for k in required_keys)\n\n# pattern 4: regex (still rule-based, just more flexible)\nimport re\ndef eval_regex(actual, pattern):\n return re.search(pattern, actual) is not None\n\nprint(eval_exact(\"yes\", \"yes\"))\nprint(eval_contains(\"The answer is 42.\", \"42\"))\nprint(eval_shape('{\"name\":\"maya\",\"score\":7}', [\"name\", \"score\"]))\nprint(eval_regex(\"order 1234\", r\"order \\d+\"))\n","runnable":true},{"type":"predict","id":"evals/writing-evals/s04-04-predict-the-pass-rate","xp":3,"hint":[{"level":1,"body":"Lowercase both sides, then check substring. Cases 1, 2, and 4 all contain `paris` after lowercasing. Case 3 has `lyon`.","cost":0},{"level":2,"body":"Three pass, one fails. Output is `3/4`.","cost":0}],"personalize":false,"phase":"build","estSeconds":70,"concept":"pass-rate-counting","code":"cases = [\n {\"actual\": \"Paris\", \"expected\": \"Paris\"},\n {\"actual\": \"The capital is Paris.\",\"expected\": \"Paris\"},\n {\"actual\": \"Lyon\", \"expected\": \"Paris\"},\n {\"actual\": \"PARIS\", \"expected\": \"Paris\"},\n]\n\npassed = 0\nfor c in cases:\n if c[\"expected\"].lower() in c[\"actual\"].lower():\n passed += 1\n\nprint(f\"{passed}/{len(cases)}\")\n","prompt":"Read the eval loop. It uses the case-insensitive\n*contains* check. How many of the four cases pass?\n","grader":{"kind":"stdout-equality","expected":"3/4","normalize":"collapse-trailing-newline"}},{"type":"fill","id":"evals/writing-evals/s05-05-fill-the-assertion","xp":3,"hint":[{"level":1,"body":"It's the Python keyword that raises `AssertionError` when the expression is False.","cost":0},{"level":2,"body":"It's `assert`. The line becomes `assert actual == \"sam@example.com\"`.","cost":0}],"personalize":false,"phase":"build","estSeconds":60,"concept":"assert-keyword","prompt":"Cursor wrote a pytest-style eval, but the assertion line is missing\nthe keyword that turns a comparison into a passing or failing test.\nFill in the keyword.\n\nKeyword: ___\n","code":"def fake_extract(text):\n return \"sam@example.com\"\n\ndef test_extract_email():\n actual = fake_extract(\"ping me at sam@example.com\")\n ___ actual == \"sam@example.com\"\n\ntest_extract_email()\nprint(\"ok\")\n","blanks":[{"id":"keyword","accept":["assert"],"caseSensitive":true,"normalize":"trim"}]},{"type":"fix","id":"evals/writing-evals/s06-06-fix-the-flaky-eval","xp":4,"hint":[{"level":1,"body":"AI output capitalization is unreliable. Lowercase both sides before comparing.","cost":0},{"level":2,"body":"Change line 4 to `ok = actual.lower() == expected.lower()`.","cost":0}],"personalize":false,"phase":"build","estSeconds":90,"concept":"case-insensitive-eval","brokenCode":"expected = \"yes\"\nactual = \"YES\"\n\nok = actual == expected\nprint(\"pass\" if ok else \"fail\")\n","prompt":"Claude wrote a yes/no classifier eval. The `actual` answer is `\"YES\"`\nbut the eval expects `\"yes\"` and fails. The fix is to make the\ncomparison case-insensitive. Change line 4 so the script prints\n`pass`.\n\nExpected output:\n```\npass\n```\n","grader":{"kind":"stdout-equality","expected":"pass","normalize":"collapse-trailing-newline"},"bugLines":[4],"revealAfter":4},{"type":"fix","id":"evals/writing-evals/s07-07-fix-the-overstrict-eval","xp":4,"hint":[{"level":1,"body":"Exact equality is too strict for prose. Use `in` for a substring check.","cost":0},{"level":2,"body":"Change line 4 to `ok = expected in actual`. That checks whether `Paris` appears anywhere in the sentence.","cost":0}],"personalize":false,"phase":"build","estSeconds":95,"concept":"substring-eval","brokenCode":"expected = \"Paris\"\nactual = \"The capital of France is Paris.\"\n\nok = actual == expected\nprint(\"pass\" if ok else \"fail\")\n","prompt":"Cursor wrote an eval that checks whether the model mentioned `Paris`\nsomewhere in its answer. It used exact equality, which fails because\nthe model returned a full sentence. Switch the check to a *contains*\ncomparison so it passes.\n\nExpected output:\n```\npass\n```\n","grader":{"kind":"stdout-equality","expected":"pass","normalize":"collapse-trailing-newline"},"bugLines":[4],"revealAfter":4},{"type":"write","id":"evals/writing-evals/s08-08-write-the-suite","xp":5,"hint":[{"level":1,"body":"Loop over cases, call `classify` on each `input`, compare to `expected` (case-insensitive), count passes.","cost":0},{"level":2,"body":"Initialize `passed = 0`. In the loop, increment when `actual.lower() == c[\"expected\"].lower()`. Return `(passed, len(cases))`.","cost":0}],"personalize":false,"phase":"build","estSeconds":140,"concept":"build-eval-suite","prompt":"Build a small eval runner. The starter has a list of cases (each with\n`name`, `input`, and `expected`) and a stub `classify(text)` that\nreturns `\"yes\"` or `\"no\"` based on whether the text contains\n`\"cancel\"`. Write a function `run_suite(cases)` that:\n\n- Calls `classify(case[\"input\"])` for each case.\n- Compares the lowercased actual to the lowercased expected.\n- Returns a tuple `(passed, total)`.\n\nThen call `run_suite(cases)` and print the result as\n`passed/total`.\n\nExpected output:\n```\n3/3\n```\n","starter":"cases = [\n {\"name\": \"explicit_cancel\",\n \"input\": \"please cancel my order\",\n \"expected\": \"yes\"},\n {\"name\": \"no_intent\",\n \"input\": \"what's the status of my order\",\n \"expected\": \"no\"},\n {\"name\": \"shouty_cancel\",\n \"input\": \"CANCEL THIS NOW\",\n \"expected\": \"yes\"},\n]\n\ndef classify(text):\n return \"yes\" if \"cancel\" in text.lower() else \"no\"\n\n# define run_suite(cases) below\n\npassed, total = run_suite(cases)\nprint(f\"{passed}/{total}\")\n","grader":{"kind":"stdout-equality","expected":"3/3","normalize":"collapse-trailing-newline"},"solution":"cases = [\n {\"name\": \"explicit_cancel\",\n \"input\": \"please cancel my order\",\n \"expected\": \"yes\"},\n {\"name\": \"no_intent\",\n \"input\": \"what's the status of my order\",\n \"expected\": \"no\"},\n {\"name\": \"shouty_cancel\",\n \"input\": \"CANCEL THIS NOW\",\n \"expected\": \"yes\"},\n]\n\ndef classify(text):\n return \"yes\" if \"cancel\" in text.lower() else \"no\"\n\ndef run_suite(cases):\n passed = 0\n for c in cases:\n actual = classify(c[\"input\"])\n if actual.lower() == c[\"expected\"].lower():\n passed += 1\n return passed, len(cases)\n\npassed, total = run_suite(cases)\nprint(f\"{passed}/{total}\")\n","hiddenTests":[]},{"type":"checkpoint","id":"evals/writing-evals/s09-09-checkpoint","xp":8,"hint":[],"personalize":false,"phase":"check","estSeconds":130,"concept":"regression-eval","prompt":"Last one. The starter has a `regression_cases` list and a stub\nclassifier that has a known bug — it returns `\"no\"` for the input\n`\"cancel my account\"`, which should be `\"yes\"`. Add the missing case\nto `regression_cases` so it would catch this exact bug. Run the suite\nand confirm the output is `2/3` (two pass, one — the new regression\ncase — fails).\n\nAdd one entry: input `\"cancel my account\"`, expected `\"yes\"`, name\n`\"account_cancel\"`.\n\nExpected output:\n```\n2/3\n```\n","starter":"def classify(text):\n # buggy: misses \"account\" cancel intent\n if \"order\" in text.lower() and \"cancel\" in text.lower():\n return \"yes\"\n return \"no\"\n\nregression_cases = [\n {\"name\": \"explicit_cancel\",\n \"input\": \"please cancel my order\",\n \"expected\": \"yes\"},\n {\"name\": \"no_intent\",\n \"input\": \"what's the status of my order\",\n \"expected\": \"no\"},\n # add the regression case here\n]\n\ndef run_suite(cases):\n passed = 0\n for c in cases:\n if classify(c[\"input\"]).lower() == c[\"expected\"].lower():\n passed += 1\n return passed, len(cases)\n\np, t = run_suite(regression_cases)\nprint(f\"{p}/{t}\")\n","grader":{"kind":"stdout-equality","expected":"2/3","normalize":"collapse-trailing-newline"},"solution":"def classify(text):\n if \"order\" in text.lower() and \"cancel\" in text.lower():\n return \"yes\"\n return \"no\"\n\nregression_cases = [\n {\"name\": \"explicit_cancel\",\n \"input\": \"please cancel my order\",\n \"expected\": \"yes\"},\n {\"name\": \"no_intent\",\n \"input\": \"what's the status of my order\",\n \"expected\": \"no\"},\n {\"name\": \"account_cancel\",\n \"input\": \"cancel my account\",\n \"expected\": \"yes\"},\n]\n\ndef run_suite(cases):\n passed = 0\n for c in cases:\n if classify(c[\"input\"]).lower() == c[\"expected\"].lower():\n passed += 1\n return passed, len(cases)\n\np, t = run_suite(regression_cases)\nprint(f\"{p}/{t}\")\n","passThreshold":0.66}],"xpTotal":31},{"slug":"llm-as-judge","title":"LLM-as-judge — when the judge is another model","estMinutes":12,"prerequisites":["writing-evals"],"status":"live","steps":[{"type":"read","id":"evals/llm-as-judge/s01-01-intro","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":120,"concept":"llm-as-judge-pattern","body":"$2b","cta":"Got it","code":"# the judge is just a model with a critique prompt and a binary verdict.\n\ndef judge(question, answer, rubric):\n # in real code: a model call constrained to {\"passed\": bool, \"critique\": str}.\n # here: a deterministic mock that returns canned verdicts.\n if \"Paris\" in answer and len(answer) < 50:\n return {\"passed\": True, \"critique\": \"Concise and correct.\"}\n if \"Paris\" in answer:\n return {\"passed\": True, \"critique\": \"Correct but verbose.\"}\n return {\"passed\": False, \"critique\": \"Did not name Paris.\"}\n\ncases = [\n (\"What's the capital of France?\", \"Paris.\"),\n (\"What's the capital of France?\", \"Paris is the capital, located in northern France.\"),\n (\"What's the capital of France?\", \"I think maybe Lyon?\"),\n]\nfor q, a in cases:\n v = judge(q, a, \"Reward correct, concise answers.\")\n print(f\"{a!r}: pass={v['passed']} — {v['critique']}\")\n","runnable":true},{"type":"mc","id":"evals/llm-as-judge/s02-02-pairwise-or-rubric","xp":2,"hint":[{"level":1,"body":"Pairwise needs *two outputs to compare* and returns a relative winner. Which scenarios actually have two outputs to compare?","cost":0},{"level":2,"body":"Only C has two prompt drafts to compare. A, B, D all need absolute pass/fail or scores.","cost":0}],"personalize":false,"phase":"warmup","estSeconds":60,"concept":"pairwise-vs-rubric-fit","prompt":"Pairwise asks \"A or B better?\" → returns A/B/tie. Rubric asks\n\"does this output pass?\" → returns pass/fail. Which TWO scenarios\nabove are the strongest fit for **pairwise**?\n","options":[{"id":"a_and_d","label":"A (regression) and D (production gate)","explain":"Both need an absolute quality floor — 'pass or fail per case' — and run on every input. Pairwise gives you a relative winner, not a floor."},{"id":"b_and_c","label":"B (daily dashboard) and C (A/B prompt test)","explain":"B needs an absolute number for a trend line. Pairwise gives you 'A vs B,' not 'how good is today.'"},{"id":"a_and_c","label":"A (regression) and C (A/B prompt test)","explain":"A needs a per-case pass/fail to gate CI. Pairwise can't say 'this case got worse' — only 'this output beat that output.'"},{"id":"c_and_d","label":"C (A/B prompt test) and D (production gate)","explain":"D needs an absolute floor per output. Pairwise compares two outputs, doesn't grade one."},{"id":"c_only_paired_correctly","label":"C only — the others all fit rubric"},{"id":"pairwise_for_c_and_b_dashboard","label":"Both A/B testing scenarios — C is the only A/B test here","explain":"B is a dashboard, not an A/B test."}],"answerIds":["c_only_paired_correctly"],"shuffle":false,"code":"# four eval scenarios. picking the wrong shape costs you reliability.\nscenarios = {\n \"A\": \"Regression eval: same prompt v1 vs v2, run on 50 fixed cases, fail CI on regression.\",\n \"B\": \"Daily quality dashboard: one number per day showing 'how well is the model doing'.\",\n \"C\": \"A/B prompt test: which of two prompt drafts produces better support replies on a 30-case set?\",\n \"D\": \"Production gate: every output must meet a minimum quality bar before being shown to users.\",\n}\nfor k, v in scenarios.items():\n print(f\"{k}: {v}\")\n","runnable":true},{"type":"read","id":"evals/llm-as-judge/s03-03-the-four-biases","xp":2,"hint":[],"personalize":false,"phase":"build","estSeconds":110,"concept":"judge-bias-taxonomy","body":"$2c","cta":"Got it","code":"# the bias hall of fame, named with their measured effect sizes.\n\nBIASES = {\n \"position\": \"Judge prefers whichever response appears first.\",\n \"length\": \"Judge prefers longer outputs even when shorter is correct.\",\n \"self_preference\": \"Judge favors outputs from its own model family.\",\n \"style\": \"Judge over-weights confident, well-formatted prose.\",\n}\n\n# source-tagged effect sizes from the literature:\nEFFECT_SIZES = {\n \"position\": \"GPT-4 flips ~35% of pairs on order swap (Zheng et al. 2023)\",\n \"length\": \"Documented in MT-Bench paper; 'repetitive list attack' wins.\",\n \"self_pref\": \"GPT-4 favors GPT-4 outputs; Claude favors Claude (Zheng).\",\n \"style\": \"Effect 0.76-0.92 — dwarfs position bias (Soumik 2026 preprint).\",\n}\n\nfor name, desc in BIASES.items():\n print(f\"{name}: {desc}\")\n","runnable":true},{"type":"predict","id":"evals/llm-as-judge/s04-04-predict-the-bias","xp":3,"hint":[{"level":1,"body":"The judge always returns \"A\". So v1==\"A\" AND v2==\"A\". The `consistent` check requires `(v1==\"A\" AND v2==\"B\")` OR `(v1==\"B\" AND v2==\"A\")` — i.e., one slot wins round 1 and the OTHER slot wins round 2. Both being \"A\" means neither branch matches.","cost":0},{"level":2,"body":"`consistent? False` — bias detected. The both-orders mitigation just caught a judge that ignores content entirely.","cost":0}],"personalize":false,"phase":"build","estSeconds":80,"concept":"position-bias-detection","code":"# a position-biased mock judge: always picks whatever sits in slot A,\n# regardless of which physical output is in there.\n\ndef fake_judge_pairwise(input_text, a, b):\n return \"A\" # bias: always picks slot A, ignores content\n\nX = \"Output X (correct, concise)\"\nY = \"Output Y (verbose, partially wrong)\"\n\n# round 1: X in slot A, Y in slot B\nv1 = fake_judge_pairwise(\"Which is better?\", X, Y)\n# round 2: Y in slot A, X in slot B\nv2 = fake_judge_pairwise(\"Which is better?\", Y, X)\n\n# consistent? if the SAME physical output won both rounds, yes.\n# round 1 winner=A means X won. round 2 winner=A means Y won.\n# different physical outputs won → not consistent → bias detected.\nconsistent = (v1 == \"A\" and v2 == \"B\") or (v1 == \"B\" and v2 == \"A\")\n\nprint(f\"order 1: winner={v1}\")\nprint(f\"order 2: winner={v2}\")\nprint(f\"consistent? {consistent}\")\n","prompt":"Read the editor. The fake judge has pure position bias — it always\npicks slot A, no matter what content sits there. We run the same\npair in both orders, then check whether the same physical output\nwon both times.\n\nWhat does the script print on the third line?\n","grader":{"kind":"stdout-equality","expected":"order 1: winner=A\norder 2: winner=A\nconsistent? False\n","normalize":"collapse-trailing-newline"}},{"type":"fill","id":"evals/llm-as-judge/s05-05-fill-the-rubric-call","xp":3,"hint":[{"level":1,"body":"The verdict dict has a `\"passed\"` key whose value is `True` or `False`. Read that field directly — no comparison against a number.","cost":0},{"level":2,"body":"`verdict[\"passed\"]` — the boolean is the whole answer. No `>= 4` threshold, no `== \"PASS\"` string compare.","cost":0}],"personalize":false,"phase":"build","estSeconds":65,"concept":"rubric-judge-binary-verdict","prompt":"The judge is a function `judge_rubric(question, answer, rubric)`\nthat returns a dict like `{\"passed\": True, \"critique\": \"...\"}`.\nYour code needs to read the verdict and branch on the binary\n`passed` field — never on a numeric score.\n\nFill in the missing condition so the loop prints `pass` for\npassing cases and `fail` for failing ones.\n\nExpected output:\n```\npass\nfail\n```\n","code":"def judge_rubric(question, answer, rubric):\n if \"Paris\" in answer:\n return {\"passed\": True, \"critique\": \"Names Paris.\"}\n return {\"passed\": False, \"critique\": \"Does not name Paris.\"}\n\ncases = [\n (\"Paris.\"),\n (\"Lyon, I think.\"),\n]\nrubric = \"Reward correct city name.\"\nfor answer in cases:\n verdict = judge_rubric(\"capital of France?\", answer, rubric)\n if ___:\n print(\"pass\")\n else:\n print(\"fail\")\n","blanks":[{"id":"pass_check","accept":["verdict[\"passed\"]","verdict['passed']","verdict.get(\"passed\")","verdict.get('passed')"],"caseSensitive":true,"normalize":"trim"}]},{"type":"fix","id":"evals/llm-as-judge/s06-06-fix-the-likert-trap","xp":4,"hint":[{"level":1,"body":"Read `v[\"passed\"]` instead of computing a threshold over `v[\"score\"]`. The binary field is what the rubric is calibrated against; the score is incidental.","cost":0},{"level":2,"body":"Change line 8 to:\n\n```python\nship = v[\"passed\"]\n```\n","cost":0}],"personalize":false,"phase":"build","estSeconds":100,"concept":"binary-not-likert","brokenCode":"verdicts = [\n {\"score\": 4, \"passed\": False, \"critique\": \"Well-written but factually wrong.\"},\n {\"score\": 3, \"passed\": True, \"critique\": \"Correct, plain prose.\"},\n]\n\nfor i, v in enumerate(verdicts, start=1):\n # bug: numeric threshold ignores the binary verdict the rubric was designed for.\n ship = v[\"score\"] >= 4\n print(f\"case {i}: ship={ship}\")\n","prompt":"The judge returns both a 1-5 numeric score AND a binary `passed`\nflag (some teams emit both during transition). The code is using\nthe score with a hardcoded threshold — `score >= 4` — but the\nrubric was designed for binary pass/fail and the 1-5 axis is\nuncalibrated. As Hamel Husain puts it: averaging Likert scores\nproduces false precision.\n\nTwo cases come in: one where the model judged 4/5 with `passed: False`\n(the answer is technically wrong but well-written — style bias),\nand one where the model judged 3/5 with `passed: True` (the answer\nis correct but plain).\n\nFix the gate so it reads the binary verdict instead of the numeric\nscore.\n\nExpected output:\n```\ncase 1: ship=False\ncase 2: ship=True\n```\n","grader":{"kind":"stdout-equality","expected":"case 1: ship=False\ncase 2: ship=True\n","normalize":"collapse-trailing-newline"},"bugLines":[8],"revealAfter":4},{"type":"fix","id":"evals/llm-as-judge/s07-07-fix-the-position-bias","xp":4,"hint":[{"level":1,"body":"Run the judge twice: once as `(question, output_x, output_y)`, once as `(question, output_y, output_x)`. If both rounds pick the same physical output, that's a real winner. If they disagree, return `\"tie\"`.","cost":0},{"level":2,"body":"Replace line 11 with the both-orders mitigation:\n\n```python\nv1 = biased_judge(question, output_x, output_y) # X in A, Y in B\nv2 = biased_judge(question, output_y, output_x) # Y in A, X in B\nx_won_both = (v1 == \"A\" and v2 == \"B\")\ny_won_both = (v1 == \"B\" and v2 == \"A\")\nif x_won_both: result = \"X\"\nelif y_won_both: result = \"Y\"\nelse: result = \"tie\"\n```\n","cost":0}],"personalize":false,"phase":"build","estSeconds":110,"concept":"both-orders-mitigation","brokenCode":"# the judge is biased toward whichever option sits in slot A.\ndef biased_judge(question, a, b):\n return \"A\" # always picks A, regardless of content\n\nquestion = \"Which output is better?\"\noutput_x = \"Output X\"\noutput_y = \"Output Y\"\n\n# bug: only one order tested, so we ship whatever sits in slot A.\nwinner = biased_judge(question, output_x, output_y)\nresult = \"X\" if winner == \"A\" else \"Y\"\nprint(f\"result: {result}\")\n","prompt":"The pairwise judge is calling itself only ONCE per pair — order\n`(a, b)`. Position bias means whichever output sits in slot `A`\ngets ~35% more votes than it deserves on close pairs.\n\nFix the code to run BOTH orders and only declare a winner when the\njudge is *consistent* (same physical output wins regardless of\nposition). On disagreement, return `\"tie\"`.\n\nExpected output:\n```\nresult: tie\n```\n","grader":{"kind":"stdout-equality","expected":"result: tie","normalize":"collapse-trailing-newline"},"bugLines":[11],"revealAfter":5},{"type":"write","id":"evals/llm-as-judge/s08-08-write-the-rubric-judge","xp":5,"hint":[{"level":1,"body":"Loop over cases, call the judge, increment a counter on `v[\"passed\"]`. After the loop compute `total - passed` for failures and `passed / total` for the rate.","cost":0},{"level":2,"body":"```python\ndef run_judge_suite(cases):\n passed_count = 0\n for c in cases:\n v = judge_rubric(c[\"question\"], c[\"answer\"], c[\"rubric\"])\n if v[\"passed\"]:\n passed_count += 1\n total = len(cases)\n return {\n \"total\": total,\n \"passed\": passed_count,\n \"failed\": total - passed_count,\n \"pass_rate\": round(passed_count / total, 2),\n }\n```\n","cost":0}],"personalize":false,"phase":"build","estSeconds":180,"concept":"rubric-judge-suite","prompt":"Build a rubric-style eval suite. Write `run_judge_suite(cases)` that:\n\n- Takes a list of dicts, each shaped\n `{\"question\": str, \"answer\": str, \"rubric\": str}`.\n- For each case, calls `judge_rubric(question, answer, rubric)`.\n The judge returns `{\"passed\": bool, \"critique\": str}`.\n- Counts how many cases pass.\n- Returns a dict `{\"total\": , \"passed\": , \"failed\": , \"pass_rate\": <0.0-1.0 rounded to 2 places>}`.\n\nThe script will run a 4-case suite. Expected output:\n```\ntotal=4 passed=2 failed=2 pass_rate=0.5\n```\n","starter":"def judge_rubric(question, answer, rubric):\n # mock judge: passes if \"Paris\" appears in answer.\n if \"Paris\" in answer:\n return {\"passed\": True, \"critique\": \"Names Paris.\"}\n return {\"passed\": False, \"critique\": \"Does not name Paris.\"}\n\ncases = [\n {\"question\": \"capital of France?\", \"answer\": \"Paris.\", \"rubric\": \"Reward correct answer.\"},\n {\"question\": \"capital of France?\", \"answer\": \"Lyon, I think.\", \"rubric\": \"Reward correct answer.\"},\n {\"question\": \"capital of France?\", \"answer\": \"Paris, in northern France.\", \"rubric\": \"Reward correct, concise answer.\"},\n {\"question\": \"capital of France?\", \"answer\": \"Some city in the south, maybe.\", \"rubric\": \"Reward correct answer.\"},\n]\n\n# define run_judge_suite(cases) below\n\nresult = run_judge_suite(cases)\nprint(f\"total={result['total']} passed={result['passed']} failed={result['failed']} pass_rate={result['pass_rate']}\")\n","grader":{"kind":"stdout-equality","expected":"total=4 passed=2 failed=2 pass_rate=0.5","normalize":"collapse-trailing-newline"},"solution":"$2d","hiddenTests":[]},{"type":"checkpoint","id":"evals/llm-as-judge/s09-09-checkpoint","xp":8,"hint":[],"personalize":false,"phase":"check","estSeconds":220,"concept":"position-bias-aware-pairwise","prompt":"Final drill. Build a pairwise judge that detects its OWN position\nbias and returns `tie` when it disagrees with itself across orders.\n\nWrite `pairwise(question, output_a, output_b)` that:\n\n- Calls `fake_judge(question, first, second)` twice:\n - Once with `first=output_a, second=output_b` — read result as\n \"the output named output_a sits in slot A this round.\"\n - Once with `first=output_b, second=output_a` — output_b is now\n in slot A.\n- Each call returns `\"A\"` or `\"B\"` (the *slot* that won).\n- Translate slot wins back to which physical output won that round.\n- If the SAME physical output won both rounds, return that output's\n label (`\"a\"` or `\"b\"`).\n- If the rounds disagreed, return `\"tie\"`.\n\nThen run a multi-case suite. Expected output:\n```\ncase 1: a wins (consistent)\ncase 2: tie (position-biased — judge disagrees with itself)\ncase 3: b wins (consistent)\n```\n","starter":"# the fake judge has a known length bias: always picks slot with the longer string.\ndef fake_judge(question, first, second):\n return \"A\" if len(first) > len(second) else \"B\"\n\n# define pairwise(question, output_a, output_b) below\n\ncases = [\n # case 1: output_a is genuinely longer → wins both orders → \"a wins\"\n (\"Pick the better answer.\", \"A long detailed correct answer here.\", \"Short.\"),\n # case 2: equal length → judge fluctuates → \"tie\"\n (\"Pick the better answer.\", \"Same exact length here\", \"Same exact length here\"),\n # case 3: output_b is genuinely longer → wins both orders → \"b wins\"\n (\"Pick the better answer.\", \"Brief.\", \"A long detailed correct answer here.\"),\n]\n\nfor i, (q, a, b) in enumerate(cases, start=1):\n result = pairwise(q, a, b)\n label = \"a wins (consistent)\" if result == \"a\" else \\\n \"b wins (consistent)\" if result == \"b\" else \\\n \"tie (position-biased — judge disagrees with itself)\"\n print(f\"case {i}: {label}\")\n","grader":{"kind":"stdout-equality","expected":"case 1: a wins (consistent)\ncase 2: tie (position-biased — judge disagrees with itself)\ncase 3: b wins (consistent)\n","normalize":"collapse-trailing-newline"},"solution":"$2e","passThreshold":0.66}],"xpTotal":32},{"slug":"the-rise-of-evals-as-a-discipline","title":"How evals went from research curiosity to the only thing that ships — a five-year history","estMinutes":20,"prerequisites":["llm-as-judge"],"status":"live","steps":[{"type":"read","id":"evals/the-rise-of-evals-as-a-discipline/s01-01-the-prompt-engineering-era","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":220,"concept":"prompt-engineering-era","body":"$2f","cta":"Got it","runnable":true},{"type":"read","id":"evals/the-rise-of-evals-as-a-discipline/s02-02-the-eval-turn","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":220,"concept":"the-eval-turn","body":"$30","cta":"Got it","runnable":true},{"type":"mc","id":"evals/the-rise-of-evals-as-a-discipline/s03-03-which-company-survives-a-model-swap","xp":2,"hint":[{"level":1,"body":"The question 'does my product still work after a model swap?' is only answerable if you can run a known set of inputs through the new model and compare to known-good outputs. That's an eval suite. Only one company has one.","cost":0}],"personalize":false,"phase":"build","estSeconds":70,"concept":"model-swap-survival","prompt":"Tomorrow, GPT-5 ships. Every team's existing prompts will behave\ndifferently — some better, some worse, some catastrophically wrong\non inputs they used to handle.\n\nWhich company still has a working product on day 2 with high\nconfidence?\n","options":[{"id":"a","label":"A — VibeCo (founder tunes by feel)","explain":"No test cases means no way to detect that GPT-5 broke anything until customers complain. The vibes era's exact failure mode."},{"id":"b","label":"B — TestsBeforePromptCo (200 cases in CI)"},{"id":"c","label":"C — DemoDrivenCo (three README examples)","explain":"Three hand-picked inputs is better than zero, but it's still demo-driven evals. The cases that break on a model swap are almost never the ones in your README — they're the long-tail inputs your customers actually send."},{"id":"d","label":"D — PromptLibraryCo (60 prompts, no programmatic check)","explain":"Lots of prompts but no way to verify them. A library of unverified prompts is just a bigger pile of vibes. Quantity doesn't replace measurement."}],"answerIds":["b"],"shuffle":false,"code":"# four companies. each has shipped an LLM-powered feature.\n# GPT-5 launches tomorrow. which one still has a working product on day 2?\ncompanies = {\n \"A\": {\n \"name\": \"VibeCo\",\n \"process\": \"founder tunes the prompt by hand, ships when it 'feels right', no test cases on disk\",\n },\n \"B\": {\n \"name\": \"TestsBeforePromptCo\",\n \"process\": \"200 input/expected pairs in a YAML file; CI runs them on every prompt change AND every model version; release blocked on regressions\",\n },\n \"C\": {\n \"name\": \"DemoDrivenCo\",\n \"process\": \"three hand-picked example inputs in the README; runs them manually before every deploy\",\n },\n \"D\": {\n \"name\": \"PromptLibraryCo\",\n \"process\": \"60 prompts in a notion doc, color-coded by which model they were tuned for; no programmatic check\",\n },\n}\nfor k, v in companies.items():\n print(f\"{k}: {v['name']:25} — {v['process']}\")\n","runnable":true},{"type":"read","id":"evals/the-rise-of-evals-as-a-discipline/s04-04-case-studies","xp":1,"hint":[],"personalize":false,"phase":"build","estSeconds":220,"concept":"eval-case-studies","body":"$31","cta":"Got it","runnable":true},{"type":"mc","id":"evals/the-rise-of-evals-as-a-discipline/s05-05-eval-first-vs-prompt-first","xp":2,"hint":[{"level":1,"body":"Eval-first means the cases that define 'success' exist BEFORE the prompt that's supposed to produce success. The prompt is written to pass the evals, not the other way around. Only one flow does this in that order.","cost":0}],"personalize":false,"phase":"build","estSeconds":70,"concept":"eval-first-vs-prompt-first","prompt":"Anthropic's \"Building Effective Agents\" mantra: **evals come first,\nprompts come second**. Which of these four workflows is genuinely\neval-first?\n","options":[{"id":"a","label":"A — write prompt, then write evals 'eventually'","explain":"'Evals eventually' means evals never. By the time the prompt is shipped, the team has moved on and the evals never get written. This is the vibes era with extra steps."},{"id":"b","label":"B — write 30 cases first, then write the prompt to pass them"},{"id":"c","label":"C — ship, wait for complaints, add eval per complaint","explain":"Reactive eval growth is better than no evals, but it's still prompt-first. The customer is your QA loop. This is the Klarna pattern."},{"id":"d","label":"D — let GPT-4 grade its own output, no ground truth on disk","explain":"This is the tautological-eval trap from @@pd-path:path.ref:evals@@ lesson 01 step 4. The model judging itself is not an eval. Without ground truth, you're measuring nothing."}],"answerIds":["b"],"shuffle":false,"code":"$32","runnable":true},{"type":"read","id":"evals/the-rise-of-evals-as-a-discipline/s06-06-why-the-eval-engineer-eats-the-prompt-engineer","xp":1,"hint":[],"personalize":false,"phase":"build","estSeconds":220,"concept":"eval-engineer-vs-prompt-engineer","body":"$33","cta":"Got it","runnable":true},{"type":"write","id":"evals/the-rise-of-evals-as-a-discipline/s07-07-write-the-eval-readiness-audit","xp":5,"hint":[{"level":1,"body":"Five conditionals, each adds a fixed number of points. Use `.get()` with defaults so missing fields don't crash. Then a four-way threshold for the verdict (>=80, >=50, >=20, else).","cost":0},{"level":2,"body":"EvalMatureCo hits all five: 25 + 15 + 25 + 15 + 20 = 100. VibeCo hits zero conditions, score 0. Thresholds: >=80 mature, >=50 aware, >=20 curious, else vibes era.","cost":0}],"personalize":false,"phase":"build","estSeconds":200,"concept":"eval-readiness-audit","prompt":"Write `eval_readiness(team)` that takes a team profile (dict) and\nreturns a dict with two fields:\n\n- `score`: integer 0-100, higher means MORE eval discipline (good)\n- `verdict`: string, one of:\n - `\"eval-mature\"` if score >= 80\n - `\"eval-aware\"` if score >= 50\n - `\"eval-curious\"` if score >= 20\n - `\"vibes era\"` if score < 20\n\nScore the team on these signals (each adds points to the readiness\ntotal):\n\n- `has_test_set` is True: add 25 (ground-truth cases exist)\n- `has_judge_prompt` is True: add 15 (rubric or LLM-as-judge defined somewhere)\n- `ci_runs_evals` is True: add 25 (the regression gate)\n- `tracks_eval_history` is True: add 15 (can compare runs over time)\n- `eval_count_per_feature` >= 20: add 20 (enough cases to catch obvious regressions)\n\nTwo teams run. Expected output:\n```\nEvalMatureCo: {'score': 100, 'verdict': 'eval-mature'}\nVibeCo: {'score': 0, 'verdict': 'vibes era'}\n```\n","starter":"eval_mature_co = {\n \"name\": \"EvalMatureCo\",\n \"has_test_set\": True,\n \"has_judge_prompt\": True,\n \"ci_runs_evals\": True,\n \"tracks_eval_history\": True,\n \"eval_count_per_feature\": 50,\n}\n\nvibe_co = {\n \"name\": \"VibeCo\",\n \"has_test_set\": False,\n \"has_judge_prompt\": False,\n \"ci_runs_evals\": False,\n \"tracks_eval_history\": False,\n \"eval_count_per_feature\": 0,\n}\n\n# define eval_readiness(team) below\n\nprint(f\"EvalMatureCo: {eval_readiness(eval_mature_co)}\")\nprint(f\"VibeCo: {eval_readiness(vibe_co)}\")\n","grader":{"kind":"stdout-equality","expected":"EvalMatureCo: {'score': 100, 'verdict': 'eval-mature'}\nVibeCo: {'score': 0, 'verdict': 'vibes era'}\n","normalize":"collapse-trailing-newline"},"solution":"$34","hiddenTests":[]},{"type":"checkpoint","id":"evals/the-rise-of-evals-as-a-discipline/s08-08-checkpoint","xp":8,"hint":[],"personalize":false,"phase":"check","estSeconds":240,"concept":"triage-teams-for-model-launch","prompt":"$35","starter":"teams = [\n {\"name\": \"CursorClone\", \"has_test_set\": True, \"has_judge_prompt\": True, \"ci_runs_evals\": True, \"tracks_eval_history\": True, \"eval_count_per_feature\": 80},\n {\"name\": \"SeriesBAI\", \"has_test_set\": True, \"has_judge_prompt\": True, \"ci_runs_evals\": True, \"tracks_eval_history\": False, \"eval_count_per_feature\": 30},\n {\"name\": \"SeedAI\", \"has_test_set\": True, \"has_judge_prompt\": False, \"ci_runs_evals\": False, \"tracks_eval_history\": False, \"eval_count_per_feature\": 10},\n {\"name\": \"ConsultingShop\", \"has_test_set\": True, \"has_judge_prompt\": True, \"ci_runs_evals\": False, \"tracks_eval_history\": False, \"eval_count_per_feature\": 5},\n {\"name\": \"PromptShopCo\", \"has_test_set\": False, \"has_judge_prompt\": False, \"ci_runs_evals\": False, \"tracks_eval_history\": False, \"eval_count_per_feature\": 0},\n]\n\n# define triage_teams(teams) below\n\nresult = triage_teams(teams)\nprint(f\"verdicts: {result['verdicts']}\")\nprint(f\"most at risk: {result['most_at_risk']}\")\n","grader":{"kind":"stdout-equality","expected":"verdicts: {'CursorClone': 'eval-mature', 'SeriesBAI': 'eval-mature', 'SeedAI': 'eval-curious', 'ConsultingShop': 'eval-curious', 'PromptShopCo': 'vibes era'}\nmost at risk: PromptShopCo\n","normalize":"collapse-trailing-newline"},"solution":"$36","passThreshold":0.66}],"xpTotal":21},{"slug":"model-validation-gates","title":"A gate that can't fail isn't a gate","estMinutes":8,"prerequisites":[],"status":"live","steps":[{"type":"read","id":"evals/model-validation-gates/s01-01-intro","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":120,"concept":"validation-gate-anatomy","body":"$37","cta":"Got it","code":"# a gate is three named things. if you can't name all three,\n# you don't have a gate — you have a feeling.\ngate = {\n \"name\": \"accuracy-floor\",\n \"metric\": \"accuracy\",\n \"threshold\": 0.85,\n \"eval_set\": \"holdout-v3 (200 cases, frozen 2026-01-12)\",\n \"rule\": \"block deploy if metric < threshold\",\n}\n\ncandidate = {\"model\": \"support-router-v9\", \"accuracy\": 0.83}\n\npassed = candidate[\"accuracy\"] >= gate[\"threshold\"]\ndecision = \"SHIP\" if passed else \"BLOCK\"\nprint(f\"{gate['name']}: {candidate['accuracy']} vs {gate['threshold']} -> {decision}\")\n","runnable":true},{"type":"mc","id":"evals/model-validation-gates/s02-02-spot-the-theater","xp":2,"hint":[{"level":1,"body":"For each gate, ask: what score would a candidate need to *fail* it? One of the four has no such score.","cost":0},{"level":2,"body":"Gate C: accuracy is never negative, so `>= 0.0` passes for a coin flip, a broken model, an empty model. Fourteen months of green told you nothing — it structurally cannot fail.","cost":0}],"personalize":false,"phase":"build","estSeconds":70,"concept":"theater-gate-detection","prompt":"A new teammate proudly reports \"all four gates are green, and gate C\nhas been green for 14 months straight.\" Which gate is theater — the\none that provides zero protection no matter what model you feed it?\n","options":[{"id":"gate_a","label":"Gate A — accuracy >= 0.85 on the frozen holdout","explain":"A real absolute gate. A candidate scoring 0.84 gets blocked — it can fail, so its green means something."},{"id":"gate_b","label":"Gate B — accuracy >= last shipped minus 0.01","explain":"A real regression gate. It moves with the baseline, which is the point — it fails exactly when users would get a worse model than they have."},{"id":"gate_c","label":"Gate C — accuracy >= 0.0"},{"id":"gate_d","label":"Gate D — 100% schema-valid on 12 adversarial cases","explain":"Small, but strict: one malformed output out of twelve blocks the deploy. Tiny eval sets are a coverage worry, not theater — this gate can and will fail."}],"answerIds":["gate_c"],"shuffle":false,"code":"# four gates from a real-ish deploy pipeline. one of them has\n# never blocked a deploy and never will.\ngates = [\n {\"id\": \"a\", \"check\": \"accuracy >= 0.85\",\n \"eval_set\": \"frozen holdout, 200 cases\"},\n {\"id\": \"b\", \"check\": \"accuracy >= last_shipped_accuracy - 0.01\",\n \"eval_set\": \"frozen holdout, 200 cases\"},\n {\"id\": \"c\", \"check\": \"accuracy >= 0.0\",\n \"eval_set\": \"frozen holdout, 200 cases\"},\n {\"id\": \"d\", \"check\": \"schema_valid_rate >= 1.0\",\n \"eval_set\": \"12 adversarial format cases\"},\n]\n\nfor g in gates:\n print(f\"gate {g['id']}: {g['check']} [{g['eval_set']}]\")\n","runnable":true},{"type":"predict","id":"evals/model-validation-gates/s03-03-predict-the-decision","xp":3,"hint":[{"level":1,"body":"Run each gate separately. 0.84 against a 0.80 floor is one answer; 0.84 against 0.86 minus 0.01 is another.","cost":0},{"level":2,"body":"Absolute: 0.84 >= 0.80, pass. Regression: 0.84 >= 0.85 is false, fail. The rule is AND, so the last line is `BLOCK` — good enough in general, worse than what users already have.","cost":0}],"personalize":false,"phase":"build","estSeconds":70,"concept":"absolute-vs-regression-gate","code":"baseline = {\"model\": \"router-v8\", \"accuracy\": 0.86} # what users have\ncandidate = {\"model\": \"router-v9\", \"accuracy\": 0.84} # what you want to ship\n\n# absolute gate: is it good enough, full stop?\nabsolute_ok = candidate[\"accuracy\"] >= 0.80\n# regression gate: is it at least as good as what's live (minus tolerance)?\nregression_ok = candidate[\"accuracy\"] >= baseline[\"accuracy\"] - 0.01\n\nprint(f\"absolute: {'pass' if absolute_ok else 'fail'}\")\nprint(f\"regression: {'pass' if regression_ok else 'fail'}\")\nprint(\"SHIP\" if absolute_ok and regression_ok else \"BLOCK\")\n","prompt":"Candidate v9 scores 0.84. The absolute floor is 0.80; the regression\ngate allows at most a 0.01 drop from the live model's 0.86. Both\ngates must pass to ship. Predict all three lines.\n","grader":{"kind":"stdout-equality","expected":"absolute: pass\nregression: fail\nBLOCK\n","normalize":"collapse-trailing-newline"}},{"type":"fill","id":"evals/model-validation-gates/s04-04-fill-the-regression-check","xp":3,"hint":[{"level":1,"body":"Three names are in scope. One is the score being judged, one is the wiggle room — the blank is the thing being defended.","cost":0},{"level":2,"body":"`baseline_score` — the last shipped model's number. Writing `candidate_score` there would compare the candidate to itself, and you'd have built the theater gate from step 2.","cost":0}],"personalize":false,"phase":"build","estSeconds":60,"concept":"regression-gate-comparison","prompt":"A regression gate compares the candidate to the last *shipped*\nmodel, not to a fixed number — the bar moves up as your models do.\nFill in the variable the candidate is measured against.\n\nExpected output:\n```\nFalse\nTrue\n```\n","code":"def regression_gate(candidate_score, baseline_score, tolerance):\n # pass only if the candidate is within `tolerance` of the\n # score users are currently getting.\n return candidate_score >= ___ - tolerance\n\nprint(regression_gate(0.84, 0.86, 0.01))\nprint(regression_gate(0.86, 0.86, 0.01))\n","blanks":[{"id":"baseline_ref","accept":["baseline_score"],"caseSensitive":true,"normalize":"trim"}]},{"type":"fix","id":"evals/model-validation-gates/s05-05-fix-the-theater-gate","xp":4,"hint":[{"level":1,"body":"`candidate_score >= candidate_score - 0.02` is true for every number that will ever exist. The threshold has to be anchored to something the candidate doesn't control.","cost":0},{"level":2,"body":"Line 5 becomes `threshold = baseline_score - 0.02`. Now the bar is 0.84, the candidate's 0.79 misses it, and the gate blocks a seven-point regression instead of applauding it.","cost":0}],"personalize":false,"phase":"build","estSeconds":90,"concept":"unrig-the-gate","brokenCode":"baseline_score = 0.86 # the model users have right now\ncandidate_score = 0.79 # today's candidate\n\n# bug: the bar is derived from the candidate itself. it can never fail.\nthreshold = candidate_score - 0.02\npassed = candidate_score >= threshold\n\nprint(\"SHIP\" if passed else \"BLOCK\")\n","prompt":"This deploy gate has been green for every release in history —\nincluding today's candidate, which scores 0.79 against a live model\nat 0.86. Look at line 5: the threshold is computed from the\n*candidate's own score*, so any model beats itself minus 0.02.\nGreen forever, by construction.\n\nFix line 5 so the threshold comes from the baseline — the score\nusers currently get — and today's regression finally gets caught.\n\nExpected output:\n```\nBLOCK\n```\n","grader":{"kind":"stdout-equality","expected":"BLOCK","normalize":"collapse-trailing-newline"},"bugLines":[5],"revealAfter":4},{"type":"write","id":"evals/model-validation-gates/s06-06-write-the-gate-runner","xp":5,"hint":[{"level":1,"body":"Track a boolean that starts `True` and flips on any failure. Look up each gate's score with `scores[gate[\"metric\"]]`, print the per-gate line as you go, and return the verdict after the loop.","cost":0},{"level":2,"body":"```python\ndef run_gates(gates, scores):\n all_passed = True\n for gate in gates:\n score = scores[gate[\"metric\"]]\n if score >= gate[\"threshold\"]:\n print(f\"{gate['name']}: pass\")\n else:\n print(f\"{gate['name']}: FAIL ({score} < {gate['threshold']})\")\n all_passed = False\n return \"SHIP\" if all_passed else \"BLOCK\"\n```\n\nNote the FAIL line prints the raw values — `0.9`, not `0.90` — because that's how Python renders the float.\n","cost":0}],"personalize":false,"phase":"build","estSeconds":150,"concept":"gate-runner-suite","prompt":"The deploy pipeline runs every gate and prints a receipt for each —\nnot just the final verdict, because \"BLOCK\" with no named culprit\nstarts an hour of archaeology.\n\nWrite `run_gates(gates, scores)` that:\n\n- Loops over `gates` in order. Each gate has `name`, `metric`,\n `threshold`; each metric's measured value is in `scores`.\n- Prints `\": pass\"` when `scores[metric] >= threshold`.\n- Prints `\": FAIL ( < )\"` otherwise.\n- Returns `\"SHIP\"` if every gate passed, else `\"BLOCK\"`.\n\nExpected output:\n```\naccuracy-floor: pass\nschema-valid: pass\nhelpfulness: FAIL (0.84 < 0.9)\nBLOCK\n```\n","starter":"gates = [\n {\"name\": \"accuracy-floor\", \"metric\": \"accuracy\", \"threshold\": 0.80},\n {\"name\": \"schema-valid\", \"metric\": \"format_ok\", \"threshold\": 1.00},\n {\"name\": \"helpfulness\", \"metric\": \"helpful\", \"threshold\": 0.90},\n]\n\nscores = {\"accuracy\": 0.87, \"format_ok\": 1.00, \"helpful\": 0.84}\n\n# define run_gates(gates, scores) below\n\nprint(run_gates(gates, scores))\n","grader":{"kind":"stdout-equality","expected":"accuracy-floor: pass\nschema-valid: pass\nhelpfulness: FAIL (0.84 < 0.9)\nBLOCK\n","normalize":"collapse-trailing-newline"},"solution":"gates = [\n {\"name\": \"accuracy-floor\", \"metric\": \"accuracy\", \"threshold\": 0.80},\n {\"name\": \"schema-valid\", \"metric\": \"format_ok\", \"threshold\": 1.00},\n {\"name\": \"helpfulness\", \"metric\": \"helpful\", \"threshold\": 0.90},\n]\n\nscores = {\"accuracy\": 0.87, \"format_ok\": 1.00, \"helpful\": 0.84}\n\ndef run_gates(gates, scores):\n all_passed = True\n for gate in gates:\n score = scores[gate[\"metric\"]]\n if score >= gate[\"threshold\"]:\n print(f\"{gate['name']}: pass\")\n else:\n print(f\"{gate['name']}: FAIL ({score} < {gate['threshold']})\")\n all_passed = False\n return \"SHIP\" if all_passed else \"BLOCK\"\n\nprint(run_gates(gates, scores))\n","hiddenTests":[]},{"type":"checkpoint","id":"evals/model-validation-gates/s07-07-checkpoint","xp":8,"hint":[{"level":1,"body":"Collect the failed names first with a list comprehension — every branch after that reads from it. Empty list means SHIP; otherwise the override (or its absence) picks between the two remaining strings.","cost":0},{"level":2,"body":"```python\ndef decide(gate_results, override):\n failed = [g[\"name\"] for g in gate_results if not g[\"passed\"]]\n if not failed:\n return \"SHIP\"\n names = \", \".join(failed)\n if override is None:\n return f\"BLOCK (failed: {names})\"\n return f\"SHIP-OVERRIDE by {override['who']}: {override['reason']} (failed: {names})\"\n```\n\nNotice the override string still lists the failed gates — the record keeps *both* the red gates and the human who chose to ship past them.\n","cost":0}],"personalize":false,"phase":"check","estSeconds":160,"concept":"override-with-receipts","prompt":"Final drill: the override path. Gates get overridden in real\npipelines — the sin isn't overriding, it's overriding without a\nrecord. Write `decide(gate_results, override)`:\n\n- `gate_results` is a list of `{\"name\": str, \"passed\": bool}`.\n- If every gate passed, return `\"SHIP\"`.\n- If any failed and `override` is `None`, return\n `\"BLOCK (failed: )\"` — failed gate names joined with\n `\", \"`.\n- If any failed and `override` is a dict with `who` and `reason`,\n return\n `\"SHIP-OVERRIDE by : (failed: )\"` —\n the deploy happens, and the log names the human who owns it.\n\nExpected output:\n```\nSHIP\nBLOCK (failed: regression)\nSHIP-OVERRIDE by maya (eng lead): regression is in a deprecated locale (failed: regression)\n```\n","starter":"# define decide(gate_results, override) below\n\nruns = [\n ([{\"name\": \"accuracy-floor\", \"passed\": True},\n {\"name\": \"regression\", \"passed\": True}], None),\n ([{\"name\": \"accuracy-floor\", \"passed\": True},\n {\"name\": \"regression\", \"passed\": False}], None),\n ([{\"name\": \"accuracy-floor\", \"passed\": True},\n {\"name\": \"regression\", \"passed\": False}],\n {\"who\": \"maya (eng lead)\", \"reason\": \"regression is in a deprecated locale\"}),\n]\n\nfor results, override in runs:\n print(decide(results, override))\n","grader":{"kind":"stdout-equality","expected":"SHIP\nBLOCK (failed: regression)\nSHIP-OVERRIDE by maya (eng lead): regression is in a deprecated locale (failed: regression)\n","normalize":"collapse-trailing-newline"},"solution":"def decide(gate_results, override):\n failed = [g[\"name\"] for g in gate_results if not g[\"passed\"]]\n if not failed:\n return \"SHIP\"\n names = \", \".join(failed)\n if override is None:\n return f\"BLOCK (failed: {names})\"\n return f\"SHIP-OVERRIDE by {override['who']}: {override['reason']} (failed: {names})\"\n\nruns = [\n ([{\"name\": \"accuracy-floor\", \"passed\": True},\n {\"name\": \"regression\", \"passed\": True}], None),\n ([{\"name\": \"accuracy-floor\", \"passed\": True},\n {\"name\": \"regression\", \"passed\": False}], None),\n ([{\"name\": \"accuracy-floor\", \"passed\": True},\n {\"name\": \"regression\", \"passed\": False}],\n {\"who\": \"maya (eng lead)\", \"reason\": \"regression is in a deprecated locale\"}),\n]\n\nfor results, override in runs:\n print(decide(results, override))\n","passThreshold":0.66}],"xpTotal":26}],"liveLessonCount":5,"liveStepCount":34,"xpTotal":112}},"chapter":"$1c:2:props:tree:detail","lesson":"$1c:2:props:tree:detail:lessons:4","step":"$1c:2:props:tree:detail:lessons:4:steps:1","stepIndex":1,"nav":{"tocNext":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":2},"tocPrev":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":0},"nextByPath":{"developer":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":2},"marketer":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":2},"designer":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":2},"customer-service":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":2},"copywriter":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":2},"data-analyst":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":2},"project-manager":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":2},"hr-specialist":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":2},"operations":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":2},"lawyer":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":2}},"prevByPath":{"developer":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":0},"marketer":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":0},"designer":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":0},"customer-service":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":0},"copywriter":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":0},"data-analyst":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":0},"project-manager":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":0},"hr-specialist":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":0},"operations":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":0},"lawyer":{"chapterSlug":"evals","lessonSlug":"model-validation-gates","stepIndex":0}}}}]]

A gate that can't fail isn't a gate — step 2 of 7