# Reproduce. Classify. Route. In that order. A bad prediction lands in the inbox: "your AI said my invoice total was $1,240 — it's $940." Lesson 01 taught you to open the trace. Lesson 02 gave you the four classes. This lesson is the runbook that connects them: what you actually *do*, in what order, before anyone touches a prompt. ## Step 1: reproduce — with the logged input, not a paraphrase The report you receive is a paraphrase. Support summarizes what the user said. The user summarizes what they typed. Feed either summary back into the pipeline and you're debugging a different input than the one that failed. LLM pipelines are brutally sensitive to exact wording — the paraphrase often *works fine*, you declare the bug unreproducible, and it bites the next customer a week later. **The only input that counts is the one in the trace.** Pull `logged_input`, re-run the pipeline on it, and confirm you see the same wrong output. Can't reproduce? Triage is blocked. Fix your logging before you fix anything else, because right now you're chasing a ghost. ## Step 2: classify — the four classes, one more angle Same taxonomy as lesson 02, no new buckets. But notice what the four classes really are — each one names the *kind* of problem: 1. **Retrieval** — a data problem. The wrong context went in. 2. **Prompt** — an expectation problem. You asked for something other than what you meant. 3. **Hallucination** — a model problem. Good input, fabricated output. 4. **Parse** — a code problem. Correct output, mangled on the way out. Data, expectation, model, code. Run `classify_failure(trace)` from lesson 02 and get one word back. Notice which category is missing: "the model is dumb." That phrase names no layer, no owner, and no fix. It's what triage sounds like when nobody read the trace — a shrug wearing the costume of a diagnosis. Three of the four classes don't involve the model misbehaving at all. ## Step 3: route — every class has an owner Classification is only useful because each class routes somewhere different: retrieval bugs go to whoever owns the pipeline that feeds the model, prompt bugs go to whoever owns the spec, hallucinations go to grounding and guardrails, parse bugs go to the post-processing code. Route a parse bug to the prompt owner and you get a "more careful" prompt edit that fixes nothing and breaks two adjacent cases. Run the editor. The whole runbook is a gate, a lookup, and a routing line — small enough to hold in your head on a bad Tuesday.36:T4f5,ROUTES = { "retrieval": "retrieval pipeline", "prompt": "prompt + spec", "hallucination": "grounding + guardrails", "parse": "post-processing code", } reports = [ { "id": 9101, "reproduced_with_logged_input": False, "trace": { "retrieved_chunks": ["some chunk"], "retrieved_chunks_match_query": True, "raw_output": "answer", "output_after_postprocess": "answer", }, }, { "id": 9102, "reproduced_with_logged_input": True, "trace": { "retrieved_chunks": ["ticket from ALT-CUST-04 (wrong account)"], "retrieved_chunks_match_query": False, "raw_output": "Your refund was processed.", "output_after_postprocess": "Your refund was processed.", }, }, { "id": 9103, "reproduced_with_logged_input": True, "trace": { "retrieved_chunks": ["correct ticket for CUST-04"], "retrieved_chunks_match_query": True, "raw_output": '{"refund": "$48.20"}', "output_after_postprocess": '{"refund": "48.20"}', }, }, ] # define triage(report) below for r in reports: print(f"ticket_{r['id']}: {triage(r)}") 37:T6c3,ROUTES = { "retrieval": "retrieval pipeline", "prompt": "prompt + spec", "hallucination": "grounding + guardrails", "parse": "post-processing code", } reports = [ { "id": 9101, "reproduced_with_logged_input": False, "trace": { "retrieved_chunks": ["some chunk"], "retrieved_chunks_match_query": True, "raw_output": "answer", "output_after_postprocess": "answer", }, }, { "id": 9102, "reproduced_with_logged_input": True, "trace": { "retrieved_chunks": ["ticket from ALT-CUST-04 (wrong account)"], "retrieved_chunks_match_query": False, "raw_output": "Your refund was processed.", "output_after_postprocess": "Your refund was processed.", }, }, { "id": 9103, "reproduced_with_logged_input": True, "trace": { "retrieved_chunks": ["correct ticket for CUST-04"], "retrieved_chunks_match_query": True, "raw_output": '{"refund": "$48.20"}', "output_after_postprocess": '{"refund": "48.20"}', }, }, ] def triage(report): if not report["reproduced_with_logged_input"]: return "blocked: reproduce with the logged input" trace = report["trace"] if trace["retrieved_chunks"] and not trace["retrieved_chunks_match_query"]: cls = "retrieval" elif trace["raw_output"] != trace["output_after_postprocess"]: cls = "parse" elif not trace["retrieved_chunks"]: cls = "hallucination" else: cls = "prompt" return f"route to {ROUTES[cls]}" for r in reports: print(f"ticket_{r['id']}: {triage(r)}") 16:[["$","script",null,{"type":"application/ld+json","dangerouslySetInnerHTML":{"__html":"$1e"}}],["$","$L1f",null,{"tree":{"toc":{"version":"0.1.0","generatedAt":"1970-01-01T00:00:00.000Z","chapters":[{"number":0,"slug":"before-you-build","title":"from worker to builder","blurb":"you are not here to become a programmer overnight. you are here to stop doing repeated work by hand. this chapter is the map: the builder loop, old workplace deliverables turned into site artifacts, and your first reusable tool.","status":"live","lessonCount":10,"stepCount":14,"liveLessonCount":10,"liveStepCount":14,"estMinutes":74,"xpTotal":36},{"number":1,"slug":"read-the-script","title":"read the script — your first code-reading lab","blurb":"ai hands you code before you feel ready to read it. this lab covers the three reads that matter first: what the names hold, whether the function actually returns, and which branch of the if-chain really fires.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":21,"xpTotal":78},{"number":2,"slug":"read-the-data-shapes","title":"read the data shapes — the json ai just dumped","blurb":"every api response, config file, and llm reply is lists and dicts nested in a tree. read the shape on sight, walk it with a loop, and catch the wrong-layer bug ai loves to ship.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":23,"xpTotal":80},{"number":3,"slug":"read-the-failure","title":"read the failure — cursor wrote this and crashed","blurb":"when python crashes, it tells you exactly what happened and where. most non-engineers panic at the wall of text — and ai 'fixes' the crash by hiding it. learn to read the failure, then catch only what's worth catching.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":24,"xpTotal":83},{"number":4,"slug":"read-the-pipeline","title":"read the pipeline — files in, api calls out","blurb":"every real script reads a file or calls an api — usually both. learn the with-block, the status-code families, and the retry rules: the seatbelts ai forgets to put on.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":26,"xpTotal":83},{"number":7,"slug":"mutation-and-state","title":"mutation — why your code mysteriously breaks","blurb":"when a list inside a function changes the list outside the function, that's mutation. ai does this constantly without flagging it, and it's the bug class that takes the longest to find.","status":"live","lessonCount":2,"stepCount":9,"liveLessonCount":2,"liveStepCount":9,"estMinutes":10,"xpTotal":32},{"number":13,"slug":"llm-apis","title":"llm apis — talking to claude and openai","blurb":"every ai feature you ship eventually calls a model api. learn the messages pattern, how to read the response, and the four lines ai writes every single time.","status":"live","lessonCount":4,"stepCount":27,"liveLessonCount":4,"liveStepCount":27,"estMinutes":45,"xpTotal":87},{"number":14,"slug":"structured-output","title":"structured output — making the model return json","blurb":"free-form text breaks every pipeline. learn the schema-first pattern ai uses to get reliable json back, validate it with pydantic, and catch the model's lies before they hit prod.","status":"live","lessonCount":3,"stepCount":25,"liveLessonCount":3,"liveStepCount":25,"estMinutes":38,"xpTotal":80},{"number":15,"slug":"mcp","title":"mcp — the model context protocol","blurb":"mcp is the new standard for plugging tools and data sources into ai agents. learn what an mcp server actually is, how claude code lists tools, and why this is replacing one-off integrations everywhere.","status":"live","lessonCount":3,"stepCount":26,"liveLessonCount":3,"liveStepCount":26,"estMinutes":40,"xpTotal":85},{"number":16,"slug":"agent-loops","title":"agent loops — tool_use and the request/tool/respond cycle","blurb":"an agent isn't magic. it's a while loop. learn the actual cycle claude code, cursor, and every other agent uses: model returns tool_use, you run the tool, you send the result back, repeat until end_turn.","status":"live","lessonCount":6,"stepCount":48,"liveLessonCount":6,"liveStepCount":48,"estMinutes":71,"xpTotal":154},{"number":17,"slug":"git-and-github","title":"git and github cli — the ai builder's actual workflow","blurb":"cursor and claude code commit on your behalf. reading those commits — and undoing the bad ones — is your job. learn the three-state model, the commands you'll run every day, and what `gh` does that `git` can't.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":36,"xpTotal":79},{"number":18,"slug":"secrets-and-env","title":"secrets — .env, api keys, and what not to commit","blurb":"ai ships keys to github all the time. learn the .env pattern, why os.getenv is non-negotiable, what to do when a key leaks, and the gitignore lines you need on day one.","status":"live","lessonCount":2,"stepCount":16,"liveLessonCount":2,"liveStepCount":16,"estMinutes":16,"xpTotal":58},{"number":19,"slug":"prompting","title":"prompting cursor and claude code effectively","blurb":"the difference between a one-shot ai session and a four-hour debugging spiral is almost always the first prompt. learn the structure that gets you usable code.","status":"live","lessonCount":5,"stepCount":36,"liveLessonCount":5,"liveStepCount":36,"estMinutes":59,"xpTotal":118},{"number":20,"slug":"agent-traces","title":"reading agent traces and telemetry","blurb":"when an agent fails, the trace tells you exactly where. learn to read tool calls, tool results, and stop reasons — the json breadcrumbs every agent leaves behind.","status":"live","lessonCount":3,"stepCount":25,"liveLessonCount":3,"liveStepCount":25,"estMinutes":30,"xpTotal":88},{"number":21,"slug":"evals","title":"eval-driven ai development","blurb":"if you can't test it, you can't ship it. learn the simple-but-strict eval patterns that separate ai features that work from ones that just feel like they do.","status":"live","lessonCount":5,"stepCount":34,"liveLessonCount":5,"liveStepCount":34,"estMinutes":57,"xpTotal":112},{"number":22,"slug":"context-and-retrieval","title":"context and retrieval — feeding the model real data","blurb":"rag without the overengineering. chunking, embeddings, vector search, and the small set of patterns that make a model answer from your data instead of its training set.","status":"live","lessonCount":5,"stepCount":44,"liveLessonCount":5,"liveStepCount":44,"estMinutes":75,"xpTotal":144},{"number":23,"slug":"production-tradeoffs","title":"production tradeoffs — cost, latency, quality","blurb":"the three numbers every shipped llm feature lives or dies by. token math, caching, streaming, batching, and the small set of decisions that move the product more than a model swap ever will.","status":"live","lessonCount":4,"stepCount":33,"liveLessonCount":4,"liveStepCount":33,"estMinutes":57,"xpTotal":111},{"number":24,"slug":"debugging-output","title":"debugging broken ai output","blurb":"when the model lies to your customer. the methodology for narrowing down what went wrong, the four most-common breakage classes, and the discipline that separates 'we shipped a fix' from 'we blamed the model and shrugged'.","status":"live","lessonCount":4,"stepCount":34,"liveLessonCount":4,"liveStepCount":34,"estMinutes":61,"xpTotal":112},{"number":25,"slug":"capstone","title":"capstone — ship the system","blurb":"wire it all together. the prompt, the call, the validation, the trace, the eval, the MCP tool. less a tutorial demo, more the smallest end-to-end llm feature you could ship to a real user. (retrieval and prompt-cache cost work live in chapters 22-23 — extend the capstone with them when you scale past the demo input set.)","status":"live","lessonCount":7,"stepCount":62,"liveLessonCount":7,"liveStepCount":62,"estMinutes":100,"xpTotal":219},{"number":26,"slug":"agent-harnesses","title":"agent harnesses — the layer between you and the raw api","blurb":"claude code, cursor, aider, codex cli — they're all the same four layers wrapped around the same model api. learn what those layers are, what each adds, and what you'd build yourself if you had to.","status":"live","lessonCount":4,"stepCount":28,"liveLessonCount":4,"liveStepCount":28,"estMinutes":55,"xpTotal":87},{"number":27,"slug":"ai-image-generation","title":"ai image generation — from prompt to production asset","blurb":"the 2026 image model landscape, the prompts that work, and the pipeline that turns one good idea into a hundred ready-to-ship images. nano banana 2 for volume, nano banana pro for fidelity, flux, midjourney, ideogram, gpt-image-2. when each wins and what they cost.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":60,"xpTotal":67},{"number":28,"slug":"ai-video-generation","title":"ai video generation — omni, veo 3.1, sora, and the second wave","blurb":"video is the hardest content type to generate, the most expensive, and the most strategically interesting. learn the 2026 model lineup, the camera-control patterns that separate slop from craft, and the cost math that decides whether your idea is viable.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":50,"xpTotal":61},{"number":29,"slug":"programmatic-design","title":"programmatic design — code-driven video and the new design pipeline","blurb":"ai generates raw assets; code stitches them into something shippable. hyperframes, remotion, claude design — when each tool wins, how they combine, and the data-driven workflows that turn one template into a hundred videos.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":46,"xpTotal":66},{"number":30,"slug":"harness-engineering","title":"harness engineering — the discipline behind the model","blurb":"every coding agent is a model plus a harness. the model is bought; the harness is engineered. learn the craft: how to ratchet rules from failures, fight context rot, design long-horizon loops, wire hooks as enforcement, and read the haas shift that's reshaping what you build vs buy.","status":"live","lessonCount":6,"stepCount":52,"liveLessonCount":6,"liveStepCount":52,"estMinutes":126,"xpTotal":143},{"number":31,"slug":"intro-to-terminal","title":"intro to terminal — the text way to drive your computer","blurb":"you've never opened a terminal. by the end of this chapter you have, and you can move around your files, make folders, and read files without touching the mouse. it's a keyboard shortcut, not a cockpit. every tool in the rest of this course assumes you can do this, so we do it first.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":26,"xpTotal":50},{"number":32,"slug":"intro-to-claude-cli","title":"intro to claude cli — the door to real building","blurb":"you've used claude in a chat window. the claude cli is the same model with its hands on your actual files. this chapter installs it, signs you in, and runs your first real command. by the end you've watched an ai read, plan, and change things on your machine, and you know when to reach for the cli instead of the chat box.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":32,"xpTotal":48},{"number":33,"slug":"intro-to-codex-cli","title":"intro to openai codex cli — a second tool in the kit","blurb":"you know the claude cli. the openai codex cli does the same job, an ai working in your terminal on your real files, with a different company behind it. this chapter installs it, signs you in, and runs your first command. most of what you already know carries straight over, so this chapter is mostly about what is different and when to reach for which.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":32,"xpTotal":45},{"number":34,"slug":"team-skills","title":"claude skills for teams — playbooks your whole team shares","blurb":"a claude skill is a packaged set of instructions — your team's playbook — that claude loads when it's relevant, so nobody has to re-explain it. this chapter is for people who manage teams. it covers what a skill is, how a team shares and provisions skills, real examples for hr, legal, and ops work, when a skill beats a one-off prompt, and the governance you need before any skill touches real work.","status":"live","lessonCount":4,"stepCount":28,"liveLessonCount":4,"liveStepCount":28,"estMinutes":41,"xpTotal":60},{"number":35,"slug":"dataframes-numpy-pandas","title":"dataframes with numpy and pandas","blurb":"tables are the working surface of applied ml. learn rows, columns, missing values, joins, aggregates, and the dataframe habits ai-generated notebooks assume.","status":"live","lessonCount":6,"stepCount":42,"liveLessonCount":6,"liveStepCount":42,"estMinutes":43,"xpTotal":156},{"number":36,"slug":"sql-for-ml-datasets","title":"sql for ml datasets","blurb":"most training data starts in a database. learn the select, join, filter, aggregate, and leakage traps that decide whether a model is learning signal or nonsense.","status":"live","lessonCount":4,"stepCount":28,"liveLessonCount":4,"liveStepCount":28,"estMinutes":28,"xpTotal":104},{"number":37,"slug":"dataset-pipelines","title":"dataset formats, ingestion, and validation pipelines","blurb":"a dataset is a product surface. build ingestion, validation, partitions, manifests, and checkpoints so the next run is not a mystery.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":38,"slug":"ml-math-and-stats","title":"ml math and statistics that actually show up","blurb":"vectors, probability, distributions, correlation, and uncertainty are not trivia. they are how you read model behavior without worshipping it.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":39,"slug":"supervised-learning-workflows","title":"supervised learning workflows","blurb":"labels, splits, baselines, training, prediction, and evaluation. the supervised workflow is the first complete model loop.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":40,"xpTotal":130},{"number":40,"slug":"unsupervised-learning-and-embeddings","title":"unsupervised learning, embeddings, and recommenders","blurb":"not every useful model has labels. cluster, compare, retrieve, and recommend by turning examples into useful neighborhoods.","status":"live","lessonCount":4,"stepCount":28,"liveLessonCount":4,"liveStepCount":28,"estMinutes":32,"xpTotal":104},{"number":41,"slug":"metrics-and-error-analysis","title":"metrics, slices, and error analysis","blurb":"accuracy is a blunt instrument. learn confusion matrices, precision, recall, thresholds, slices, and failure notes so model quality has evidence.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":42,"slug":"pytorch-tensors-and-autograd","title":"pytorch tensors and autograd","blurb":"read tensor code without flinching. tensors, shapes, broadcasting, gradients, and autograd are the grammar of modern deep learning scripts.","status":"live","lessonCount":4,"stepCount":28,"liveLessonCount":4,"liveStepCount":28,"estMinutes":30,"xpTotal":104},{"number":43,"slug":"training-loops-and-optimizers","title":"training loops, backprop, optimizers, and schedulers","blurb":"the training loop is where models change. learn loss, gradients, optimizer steps, schedules, checkpoints, and the bugs ai ships there.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":44,"slug":"deep-learning-architectures","title":"cnns, transformers, and useful llm internals","blurb":"architecture literacy for builders: convolution, attention, tokens, decoding, kv cache, quantization, and what those choices do to cost and behavior.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":45,"slug":"feature-experiments-registries","title":"feature pipelines, experiment tracking, and registries","blurb":"features, runs, configs, artifacts, and registries are how ml work becomes repeatable instead of a lucky notebook.","status":"live","lessonCount":5,"stepCount":35,"liveLessonCount":5,"liveStepCount":35,"estMinutes":37,"xpTotal":130},{"number":46,"slug":"model-serving-and-mlops","title":"model serving, ci/cd, and mlops","blurb":"serving a model means handling inputs, versions, routes, batch jobs, ci gates, rollback, and production failures deliberately.","status":"live","lessonCount":6,"stepCount":42,"liveLessonCount":6,"liveStepCount":42,"estMinutes":44,"xpTotal":156},{"number":47,"slug":"monitoring-cloud-portfolio","title":"monitoring, drift, cloud scale, and portfolio launch","blurb":"the last mile: logs, drift, alerts, retraining decisions, cloud cost, gpu constraints, architecture docs, demos, and role stories.","status":"live","lessonCount":6,"stepCount":42,"liveLessonCount":6,"liveStepCount":42,"estMinutes":44,"xpTotal":156},{"number":48,"slug":"graphic-design-studio","title":"graphic design studio — prompt, kit, and fifty on-brand assets","blurb":"how a working designer runs image models: prompts written like creative briefs, a brand bible that travels as structured data — with the design tokens that carry it onto screens — and the seeds, reference locks, and QA filters that make fifty generated assets look like one studio shipped them.","status":"live","lessonCount":3,"stepCount":27,"liveLessonCount":3,"liveStepCount":27,"estMinutes":73,"xpTotal":82},{"number":49,"slug":"campaign-studio","title":"campaign studio — one campaign, end to end, with the gate on","blurb":"brief to voice-locked copy variants to batch plan to calendar to the numbers pass. the whole campaign as data your scripts can check — and an approval gate nothing skips. built for the person who owns the send button.","status":"live","lessonCount":3,"stepCount":22,"liveLessonCount":3,"liveStepCount":22,"estMinutes":59,"xpTotal":69},{"number":50,"slug":"support-studio","title":"support studio — the reply drafter you can trust","blurb":"how a working support agent runs AI: a policy pack the drafter can't talk around, a banned-promises linter that holds every draft, golden eval sets that make each macro earn autonomy one intent at a time, and the escalation line where money, law, and anger stay human.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":61,"xpTotal":71},{"number":51,"slug":"copy-studio","title":"copy studio — voice from evidence, variants at scale, claims that clear","blurb":"a client's voice measured from their own published copy, variant batches that pass the voice gate or die with a named reason, and a claim check that blocks the ship button while any fact is unsourced. corrections-log culture, as code.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":66,"xpTotal":72},{"number":52,"slug":"analytics-studio","title":"analytics studio — messy export to defensible memo","blurb":"how a working analyst runs AI: a cleaning log that accounts for every dropped row, sample-first checks that catch the plausible-but-wrong aggregation, a memo where every number carries the address of the cell that made it, SR 11-7-style challenge against the AI's recommendation, and the rerun test — same data, same code, same memo, or it isn't done.","status":"live","lessonCount":3,"stepCount":22,"liveLessonCount":3,"liveStepCount":22,"estMinutes":61,"xpTotal":74},{"number":53,"slug":"delivery-studio","title":"delivery studio — direct the build without becoming the bottleneck","blurb":"how a working delivery lead runs AI builders: specs with acceptance checks written before the build, a review that runs the checks instead of admiring the demo, status roll-ups computed from acceptance records instead of standup vibes, and the not-ready call made on evidence you can read out loud to a steering committee.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":56,"xpTotal":72},{"number":54,"slug":"people-studio","title":"people studio — draft fast, decide human, prove everything","blurb":"how a working hr specialist runs AI: a data firewall that keeps employee records out of prompts by code instead of willpower, recruiting briefs where every claim cites a resume line, policy drafts that cannot publish without a named legal reviewer, and the adverse-action paper trail that survives a discrimination claim.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":56,"xpTotal":72},{"number":55,"slug":"ops-studio","title":"ops studio — tribal knowledge to running automation","blurb":"how a working ops lead gets the process out of one person's head and into automation that can be trusted: SOPs captured as versioned data with the invented steps hunted down, recurring checklists with an explicit auto-run vs named-approval line, exception queues instead of silent failures, and a receipt against the export for every number in the weekly report.","status":"live","lessonCount":3,"stepCount":24,"liveLessonCount":3,"liveStepCount":24,"estMinutes":58,"xpTotal":72},{"number":56,"slug":"legal-studio","title":"legal studio — the cite-check gate and the log that survives","blurb":"how a working lawyer runs AI: a citation-verification drill where every authority gets opened at the source (the fabricated ones always look right), clause extraction that's worthless without a pinpoint, privilege rules written as a routing table instead of remembered under deadline, and the review log you'd want in the record when opposing counsel moves for sanctions.","status":"live","lessonCount":3,"stepCount":22,"liveLessonCount":3,"liveStepCount":22,"estMinutes":60,"xpTotal":69},{"number":57,"slug":"agent-studio","title":"agent studio — ship one agent feature before you stand up","blurb":"one sitting, one agent feature, end to end: a tool schema that makes the dangerous call a validation error, a loop that meters its own spend, golden cases wired before the router works, a trace you debug from instead of vibes, and the ratchet that turns tonight's incident into tomorrow's eval. practice at production tempo.","status":"live","lessonCount":3,"stepCount":23,"liveLessonCount":3,"liveStepCount":23,"estMinutes":66,"xpTotal":73}]},"accessToc":{"chapters":[{"slug":"before-you-build","lessons":[{"slug":"from-worker-to-builder","steps":[{"id":"before-you-build/from-worker-to-builder/s01-01-intro"}]},{"slug":"upgrade-dull-artifacts-into-interactive-sites","steps":[{"id":"before-you-build/upgrade-dull-artifacts-into-interactive-sites/s01-01-intro"},{"id":"before-you-build/upgrade-dull-artifacts-into-interactive-sites/s02-02-build-brief"},{"id":"before-you-build/upgrade-dull-artifacts-into-interactive-sites/s03-03-your-artifact-brief"}]},{"slug":"why-building-matters-now","steps":[{"id":"before-you-build/why-building-matters-now/s01-01-intro"}]},{"slug":"the-builder-loop","steps":[{"id":"before-you-build/the-builder-loop/s01-01-intro"}]},{"slug":"your-first-builder-brief","steps":[{"id":"before-you-build/your-first-builder-brief/s01-01-intro"}]},{"slug":"what-the-ai-is-doing","steps":[{"id":"before-you-build/what-the-ai-is-doing/s01-01-intro"}]},{"slug":"build-your-first-reusable-tool","steps":[{"id":"before-you-build/build-your-first-reusable-tool/s01-01-intro"}]},{"slug":"from-prompt-to-project","steps":[{"id":"before-you-build/from-prompt-to-project/s01-01-intro"}]},{"slug":"why-code-appears-later","steps":[{"id":"before-you-build/why-code-appears-later/s01-01-intro"}]},{"slug":"use-it-again","steps":[{"id":"before-you-build/use-it-again/s01-01-intro"},{"id":"before-you-build/use-it-again/s02-02-which-fix-compounds"},{"id":"before-you-build/use-it-again/s03-03-save-version-two"}]}]},{"slug":"read-the-script","lessons":[{"slug":"names-and-values","steps":[{"id":"read-the-script/names-and-values/s01-01-intro"},{"id":"read-the-script/names-and-values/s02-02-spot-the-type"},{"id":"read-the-script/names-and-values/s03-03-assignment"},{"id":"read-the-script/names-and-values/s04-04-predict-the-score"},{"id":"read-the-script/names-and-values/s05-05-fill-the-name"},{"id":"read-the-script/names-and-values/s06-06-fix-the-typo"},{"id":"read-the-script/names-and-values/s07-07-write-greeting"},{"id":"read-the-script/names-and-values/s08-08-checkpoint"}]},{"slug":"functions-and-the-hallucinated-return","steps":[{"id":"read-the-script/functions-and-the-hallucinated-return/s01-01-intro"},{"id":"read-the-script/functions-and-the-hallucinated-return/s02-02-which-returns"},{"id":"read-the-script/functions-and-the-hallucinated-return/s03-03-return-vs-print"},{"id":"read-the-script/functions-and-the-hallucinated-return/s04-04-predict-the-result"},{"id":"read-the-script/functions-and-the-hallucinated-return/s05-05-fix-the-missing-return"},{"id":"read-the-script/functions-and-the-hallucinated-return/s06-06-fix-print-vs-return"},{"id":"read-the-script/functions-and-the-hallucinated-return/s07-07-write-square"},{"id":"read-the-script/functions-and-the-hallucinated-return/s08-08-checkpoint"}]},{"slug":"decisions-if-elif","steps":[{"id":"read-the-script/decisions-if-elif/s01-01-intro"},{"id":"read-the-script/decisions-if-elif/s02-02-truthiness-quiz"},{"id":"read-the-script/decisions-if-elif/s03-03-eq-vs-is"},{"id":"read-the-script/decisions-if-elif/s04-04-predict-the-empty-list"},{"id":"read-the-script/decisions-if-elif/s05-05-fix-the-zero-bug"},{"id":"read-the-script/decisions-if-elif/s06-06-fix-the-elif-order"},{"id":"read-the-script/decisions-if-elif/s07-07-checkpoint"},{"id":"read-the-script/decisions-if-elif/s08-08-going-deeper"}]}]},{"slug":"read-the-data-shapes","lessons":[{"slug":"the-shapes-ai-dumps-on-you","steps":[{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s01-01-intro"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s02-02-which-is-which"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s03-03-nested-paths"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s04-04-predict-the-deep-key"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s05-05-fill-the-index"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s06-06-fix-the-keyerror"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s07-07-fix-the-list-vs-dict"},{"id":"read-the-data-shapes/the-shapes-ai-dumps-on-you/s08-08-checkpoint"}]},{"slug":"walking-the-shape","steps":[{"id":"read-the-data-shapes/walking-the-shape/s01-01-intro"},{"id":"read-the-data-shapes/walking-the-shape/s02-02-which-loop-shape"},{"id":"read-the-data-shapes/walking-the-shape/s03-03-walking-the-shape"},{"id":"read-the-data-shapes/walking-the-shape/s04-04-predict-the-list-loop"},{"id":"read-the-data-shapes/walking-the-shape/s05-05-fill-the-loop-variable"},{"id":"read-the-data-shapes/walking-the-shape/s06-06-fix-the-dict-loop"},{"id":"read-the-data-shapes/walking-the-shape/s07-07-write-sum-loop"},{"id":"read-the-data-shapes/walking-the-shape/s08-08-checkpoint"}]},{"slug":"comprehensions-and-the-wrong-layer-bug","steps":[{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s01-01-intro"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s02-02-which-comp-shape"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s03-03-with-filter"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s04-04-predict-the-comp"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s05-05-fix-the-wrong-layer"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s06-06-write-comp"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s07-07-checkpoint"},{"id":"read-the-data-shapes/comprehensions-and-the-wrong-layer-bug/s08-08-going-deeper"}]}]},{"slug":"read-the-failure","lessons":[{"slug":"read-bottom-up","steps":[{"id":"read-the-failure/read-bottom-up/s01-01-anatomy"},{"id":"read-the-failure/read-bottom-up/s02-02-which-error"},{"id":"read-the-failure/read-bottom-up/s03-03-the-five-fingerprints"},{"id":"read-the-failure/read-bottom-up/s04-04-predict-the-class"},{"id":"read-the-failure/read-bottom-up/s05-05-fix-nameerror"},{"id":"read-the-failure/read-bottom-up/s06-06-fix-attributeerror"},{"id":"read-the-failure/read-bottom-up/s07-07-write-defensive"},{"id":"read-the-failure/read-bottom-up/s08-08-checkpoint"}]},{"slug":"try-except-what-ai-misuses","steps":[{"id":"read-the-failure/try-except-what-ai-misuses/s01-01-intro"},{"id":"read-the-failure/try-except-what-ai-misuses/s02-02-which-block-runs"},{"id":"read-the-failure/try-except-what-ai-misuses/s03-03-naming-the-error"},{"id":"read-the-failure/try-except-what-ai-misuses/s04-04-predict-the-branch"},{"id":"read-the-failure/try-except-what-ai-misuses/s05-05-fill-the-keyword"},{"id":"read-the-failure/try-except-what-ai-misuses/s06-06-fix-the-bare-except"},{"id":"read-the-failure/try-except-what-ai-misuses/s07-07-write-safe-lookup"},{"id":"read-the-failure/try-except-what-ai-misuses/s08-08-checkpoint"}]},{"slug":"fail-loudly","steps":[{"id":"read-the-failure/fail-loudly/s01-01-intro"},{"id":"read-the-failure/fail-loudly/s02-02-which-raise-fires"},{"id":"read-the-failure/fail-loudly/s03-03-predict-the-raise"},{"id":"read-the-failure/fail-loudly/s04-04-fill-the-raise-keyword"},{"id":"read-the-failure/fail-loudly/s05-05-fix-the-silent-swallow"},{"id":"read-the-failure/fail-loudly/s06-06-write-validate-age"},{"id":"read-the-failure/fail-loudly/s07-07-checkpoint"},{"id":"read-the-failure/fail-loudly/s08-08-going-deeper"}]}]},{"slug":"read-the-pipeline","lessons":[{"slug":"files-without-fear","steps":[{"id":"read-the-pipeline/files-without-fear/s01-01-intro"},{"id":"read-the-pipeline/files-without-fear/s02-02-which-mode"},{"id":"read-the-pipeline/files-without-fear/s03-03-pathlib-and-manifests"},{"id":"read-the-pipeline/files-without-fear/s04-04-predict-the-suffix"},{"id":"read-the-pipeline/files-without-fear/s05-05-fill-the-mode"},{"id":"read-the-pipeline/files-without-fear/s06-06-fix-the-wrong-mode"},{"id":"read-the-pipeline/files-without-fear/s07-07-write-manifest-entry"},{"id":"read-the-pipeline/files-without-fear/s08-08-checkpoint"}]},{"slug":"talking-to-apis","steps":[{"id":"read-the-pipeline/talking-to-apis/s01-01-intro"},{"id":"read-the-pipeline/talking-to-apis/s02-02-which-status-family"},{"id":"read-the-pipeline/talking-to-apis/s03-03-parsing-the-json"},{"id":"read-the-pipeline/talking-to-apis/s04-04-predict-the-key"},{"id":"read-the-pipeline/talking-to-apis/s05-05-fill-the-method"},{"id":"read-the-pipeline/talking-to-apis/s06-06-fix-the-status-check"},{"id":"read-the-pipeline/talking-to-apis/s07-07-write-classify-status"},{"id":"read-the-pipeline/talking-to-apis/s08-08-checkpoint"}]},{"slug":"when-the-pipeline-breaks","steps":[{"id":"read-the-pipeline/when-the-pipeline-breaks/s01-01-intro"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s02-02-honor-the-retry-after"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s03-03-predict-the-status-handling"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s04-04-fill-the-raise-for-status"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s05-05-fix-the-missing-check"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s06-06-fix-the-broad-retry"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s07-07-checkpoint"},{"id":"read-the-pipeline/when-the-pipeline-breaks/s08-08-going-deeper"}]}]},{"slug":"mutation-and-state","lessons":[{"slug":"why-it-breaks","steps":[{"id":"mutation-and-state/why-it-breaks/s01-01-intro"},{"id":"mutation-and-state/why-it-breaks/s02-02-which-mutates"},{"id":"mutation-and-state/why-it-breaks/s03-03-shared-reference"},{"id":"mutation-and-state/why-it-breaks/s04-04-fix-the-shared-list"},{"id":"mutation-and-state/why-it-breaks/s05-05-checkpoint"}]},{"slug":"copy-vs-reference","steps":[{"id":"mutation-and-state/copy-vs-reference/s01-01-deep-copy"},{"id":"mutation-and-state/copy-vs-reference/s02-02-predict-the-shallow"},{"id":"mutation-and-state/copy-vs-reference/s03-03-fix-the-shallow-bug"},{"id":"mutation-and-state/copy-vs-reference/s04-04-checkpoint"}]}]},{"slug":"llm-apis","lessons":[{"slug":"before-the-api-what-the-model-is-doing","steps":[{"id":"llm-apis/before-the-api-what-the-model-is-doing/s01-01-intro"}]},{"slug":"the-messages-pattern","steps":[{"id":"llm-apis/the-messages-pattern/s01-01-intro"},{"id":"llm-apis/the-messages-pattern/s02-02-which-role-is-yours"},{"id":"llm-apis/the-messages-pattern/s03-03-reading-the-response"},{"id":"llm-apis/the-messages-pattern/s04-04-predict-the-text"},{"id":"llm-apis/the-messages-pattern/s05-05-fill-the-role"},{"id":"llm-apis/the-messages-pattern/s06-06-fix-the-message-shape"},{"id":"llm-apis/the-messages-pattern/s07-07-fix-the-response-access"},{"id":"llm-apis/the-messages-pattern/s08-08-write-ask-claude"},{"id":"llm-apis/the-messages-pattern/s09-09-checkpoint"}]},{"slug":"reading-the-response","steps":[{"id":"llm-apis/reading-the-response/s01-01-intro"},{"id":"llm-apis/reading-the-response/s02-02-which-block-do-you-read"},{"id":"llm-apis/reading-the-response/s03-03-the-five-stop-reasons"},{"id":"llm-apis/reading-the-response/s04-04-predict-the-text"},{"id":"llm-apis/reading-the-response/s05-05-fill-the-text-extract"},{"id":"llm-apis/reading-the-response/s06-06-fix-the-content-zero"},{"id":"llm-apis/reading-the-response/s07-07-fix-the-stop-assumption"},{"id":"llm-apis/reading-the-response/s08-08-write-the-extractor"},{"id":"llm-apis/reading-the-response/s09-09-checkpoint"}]},{"slug":"the-model-picker","steps":[{"id":"llm-apis/the-model-picker/s01-01-why-model-choice-is-a-product-decision"},{"id":"llm-apis/the-model-picker/s02-02-haiku-territory"},{"id":"llm-apis/the-model-picker/s03-03-which-model-here"},{"id":"llm-apis/the-model-picker/s04-04-sonnet-territory"},{"id":"llm-apis/the-model-picker/s05-05-opus-and-the-thinking-budget"},{"id":"llm-apis/the-model-picker/s06-06-pick-the-model"},{"id":"llm-apis/the-model-picker/s07-07-write-the-model-routing-memo"},{"id":"llm-apis/the-model-picker/s08-08-checkpoint"}]}]},{"slug":"structured-output","lessons":[{"slug":"getting-json-back","steps":[{"id":"structured-output/getting-json-back/s01-01-intro"},{"id":"structured-output/getting-json-back/s02-02-why-validate"},{"id":"structured-output/getting-json-back/s03-03-pydantic-models"},{"id":"structured-output/getting-json-back/s04-04-predict-the-validated"},{"id":"structured-output/getting-json-back/s05-05-fill-the-field-type"},{"id":"structured-output/getting-json-back/s06-06-fix-the-required-field"},{"id":"structured-output/getting-json-back/s07-07-fix-the-type-mismatch"},{"id":"structured-output/getting-json-back/s08-08-write-parse-extraction"},{"id":"structured-output/getting-json-back/s09-09-checkpoint"}]},{"slug":"schemas-eat-prompts","steps":[{"id":"structured-output/schemas-eat-prompts/s01-01-the-trust-boundary"},{"id":"structured-output/schemas-eat-prompts/s02-02-three-shipped-breakages"},{"id":"structured-output/schemas-eat-prompts/s03-03-which-boundary-broke"},{"id":"structured-output/schemas-eat-prompts/s04-04-schema-as-prompt-replacement"},{"id":"structured-output/schemas-eat-prompts/s05-05-where-validation-belongs"},{"id":"structured-output/schemas-eat-prompts/s06-06-the-pydantic-migration-as-war-story"},{"id":"structured-output/schemas-eat-prompts/s07-07-write-the-boundary-audit"},{"id":"structured-output/schemas-eat-prompts/s08-08-checkpoint"}]},{"slug":"dataset-schema-validation","steps":[{"id":"structured-output/dataset-schema-validation/s01-01-validate-at-the-boundary"},{"id":"structured-output/dataset-schema-validation/s02-02-the-silent-drop"},{"id":"structured-output/dataset-schema-validation/s03-03-predict-the-missing-fields"},{"id":"structured-output/dataset-schema-validation/s04-04-fill-the-membership-test"},{"id":"structured-output/dataset-schema-validation/s05-05-fix-the-range-check"},{"id":"structured-output/dataset-schema-validation/s06-06-write-validate-batch"},{"id":"structured-output/dataset-schema-validation/s07-07-claims-with-receipts"},{"id":"structured-output/dataset-schema-validation/s08-08-checkpoint"}]}]},{"slug":"mcp","lessons":[{"slug":"what-mcp-is","steps":[{"id":"mcp/what-mcp-is/s01-01-intro"},{"id":"mcp/what-mcp-is/s02-02-which-side-is-the-server"},{"id":"mcp/what-mcp-is/s03-03-the-tool-list"},{"id":"mcp/what-mcp-is/s04-04-predict-the-tool-result"},{"id":"mcp/what-mcp-is/s05-05-fill-the-tool-name"},{"id":"mcp/what-mcp-is/s06-06-fix-the-missing-argument"},{"id":"mcp/what-mcp-is/s07-07-fix-the-result-shape"},{"id":"mcp/what-mcp-is/s08-08-write-call-tool"},{"id":"mcp/what-mcp-is/s09-09-checkpoint"}]},{"slug":"writing-a-tiny-mcp-server","steps":[{"id":"mcp/writing-a-tiny-mcp-server/s01-01-intro"},{"id":"mcp/writing-a-tiny-mcp-server/s02-02-which-message-is-call-tool"},{"id":"mcp/writing-a-tiny-mcp-server/s03-03-the-tool-registry"},{"id":"mcp/writing-a-tiny-mcp-server/s04-04-predict-the-list-tools-output"},{"id":"mcp/writing-a-tiny-mcp-server/s05-05-fill-the-input-schema"},{"id":"mcp/writing-a-tiny-mcp-server/s06-06-fix-the-missing-error-response"},{"id":"mcp/writing-a-tiny-mcp-server/s07-07-fix-the-validation-gap"},{"id":"mcp/writing-a-tiny-mcp-server/s08-08-write-the-call-tool-dispatcher"},{"id":"mcp/writing-a-tiny-mcp-server/s09-09-checkpoint"}]},{"slug":"why-mcp-won","steps":[{"id":"mcp/why-mcp-won/s01-01-the-graveyard-of-tool-protocols"},{"id":"mcp/why-mcp-won/s02-02-what-openai-plugins-got-wrong"},{"id":"mcp/why-mcp-won/s03-03-which-failure-mode"},{"id":"mcp/why-mcp-won/s04-04-the-mcp-bet"},{"id":"mcp/why-mcp-won/s05-05-the-network-effect"},{"id":"mcp/why-mcp-won/s06-06-spot-the-mcp-wedge"},{"id":"mcp/why-mcp-won/s07-07-write-the-protocol-postmortem"},{"id":"mcp/why-mcp-won/s08-08-checkpoint"}]}]},{"slug":"agent-loops","lessons":[{"slug":"the-loop","steps":[{"id":"agent-loops/the-loop/s01-01-intro"},{"id":"agent-loops/the-loop/s02-02-which-stop-reason"},{"id":"agent-loops/the-loop/s03-03-tool-result-shape"},{"id":"agent-loops/the-loop/s04-04-predict-the-next-call"},{"id":"agent-loops/the-loop/s05-05-fill-the-stop-condition"},{"id":"agent-loops/the-loop/s06-06-fix-the-tool-id"},{"id":"agent-loops/the-loop/s07-07-fix-the-loop-exit"},{"id":"agent-loops/the-loop/s08-08-write-the-loop"},{"id":"agent-loops/the-loop/s09-09-checkpoint"}]},{"slug":"multi-step-tools","steps":[{"id":"agent-loops/multi-step-tools/s01-01-intro"},{"id":"agent-loops/multi-step-tools/s02-02-which-tool-runs"},{"id":"agent-loops/multi-step-tools/s03-03-tool-registry"},{"id":"agent-loops/multi-step-tools/s04-04-predict-the-chain"},{"id":"agent-loops/multi-step-tools/s05-05-fill-the-dispatch"},{"id":"agent-loops/multi-step-tools/s06-06-fix-the-missing-tool"},{"id":"agent-loops/multi-step-tools/s07-07-fix-the-parallel-ids"},{"id":"agent-loops/multi-step-tools/s08-08-write-multi-tool-agent"},{"id":"agent-loops/multi-step-tools/s09-09-checkpoint"}]},{"slug":"routing","steps":[{"id":"agent-loops/routing/s01-01-intro"},{"id":"agent-loops/routing/s02-02-when-to-route"},{"id":"agent-loops/routing/s03-03-classifier-output"},{"id":"agent-loops/routing/s04-04-predict-the-route"},{"id":"agent-loops/routing/s05-05-fill-the-dispatch"},{"id":"agent-loops/routing/s06-06-fix-the-fallback"},{"id":"agent-loops/routing/s07-07-fix-the-loose-classifier"},{"id":"agent-loops/routing/s08-08-write-the-router"},{"id":"agent-loops/routing/s09-09-checkpoint"}]},{"slug":"evaluator-optimizer","steps":[{"id":"agent-loops/evaluator-optimizer/s01-01-intro"},{"id":"agent-loops/evaluator-optimizer/s02-02-when-it-earns-its-keep"},{"id":"agent-loops/evaluator-optimizer/s03-03-judge-output-shape"},{"id":"agent-loops/evaluator-optimizer/s04-04-predict-the-iterations"},{"id":"agent-loops/evaluator-optimizer/s05-05-fill-the-exit"},{"id":"agent-loops/evaluator-optimizer/s06-06-fix-the-judge-parsing"},{"id":"agent-loops/evaluator-optimizer/s07-07-fix-the-missing-cap"},{"id":"agent-loops/evaluator-optimizer/s08-08-write-the-evaluator-optimizer"},{"id":"agent-loops/evaluator-optimizer/s09-09-checkpoint"}]},{"slug":"why-every-framework-is-thirty-lines","steps":[{"id":"agent-loops/why-every-framework-is-thirty-lines/s01-01-the-frameworks-converged"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s02-02-langgraph-walkthrough"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s03-03-the-work-is-a-graph"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s04-04-which-framework-fits"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s05-05-vercel-ai-sdk-walkthrough"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s06-06-the-five-canonical-patterns"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s07-07-when-to-stay-framework-free"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s08-08-write-the-tradeoff-memo"},{"id":"agent-loops/why-every-framework-is-thirty-lines/s09-09-checkpoint"}]},{"slug":"the-industry-map","steps":[{"id":"agent-loops/the-industry-map/s01-01-the-five-layers"},{"id":"agent-loops/the-industry-map/s02-02-which-layer-failed"},{"id":"agent-loops/the-industry-map/s03-03-the-shape-of-the-stack"}]}]},{"slug":"git-and-github","lessons":[{"slug":"the-three-states","steps":[{"id":"git-and-github/the-three-states/s01-01-intro"},{"id":"git-and-github/the-three-states/s02-02-which-command-stages"},{"id":"git-and-github/the-three-states/s03-03-status-and-diff"},{"id":"git-and-github/the-three-states/s04-04-predict-the-status"},{"id":"git-and-github/the-three-states/s05-05-fill-the-command"},{"id":"git-and-github/the-three-states/s06-06-fix-the-staged-files"},{"id":"git-and-github/the-three-states/s07-07-fix-the-pr-command"},{"id":"git-and-github/the-three-states/s08-08-write-status-summary"},{"id":"git-and-github/the-three-states/s09-09-checkpoint"}]},{"slug":"three-git-disasters-ai-shipped","steps":[{"id":"git-and-github/three-git-disasters-ai-shipped/s01-01-why-postmortems-teach-better"},{"id":"git-and-github/three-git-disasters-ai-shipped/s02-02-uber-2016-credentials-in-repo"},{"id":"git-and-github/three-git-disasters-ai-shipped/s03-03-which-control-would-have-caught-it"},{"id":"git-and-github/three-git-disasters-ai-shipped/s04-04-the-samsung-paste"},{"id":"git-and-github/three-git-disasters-ai-shipped/s05-05-the-cursor-git-add-dot-default"},{"id":"git-and-github/three-git-disasters-ai-shipped/s06-06-spot-the-failure-mode"},{"id":"git-and-github/three-git-disasters-ai-shipped/s07-07-write-your-pre-commit-checklist"},{"id":"git-and-github/three-git-disasters-ai-shipped/s08-08-checkpoint"}]},{"slug":"github-actions-for-model-checks","steps":[{"id":"git-and-github/github-actions-for-model-checks/s01-01-intro"},{"id":"git-and-github/github-actions-for-model-checks/s02-02-green-but-wrong"},{"id":"git-and-github/github-actions-for-model-checks/s03-03-predict-the-run"},{"id":"git-and-github/github-actions-for-model-checks/s04-04-fill-the-trigger"},{"id":"git-and-github/github-actions-for-model-checks/s05-05-fix-the-eval-gate"},{"id":"git-and-github/github-actions-for-model-checks/s06-06-write-run-job"},{"id":"git-and-github/github-actions-for-model-checks/s07-07-checkpoint"}]}]},{"slug":"secrets-and-env","lessons":[{"slug":"keeping-keys-safe","steps":[{"id":"secrets-and-env/keeping-keys-safe/s01-01-intro"},{"id":"secrets-and-env/keeping-keys-safe/s02-02-which-line-leaks"},{"id":"secrets-and-env/keeping-keys-safe/s03-03-loading-env-vars"},{"id":"secrets-and-env/keeping-keys-safe/s04-04-predict-the-key"},{"id":"secrets-and-env/keeping-keys-safe/s05-05-fill-the-getenv"},{"id":"secrets-and-env/keeping-keys-safe/s06-06-fix-the-hardcoded-key"},{"id":"secrets-and-env/keeping-keys-safe/s07-07-fix-the-missing-default"},{"id":"secrets-and-env/keeping-keys-safe/s08-08-write-load-config"},{"id":"secrets-and-env/keeping-keys-safe/s09-09-checkpoint"}]},{"slug":"ml-service-secrets","steps":[{"id":"secrets-and-env/ml-service-secrets/s01-01-intro"},{"id":"secrets-and-env/ml-service-secrets/s02-02-which-log-line-leaks"},{"id":"secrets-and-env/ml-service-secrets/s03-03-predict-the-scan"},{"id":"secrets-and-env/ml-service-secrets/s04-04-fill-the-lookup"},{"id":"secrets-and-env/ml-service-secrets/s05-05-fix-the-logged-key"},{"id":"secrets-and-env/ml-service-secrets/s06-06-write-scan-logs"},{"id":"secrets-and-env/ml-service-secrets/s07-07-checkpoint"}]}]},{"slug":"prompting","lessons":[{"slug":"builder-briefs-before-prompts","steps":[{"id":"prompting/builder-briefs-before-prompts/s01-01-intro"}]},{"slug":"the-prompt-craft","steps":[{"id":"prompting/the-prompt-craft/s01-01-intro"},{"id":"prompting/the-prompt-craft/s02-02-which-prompt-is-better"},{"id":"prompting/the-prompt-craft/s03-03-context-rot"},{"id":"prompting/the-prompt-craft/s04-04-predict-the-output"},{"id":"prompting/the-prompt-craft/s05-05-fill-the-constraint"},{"id":"prompting/the-prompt-craft/s06-06-fix-the-vague-prompt"},{"id":"prompting/the-prompt-craft/s07-07-fix-the-bloated-context"},{"id":"prompting/the-prompt-craft/s08-08-write-the-prompt"},{"id":"prompting/the-prompt-craft/s09-09-checkpoint"}]},{"slug":"few-shot-and-reasoning","steps":[{"id":"prompting/few-shot-and-reasoning/s01-01-intro"},{"id":"prompting/few-shot-and-reasoning/s02-02-when-few-shot-helps"},{"id":"prompting/few-shot-and-reasoning/s03-03-the-cot-trap"},{"id":"prompting/few-shot-and-reasoning/s04-04-predict-the-format-lock"},{"id":"prompting/few-shot-and-reasoning/s05-05-fill-the-example-format"},{"id":"prompting/few-shot-and-reasoning/s06-06-fix-the-cot-on-reasoning"},{"id":"prompting/few-shot-and-reasoning/s07-07-fix-the-format-mismatch"},{"id":"prompting/few-shot-and-reasoning/s08-08-write-the-prompt-builder"},{"id":"prompting/few-shot-and-reasoning/s09-09-checkpoint"}]},{"slug":"agent-config-files","steps":[{"id":"prompting/agent-config-files/s01-01-intro"},{"id":"prompting/agent-config-files/s02-02-which-belongs-in-the-file"},{"id":"prompting/agent-config-files/s03-03-anatomy-of-a-good-file"},{"id":"prompting/agent-config-files/s04-04-predict-which-rule-fires"},{"id":"prompting/agent-config-files/s05-05-fill-the-rule-block"},{"id":"prompting/agent-config-files/s06-06-fix-the-too-vague"},{"id":"prompting/agent-config-files/s07-07-fix-the-stale-pointer"},{"id":"prompting/agent-config-files/s08-08-write-a-claude-md"},{"id":"prompting/agent-config-files/s09-09-checkpoint"}]},{"slug":"what-aged-in-prompting","steps":[{"id":"prompting/what-aged-in-prompting/s01-01-prompting-as-a-moving-target"},{"id":"prompting/what-aged-in-prompting/s02-02-cot-then-vs-now"},{"id":"prompting/what-aged-in-prompting/s03-03-which-still-works"},{"id":"prompting/what-aged-in-prompting/s04-04-the-system-prompt-arms-race"},{"id":"prompting/what-aged-in-prompting/s05-05-spot-the-stale-technique"},{"id":"prompting/what-aged-in-prompting/s06-06-the-prompt-engineer-debate"},{"id":"prompting/what-aged-in-prompting/s07-07-write-the-cheatsheet-update"},{"id":"prompting/what-aged-in-prompting/s08-08-checkpoint"}]}]},{"slug":"agent-traces","lessons":[{"slug":"reading-the-trace","steps":[{"id":"agent-traces/reading-the-trace/s01-01-intro"},{"id":"agent-traces/reading-the-trace/s02-02-which-turn-failed"},{"id":"agent-traces/reading-the-trace/s03-03-the-stop-reasons"},{"id":"agent-traces/reading-the-trace/s04-04-predict-the-stop"},{"id":"agent-traces/reading-the-trace/s05-05-fill-the-tool-name"},{"id":"agent-traces/reading-the-trace/s06-06-fix-the-loop"},{"id":"agent-traces/reading-the-trace/s07-07-fix-the-arg-shape"},{"id":"agent-traces/reading-the-trace/s08-08-write-the-summarizer"},{"id":"agent-traces/reading-the-trace/s09-09-checkpoint"}]},{"slug":"trace-driven-debugging","steps":[{"id":"agent-traces/trace-driven-debugging/s01-01-intro"},{"id":"agent-traces/trace-driven-debugging/s02-02-which-pattern-is-this"},{"id":"agent-traces/trace-driven-debugging/s03-03-the-loop-signature"},{"id":"agent-traces/trace-driven-debugging/s04-04-predict-the-call-count"},{"id":"agent-traces/trace-driven-debugging/s05-05-fill-the-detector"},{"id":"agent-traces/trace-driven-debugging/s06-06-fix-the-tool-routing"},{"id":"agent-traces/trace-driven-debugging/s07-07-fix-the-prompt-bloat"},{"id":"agent-traces/trace-driven-debugging/s08-08-write-the-trace-summary"},{"id":"agent-traces/trace-driven-debugging/s09-09-checkpoint"}]},{"slug":"prediction-ids-and-inference-logs","steps":[{"id":"agent-traces/prediction-ids-and-inference-logs/s01-01-intro"},{"id":"agent-traces/prediction-ids-and-inference-logs/s02-02-pick-the-log-line"},{"id":"agent-traces/prediction-ids-and-inference-logs/s03-03-predict-the-ids"},{"id":"agent-traces/prediction-ids-and-inference-logs/s04-04-fill-the-join-key"},{"id":"agent-traces/prediction-ids-and-inference-logs/s05-05-fix-the-join"},{"id":"agent-traces/prediction-ids-and-inference-logs/s06-06-write-the-logger"},{"id":"agent-traces/prediction-ids-and-inference-logs/s07-07-checkpoint"}]}]},{"slug":"evals","lessons":[{"slug":"check-before-you-trust","steps":[{"id":"evals/check-before-you-trust/s01-01-intro"}]},{"slug":"writing-evals","steps":[{"id":"evals/writing-evals/s01-01-intro"},{"id":"evals/writing-evals/s02-02-which-eval-passes"},{"id":"evals/writing-evals/s03-03-eval-patterns"},{"id":"evals/writing-evals/s04-04-predict-the-pass-rate"},{"id":"evals/writing-evals/s05-05-fill-the-assertion"},{"id":"evals/writing-evals/s06-06-fix-the-flaky-eval"},{"id":"evals/writing-evals/s07-07-fix-the-overstrict-eval"},{"id":"evals/writing-evals/s08-08-write-the-suite"},{"id":"evals/writing-evals/s09-09-checkpoint"}]},{"slug":"llm-as-judge","steps":[{"id":"evals/llm-as-judge/s01-01-intro"},{"id":"evals/llm-as-judge/s02-02-pairwise-or-rubric"},{"id":"evals/llm-as-judge/s03-03-the-four-biases"},{"id":"evals/llm-as-judge/s04-04-predict-the-bias"},{"id":"evals/llm-as-judge/s05-05-fill-the-rubric-call"},{"id":"evals/llm-as-judge/s06-06-fix-the-likert-trap"},{"id":"evals/llm-as-judge/s07-07-fix-the-position-bias"},{"id":"evals/llm-as-judge/s08-08-write-the-rubric-judge"},{"id":"evals/llm-as-judge/s09-09-checkpoint"}]},{"slug":"the-rise-of-evals-as-a-discipline","steps":[{"id":"evals/the-rise-of-evals-as-a-discipline/s01-01-the-prompt-engineering-era"},{"id":"evals/the-rise-of-evals-as-a-discipline/s02-02-the-eval-turn"},{"id":"evals/the-rise-of-evals-as-a-discipline/s03-03-which-company-survives-a-model-swap"},{"id":"evals/the-rise-of-evals-as-a-discipline/s04-04-case-studies"},{"id":"evals/the-rise-of-evals-as-a-discipline/s05-05-eval-first-vs-prompt-first"},{"id":"evals/the-rise-of-evals-as-a-discipline/s06-06-why-the-eval-engineer-eats-the-prompt-engineer"},{"id":"evals/the-rise-of-evals-as-a-discipline/s07-07-write-the-eval-readiness-audit"},{"id":"evals/the-rise-of-evals-as-a-discipline/s08-08-checkpoint"}]},{"slug":"model-validation-gates","steps":[{"id":"evals/model-validation-gates/s01-01-intro"},{"id":"evals/model-validation-gates/s02-02-spot-the-theater"},{"id":"evals/model-validation-gates/s03-03-predict-the-decision"},{"id":"evals/model-validation-gates/s04-04-fill-the-regression-check"},{"id":"evals/model-validation-gates/s05-05-fix-the-theater-gate"},{"id":"evals/model-validation-gates/s06-06-write-the-gate-runner"},{"id":"evals/model-validation-gates/s07-07-checkpoint"}]}]},{"slug":"context-and-retrieval","lessons":[{"slug":"chunking-that-respects-structure","steps":[{"id":"context-and-retrieval/chunking-that-respects-structure/s01-01-intro"},{"id":"context-and-retrieval/chunking-that-respects-structure/s02-02-which-content-shape-breaks"},{"id":"context-and-retrieval/chunking-that-respects-structure/s03-03-the-recursive-splitter"},{"id":"context-and-retrieval/chunking-that-respects-structure/s04-04-predict-the-shred"},{"id":"context-and-retrieval/chunking-that-respects-structure/s05-05-fill-the-separator-list"},{"id":"context-and-retrieval/chunking-that-respects-structure/s06-06-fix-the-mid-sentence-cut"},{"id":"context-and-retrieval/chunking-that-respects-structure/s07-07-fix-the-missing-overlap"},{"id":"context-and-retrieval/chunking-that-respects-structure/s08-08-write-the-recursive-splitter"},{"id":"context-and-retrieval/chunking-that-respects-structure/s09-09-checkpoint"}]},{"slug":"embedding-that-fits-the-budget","steps":[{"id":"context-and-retrieval/embedding-that-fits-the-budget/s01-01-intro"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s02-02-which-model-fits"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s03-03-the-vector-space"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s04-04-predict-the-closest"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s05-05-fill-the-cosine-formula"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s06-06-fix-the-wrong-embedding-model"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s07-07-fix-the-missing-normalization"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s08-08-write-the-similarity-ranker"},{"id":"context-and-retrieval/embedding-that-fits-the-budget/s09-09-checkpoint"}]},{"slug":"retrieval-that-finds-the-right-thing","steps":[{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s01-01-intro"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s02-02-which-result-is-good"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s03-03-the-distance-threshold"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s04-04-predict-the-top-k"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s05-05-fill-the-cutoff"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s06-06-fix-the-no-threshold"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s07-07-fix-the-dup-results"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s08-08-write-the-ranker"},{"id":"context-and-retrieval/retrieval-that-finds-the-right-thing/s09-09-checkpoint"}]},{"slug":"rag-vs-long-context-vs-fine-tune","steps":[{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s01-01-the-three-way-fork"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s02-02-rag-when-you-need-freshness"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s03-03-which-fork-for-this-scenario"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s04-04-long-context-when-the-corpus-fits"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s05-05-fine-tune-when-style-or-format-is-the-product"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s06-06-which-fork-dies-first"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s07-07-graph-vector-or-both"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s08-08-the-rubric"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s09-09-write-the-fork-decision"},{"id":"context-and-retrieval/rag-vs-long-context-vs-fine-tune/s10-10-checkpoint"}]},{"slug":"retrieval-metrics-and-vector-db-shape","steps":[{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s01-01-the-eval-set"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s02-02-filter-or-search-first"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s03-03-predict-recall-precision"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s04-04-fill-the-reciprocal"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s05-05-fix-the-late-filter"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s06-06-write-recall-at-k"},{"id":"context-and-retrieval/retrieval-metrics-and-vector-db-shape/s07-07-checkpoint"}]}]},{"slug":"production-tradeoffs","lessons":[{"slug":"prompt-caching-correctly","steps":[{"id":"production-tradeoffs/prompt-caching-correctly/s01-01-intro"},{"id":"production-tradeoffs/prompt-caching-correctly/s02-02-which-piece-caches"},{"id":"production-tradeoffs/prompt-caching-correctly/s03-03-the-cache-breakpoints"},{"id":"production-tradeoffs/prompt-caching-correctly/s04-04-predict-the-savings"},{"id":"production-tradeoffs/prompt-caching-correctly/s05-05-fill-the-cache-control"},{"id":"production-tradeoffs/prompt-caching-correctly/s06-06-fix-the-cache-buster"},{"id":"production-tradeoffs/prompt-caching-correctly/s07-07-fix-the-ttl-default"},{"id":"production-tradeoffs/prompt-caching-correctly/s08-08-write-the-cost-estimator"},{"id":"production-tradeoffs/prompt-caching-correctly/s09-09-checkpoint"}]},{"slug":"read-the-token-bill","steps":[{"id":"production-tradeoffs/read-the-token-bill/s01-01-intro"},{"id":"production-tradeoffs/read-the-token-bill/s02-02-which-prompt-costs-more"},{"id":"production-tradeoffs/read-the-token-bill/s03-03-the-shape-of-a-bill"},{"id":"production-tradeoffs/read-the-token-bill/s04-04-predict-the-monthly-spend"},{"id":"production-tradeoffs/read-the-token-bill/s05-05-fill-the-cost-formula"},{"id":"production-tradeoffs/read-the-token-bill/s06-06-fix-the-pricing-ratio-error"},{"id":"production-tradeoffs/read-the-token-bill/s07-07-fix-the-units-error"},{"id":"production-tradeoffs/read-the-token-bill/s08-08-write-the-cost-estimator"},{"id":"production-tradeoffs/read-the-token-bill/s09-09-checkpoint"}]},{"slug":"the-model-price-war","steps":[{"id":"production-tradeoffs/the-model-price-war/s01-01-the-price-curve"},{"id":"production-tradeoffs/the-model-price-war/s02-02-three-bets-and-how-they-aged"},{"id":"production-tradeoffs/the-model-price-war/s03-03-which-startup-survived"},{"id":"production-tradeoffs/the-model-price-war/s04-04-the-incumbent-trap"},{"id":"production-tradeoffs/the-model-price-war/s05-05-features-now-viable"},{"id":"production-tradeoffs/the-model-price-war/s06-06-pricing-for-a-war-you-cant-predict"},{"id":"production-tradeoffs/the-model-price-war/s07-07-write-the-cost-model"},{"id":"production-tradeoffs/the-model-price-war/s08-08-checkpoint"}]},{"slug":"batch-realtime-costs","steps":[{"id":"production-tradeoffs/batch-realtime-costs/s01-01-who-is-waiting"},{"id":"production-tradeoffs/batch-realtime-costs/s02-02-which-lane"},{"id":"production-tradeoffs/batch-realtime-costs/s03-03-predict-the-bill"},{"id":"production-tradeoffs/batch-realtime-costs/s04-04-fill-the-miss-rate"},{"id":"production-tradeoffs/batch-realtime-costs/s05-05-fix-the-free-discount"},{"id":"production-tradeoffs/batch-realtime-costs/s06-06-write-monthly-bill"},{"id":"production-tradeoffs/batch-realtime-costs/s07-07-checkpoint"}]}]},{"slug":"debugging-output","lessons":[{"slug":"read-the-trace-not-the-chat","steps":[{"id":"debugging-output/read-the-trace-not-the-chat/s01-01-intro"},{"id":"debugging-output/read-the-trace-not-the-chat/s02-02-where-to-look-first"},{"id":"debugging-output/read-the-trace-not-the-chat/s03-03-the-four-failure-classes"},{"id":"debugging-output/read-the-trace-not-the-chat/s04-04-predict-the-bad-turn"},{"id":"debugging-output/read-the-trace-not-the-chat/s05-05-fill-the-find-bad-turn"},{"id":"debugging-output/read-the-trace-not-the-chat/s06-06-fix-the-fix-the-symptom"},{"id":"debugging-output/read-the-trace-not-the-chat/s07-07-fix-the-blame-the-model"},{"id":"debugging-output/read-the-trace-not-the-chat/s08-08-write-the-trace-summary"},{"id":"debugging-output/read-the-trace-not-the-chat/s09-09-checkpoint"}]},{"slug":"the-four-breakage-classes","steps":[{"id":"debugging-output/the-four-breakage-classes/s01-01-intro"},{"id":"debugging-output/the-four-breakage-classes/s02-02-classify-the-refund-bug"},{"id":"debugging-output/the-four-breakage-classes/s03-03-the-trace-anatomy"},{"id":"debugging-output/the-four-breakage-classes/s04-04-predict-the-class"},{"id":"debugging-output/the-four-breakage-classes/s05-05-fill-the-class-name"},{"id":"debugging-output/the-four-breakage-classes/s06-06-fix-the-misclassification"},{"id":"debugging-output/the-four-breakage-classes/s07-07-fix-the-parse-mangle"},{"id":"debugging-output/the-four-breakage-classes/s08-08-write-the-classifier"},{"id":"debugging-output/the-four-breakage-classes/s09-09-checkpoint"}]},{"slug":"five-real-postmortems","steps":[{"id":"debugging-output/five-real-postmortems/s01-01-the-postmortem-template"},{"id":"debugging-output/five-real-postmortems/s02-02-air-canada-refund-lawsuit"},{"id":"debugging-output/five-real-postmortems/s03-03-dpd-and-nyc-mycity"},{"id":"debugging-output/five-real-postmortems/s04-04-classify-the-five"},{"id":"debugging-output/five-real-postmortems/s05-05-recruiter-json-mangle"},{"id":"debugging-output/five-real-postmortems/s06-06-glean-style-retrieval-miss"},{"id":"debugging-output/five-real-postmortems/s07-07-which-fix-catches-the-most"},{"id":"debugging-output/five-real-postmortems/s08-08-write-the-postmortem"},{"id":"debugging-output/five-real-postmortems/s09-09-checkpoint"}]},{"slug":"prediction-failure-triage","steps":[{"id":"debugging-output/prediction-failure-triage/s01-01-reproduce-classify-route"},{"id":"debugging-output/prediction-failure-triage/s02-02-first-move"},{"id":"debugging-output/prediction-failure-triage/s03-03-predict-the-route"},{"id":"debugging-output/prediction-failure-triage/s04-04-fill-the-owners"},{"id":"debugging-output/prediction-failure-triage/s05-05-fix-the-paraphrase-repro"},{"id":"debugging-output/prediction-failure-triage/s06-06-write-the-router"},{"id":"debugging-output/prediction-failure-triage/s07-07-checkpoint"}]}]},{"slug":"capstone","lessons":[{"slug":"pick-a-project-that-ships","steps":[{"id":"capstone/pick-a-project-that-ships/s01-01-the-builders-graveyard"},{"id":"capstone/pick-a-project-that-ships/s02-02-which-project-survives-contact-with-reality"},{"id":"capstone/pick-a-project-that-ships/s03-03-the-three-shapes-that-ship"},{"id":"capstone/pick-a-project-that-ships/s04-04-the-wedge-checklist"},{"id":"capstone/pick-a-project-that-ships/s05-05-spot-the-missing-wedge"},{"id":"capstone/pick-a-project-that-ships/s06-06-scope-down-the-capstone"},{"id":"capstone/pick-a-project-that-ships/s07-07-checkpoint"}]},{"slug":"build-a-cli-agent","steps":[{"id":"capstone/build-a-cli-agent/s01-01-intro"},{"id":"capstone/build-a-cli-agent/s02-02-which-piece-is-missing"},{"id":"capstone/build-a-cli-agent/s03-03-the-tool-loop"},{"id":"capstone/build-a-cli-agent/s04-04-predict-the-final-answer"},{"id":"capstone/build-a-cli-agent/s05-05-fill-the-stop-check"},{"id":"capstone/build-a-cli-agent/s06-06-fix-the-tool-dispatch"},{"id":"capstone/build-a-cli-agent/s07-07-fix-the-message-append"},{"id":"capstone/build-a-cli-agent/s08-08-the-logging-pattern"},{"id":"capstone/build-a-cli-agent/s09-09-write-the-tool-runner"},{"id":"capstone/build-a-cli-agent/s10-10-fix-the-loop-exit"},{"id":"capstone/build-a-cli-agent/s11-11-write-the-agent"},{"id":"capstone/build-a-cli-agent/s12-12-checkpoint"}]},{"slug":"wire-the-real-model","steps":[{"id":"capstone/wire-the-real-model/s01-01-intro"},{"id":"capstone/wire-the-real-model/s02-02-where-the-key-lives"},{"id":"capstone/wire-the-real-model/s03-03-the-sdk-response-shape"},{"id":"capstone/wire-the-real-model/s04-04-predict-the-block-type"},{"id":"capstone/wire-the-real-model/s05-05-fill-the-key-load"},{"id":"capstone/wire-the-real-model/s06-06-fix-the-hardcoded-key"},{"id":"capstone/wire-the-real-model/s07-07-fix-the-block-access"},{"id":"capstone/wire-the-real-model/s08-08-write-the-shape-adapter"},{"id":"capstone/wire-the-real-model/s09-09-checkpoint"}]},{"slug":"validate-tool-inputs","steps":[{"id":"capstone/validate-tool-inputs/s01-01-intro"},{"id":"capstone/validate-tool-inputs/s02-02-which-input-needs-validation"},{"id":"capstone/validate-tool-inputs/s03-03-the-validator-surface"},{"id":"capstone/validate-tool-inputs/s04-04-predict-the-error"},{"id":"capstone/validate-tool-inputs/s05-05-fill-the-validate-call"},{"id":"capstone/validate-tool-inputs/s06-06-fix-the-untyped-tool"},{"id":"capstone/validate-tool-inputs/s07-07-fix-the-error-leak"},{"id":"capstone/validate-tool-inputs/s08-08-write-the-validator"},{"id":"capstone/validate-tool-inputs/s09-09-checkpoint"}]},{"slug":"add-evals-and-traces","steps":[{"id":"capstone/add-evals-and-traces/s01-01-intro"},{"id":"capstone/add-evals-and-traces/s02-02-which-eval-fits"},{"id":"capstone/add-evals-and-traces/s03-03-the-trace-shape"},{"id":"capstone/add-evals-and-traces/s04-04-predict-the-pass-rate"},{"id":"capstone/add-evals-and-traces/s05-05-fill-the-trace-append"},{"id":"capstone/add-evals-and-traces/s06-06-fix-the-trace-mutation"},{"id":"capstone/add-evals-and-traces/s07-07-fix-the-tautological-eval"},{"id":"capstone/add-evals-and-traces/s08-08-write-the-eval-runner"},{"id":"capstone/add-evals-and-traces/s09-09-checkpoint"}]},{"slug":"wire-an-mcp-tool","steps":[{"id":"capstone/wire-an-mcp-tool/s01-01-intro"},{"id":"capstone/wire-an-mcp-tool/s02-02-when-to-use-mcp"},{"id":"capstone/wire-an-mcp-tool/s03-03-the-mcp-protocol"},{"id":"capstone/wire-an-mcp-tool/s04-04-predict-the-tool-list"},{"id":"capstone/wire-an-mcp-tool/s05-05-fill-the-tools-call"},{"id":"capstone/wire-an-mcp-tool/s06-06-fix-the-result-shape"},{"id":"capstone/wire-an-mcp-tool/s07-07-fix-the-iserror-flag"},{"id":"capstone/wire-an-mcp-tool/s08-08-write-the-mcp-bridge"},{"id":"capstone/wire-an-mcp-tool/s09-09-checkpoint"}]},{"slug":"portfolio-ml-system-option","steps":[{"id":"capstone/portfolio-ml-system-option/s01-01-the-other-capstone"},{"id":"capstone/portfolio-ml-system-option/s02-02-what-reviewers-open"},{"id":"capstone/portfolio-ml-system-option/s03-03-predict-the-gate"},{"id":"capstone/portfolio-ml-system-option/s04-04-fill-the-gate"},{"id":"capstone/portfolio-ml-system-option/s05-05-fix-the-train-accuracy-gate"},{"id":"capstone/portfolio-ml-system-option/s06-06-write-the-scope-check"},{"id":"capstone/portfolio-ml-system-option/s07-07-checkpoint"}]}]},{"slug":"agent-harnesses","lessons":[{"slug":"project-workspaces-not-product-names","steps":[{"id":"agent-harnesses/project-workspaces-not-product-names/s01-01-intro"}]},{"slug":"what-a-harness-is","steps":[{"id":"agent-harnesses/what-a-harness-is/s01-01-intro"},{"id":"agent-harnesses/what-a-harness-is/s02-02-which-is-a-harness"},{"id":"agent-harnesses/what-a-harness-is/s03-03-the-four-layers"},{"id":"agent-harnesses/what-a-harness-is/s04-04-predict-the-layer"},{"id":"agent-harnesses/what-a-harness-is/s05-05-fill-the-dispatch"},{"id":"agent-harnesses/what-a-harness-is/s06-06-fix-the-no-retry"},{"id":"agent-harnesses/what-a-harness-is/s07-07-fix-the-hardcoded-registry"},{"id":"agent-harnesses/what-a-harness-is/s08-08-write-mini-harness"},{"id":"agent-harnesses/what-a-harness-is/s09-09-checkpoint"}]},{"slug":"architecting-an-ai-native-workflow","steps":[{"id":"agent-harnesses/architecting-an-ai-native-workflow/s01-01-intro"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s02-02-which-workflow-fits"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s03-03-mapping-the-workflow"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s04-04-predict-the-readiness-score"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s05-05-fill-the-required-fields"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s06-06-fix-the-implicit-policy"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s07-07-fix-the-wrong-metric"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s08-08-write-the-workflow-validator"},{"id":"agent-harnesses/architecting-an-ai-native-workflow/s09-09-checkpoint"}]},{"slug":"five-industries-walked-through","steps":[{"id":"agent-harnesses/five-industries-walked-through/s01-01-the-case-study-method"},{"id":"agent-harnesses/five-industries-walked-through/s02-02-home-services"},{"id":"agent-harnesses/five-industries-walked-through/s03-03-which-task-belongs-to-the-agent"},{"id":"agent-harnesses/five-industries-walked-through/s04-04-insurance-brokerage"},{"id":"agent-harnesses/five-industries-walked-through/s05-05-recruiting-firm"},{"id":"agent-harnesses/five-industries-walked-through/s06-06-which-industry-wedges-fastest"},{"id":"agent-harnesses/five-industries-walked-through/s07-07-the-incumbent-trap"},{"id":"agent-harnesses/five-industries-walked-through/s08-08-write-the-process-debt-audit"},{"id":"agent-harnesses/five-industries-walked-through/s09-09-checkpoint"}]}]},{"slug":"ai-image-generation","lessons":[{"slug":"the-image-model-landscape","steps":[{"id":"ai-image-generation/the-image-model-landscape/s01-01-the-six-families"},{"id":"ai-image-generation/the-image-model-landscape/s02-02-flux-vs-midjourney"},{"id":"ai-image-generation/the-image-model-landscape/s03-03-pick-the-model"},{"id":"ai-image-generation/the-image-model-landscape/s04-04-nano-banana-and-the-batch-economics"},{"id":"ai-image-generation/the-image-model-landscape/s05-05-text-in-image"},{"id":"ai-image-generation/the-image-model-landscape/s06-06-which-model-for-text"},{"id":"ai-image-generation/the-image-model-landscape/s07-07-the-decision-tree"},{"id":"ai-image-generation/the-image-model-landscape/s08-08-write-pick-image-model"},{"id":"ai-image-generation/the-image-model-landscape/s09-09-checkpoint"}]},{"slug":"prompting-for-real-output","steps":[{"id":"ai-image-generation/prompting-for-real-output/s01-01-why-the-girl-with-red-hair-fails"},{"id":"ai-image-generation/prompting-for-real-output/s02-02-the-eight-knobs"},{"id":"ai-image-generation/prompting-for-real-output/s03-03-fix-the-vague-prompt"},{"id":"ai-image-generation/prompting-for-real-output/s04-04-controlnet-and-reference"},{"id":"ai-image-generation/prompting-for-real-output/s05-05-which-knob-is-missing"},{"id":"ai-image-generation/prompting-for-real-output/s06-06-failure-modes"},{"id":"ai-image-generation/prompting-for-real-output/s07-07-write-score-image-prompt"},{"id":"ai-image-generation/prompting-for-real-output/s08-08-checkpoint"}]},{"slug":"the-image-pipeline","steps":[{"id":"ai-image-generation/the-image-pipeline/s01-01-the-six-stages"},{"id":"ai-image-generation/the-image-pipeline/s02-02-batch-and-filter"},{"id":"ai-image-generation/the-image-pipeline/s03-03-which-stage-is-broken"},{"id":"ai-image-generation/the-image-pipeline/s04-04-format-and-platform"},{"id":"ai-image-generation/the-image-pipeline/s05-05-fill-the-pipeline"},{"id":"ai-image-generation/the-image-pipeline/s06-06-write-cost-for-batch"},{"id":"ai-image-generation/the-image-pipeline/s07-07-checkpoint"}]}]},{"slug":"ai-video-generation","lessons":[{"slug":"the-video-model-lineup","steps":[{"id":"ai-video-generation/the-video-model-lineup/s01-01-the-landscape"},{"id":"ai-video-generation/the-video-model-lineup/s02-02-sora-and-veo"},{"id":"ai-video-generation/the-video-model-lineup/s03-03-higgsfield-and-camera-control"},{"id":"ai-video-generation/the-video-model-lineup/s04-04-the-second-wave"},{"id":"ai-video-generation/the-video-model-lineup/s05-05-text-vs-image-vs-video"},{"id":"ai-video-generation/the-video-model-lineup/s06-06-which-lane"},{"id":"ai-video-generation/the-video-model-lineup/s07-07-pick-the-model"},{"id":"ai-video-generation/the-video-model-lineup/s08-08-write-pick-video-model"},{"id":"ai-video-generation/the-video-model-lineup/s09-09-checkpoint"}]},{"slug":"camera-control-and-motion","steps":[{"id":"ai-video-generation/camera-control-and-motion/s01-01-the-static-camera-problem"},{"id":"ai-video-generation/camera-control-and-motion/s02-02-shot-vocabulary"},{"id":"ai-video-generation/camera-control-and-motion/s03-03-which-shot-is-which"},{"id":"ai-video-generation/camera-control-and-motion/s04-04-camera-moves"},{"id":"ai-video-generation/camera-control-and-motion/s05-05-the-i2v-keyframe-pattern"},{"id":"ai-video-generation/camera-control-and-motion/s06-06-spot-the-generic-prompt"},{"id":"ai-video-generation/camera-control-and-motion/s07-07-write-audit-shot-list"},{"id":"ai-video-generation/camera-control-and-motion/s08-08-checkpoint"}]},{"slug":"the-cost-math","steps":[{"id":"ai-video-generation/the-cost-math/s01-01-the-unit-economics"},{"id":"ai-video-generation/the-cost-math/s02-02-the-retake-rate-reality"},{"id":"ai-video-generation/the-cost-math/s03-03-which-tier-for-this-budget"},{"id":"ai-video-generation/the-cost-math/s04-04-the-60-second-promo"},{"id":"ai-video-generation/the-cost-math/s05-05-write-cost-per-minute"},{"id":"ai-video-generation/the-cost-math/s06-06-checkpoint"}]}]},{"slug":"programmatic-design","lessons":[{"slug":"why-programmatic","steps":[{"id":"programmatic-design/why-programmatic/s01-01-intro"},{"id":"programmatic-design/why-programmatic/s02-02-the-three-jobs"},{"id":"programmatic-design/why-programmatic/s03-03-which-task-is-code"},{"id":"programmatic-design/why-programmatic/s04-04-parametric-vs-one-shot"},{"id":"programmatic-design/why-programmatic/s05-05-the-classifier"},{"id":"programmatic-design/why-programmatic/s06-06-write-classify-video-task"},{"id":"programmatic-design/why-programmatic/s07-07-checkpoint"}]},{"slug":"hyperframes-and-remotion","steps":[{"id":"programmatic-design/hyperframes-and-remotion/s01-01-the-two-tools"},{"id":"programmatic-design/hyperframes-and-remotion/s02-02-hyperframes-tour"},{"id":"programmatic-design/hyperframes-and-remotion/s03-03-hyperframes-cli"},{"id":"programmatic-design/hyperframes-and-remotion/s04-04-remotion-tour"},{"id":"programmatic-design/hyperframes-and-remotion/s05-05-the-decision-tree"},{"id":"programmatic-design/hyperframes-and-remotion/s06-06-pick-the-tool"},{"id":"programmatic-design/hyperframes-and-remotion/s07-07-claude-design-html"},{"id":"programmatic-design/hyperframes-and-remotion/s08-08-write-pick-tool"},{"id":"programmatic-design/hyperframes-and-remotion/s09-09-checkpoint"}]},{"slug":"the-ai-native-design-pipeline","steps":[{"id":"programmatic-design/the-ai-native-design-pipeline/s01-01-the-seven-steps"},{"id":"programmatic-design/the-ai-native-design-pipeline/s02-02-where-ai-fits-each-step"},{"id":"programmatic-design/the-ai-native-design-pipeline/s03-03-which-model-which-step"},{"id":"programmatic-design/the-ai-native-design-pipeline/s04-04-website-to-hyperframes"},{"id":"programmatic-design/the-ai-native-design-pipeline/s05-05-the-cost-model"},{"id":"programmatic-design/the-ai-native-design-pipeline/s06-06-score-the-pipeline"},{"id":"programmatic-design/the-ai-native-design-pipeline/s07-07-write-score-pipeline"},{"id":"programmatic-design/the-ai-native-design-pipeline/s08-08-checkpoint"}]}]},{"slug":"harness-engineering","lessons":[{"slug":"the-harness-engineering-mindset","steps":[{"id":"harness-engineering/the-harness-engineering-mindset/s01-01-the-model-is-only-one-input"},{"id":"harness-engineering/the-harness-engineering-mindset/s02-02-the-six-pieces-of-a-harness"},{"id":"harness-engineering/the-harness-engineering-mindset/s03-03-which-piece-is-missing"},{"id":"harness-engineering/the-harness-engineering-mindset/s04-04-the-skill-issue-reframe"},{"id":"harness-engineering/the-harness-engineering-mindset/s05-05-pick-the-fix"},{"id":"harness-engineering/the-harness-engineering-mindset/s06-06-the-harness-gap"},{"id":"harness-engineering/the-harness-engineering-mindset/s07-07-write-the-harness-inventory"},{"id":"harness-engineering/the-harness-engineering-mindset/s08-08-checkpoint"}]},{"slug":"the-ratchet","steps":[{"id":"harness-engineering/the-ratchet/s01-01-failures-are-signals-not-flukes"},{"id":"harness-engineering/the-ratchet/s02-02-every-line-traces-to-a-failure"},{"id":"harness-engineering/the-ratchet/s03-03-which-rule-earned-its-line"},{"id":"harness-engineering/the-ratchet/s04-04-working-backwards-from-behavior"},{"id":"harness-engineering/the-ratchet/s05-05-the-ratchet-log"},{"id":"harness-engineering/the-ratchet/s06-06-spot-the-cruft"},{"id":"harness-engineering/the-ratchet/s07-07-write-the-ratchet-update"},{"id":"harness-engineering/the-ratchet/s08-08-audit-the-ratchet"},{"id":"harness-engineering/the-ratchet/s09-09-checkpoint"}]},{"slug":"context-engineering","steps":[{"id":"harness-engineering/context-engineering/s01-01-the-context-rot"},{"id":"harness-engineering/context-engineering/s02-02-compaction"},{"id":"harness-engineering/context-engineering/s03-03-tool-call-offloading"},{"id":"harness-engineering/context-engineering/s04-04-progressive-disclosure"},{"id":"harness-engineering/context-engineering/s05-05-which-technique-fits"},{"id":"harness-engineering/context-engineering/s06-06-the-memory-hierarchy"},{"id":"harness-engineering/context-engineering/s07-07-classify-the-data"},{"id":"harness-engineering/context-engineering/s08-08-write-the-context-budget"},{"id":"harness-engineering/context-engineering/s09-09-checkpoint"}]},{"slug":"long-horizon-execution","steps":[{"id":"harness-engineering/long-horizon-execution/s01-01-the-40-step-problem"},{"id":"harness-engineering/long-horizon-execution/s02-02-loops-intercepting-early-stops"},{"id":"harness-engineering/long-horizon-execution/s03-03-planning-as-a-separate-step"},{"id":"harness-engineering/long-horizon-execution/s04-04-splits-generation-vs-evaluation"},{"id":"harness-engineering/long-horizon-execution/s05-05-which-pattern-fixes-the-failure"},{"id":"harness-engineering/long-horizon-execution/s06-06-hooks-as-the-enforcement-layer"},{"id":"harness-engineering/long-horizon-execution/s07-07-the-type-check-backpressure-pattern"},{"id":"harness-engineering/long-horizon-execution/s08-08-write-the-hook-router"},{"id":"harness-engineering/long-horizon-execution/s09-09-write-the-completion-guard"},{"id":"harness-engineering/long-horizon-execution/s10-10-checkpoint"}]},{"slug":"the-haas-shift","steps":[{"id":"harness-engineering/the-haas-shift/s01-01-llm-apis-to-harness-apis"},{"id":"harness-engineering/the-haas-shift/s02-02-the-convergence"},{"id":"harness-engineering/the-haas-shift/s03-03-which-haas-fits"},{"id":"harness-engineering/the-haas-shift/s04-04-harnesses-dont-shrink-they-move"},{"id":"harness-engineering/the-haas-shift/s05-05-the-overfitting-feedback-loop"},{"id":"harness-engineering/the-haas-shift/s06-06-mcp-security-as-prompt-injection"},{"id":"harness-engineering/the-haas-shift/s07-07-which-component-is-debt"},{"id":"harness-engineering/the-haas-shift/s08-08-write-the-build-vs-buy"},{"id":"harness-engineering/the-haas-shift/s09-09-checkpoint"}]},{"slug":"pipeline-boundaries","steps":[{"id":"harness-engineering/pipeline-boundaries/s01-01-the-boundary-is-an-artifact"},{"id":"harness-engineering/pipeline-boundaries/s02-02-spot-the-risky-handoff"},{"id":"harness-engineering/pipeline-boundaries/s03-03-predict-the-gate"},{"id":"harness-engineering/pipeline-boundaries/s04-04-fill-the-gate"},{"id":"harness-engineering/pipeline-boundaries/s05-05-fix-the-overtrust"},{"id":"harness-engineering/pipeline-boundaries/s06-06-write-the-advance-check"},{"id":"harness-engineering/pipeline-boundaries/s07-07-checkpoint"}]}]},{"slug":"intro-to-terminal","lessons":[{"slug":"what-the-terminal-is","steps":[{"id":"intro-to-terminal/what-the-terminal-is/s01-01-not-a-cockpit"},{"id":"intro-to-terminal/what-the-terminal-is/s02-02-what-it-actually-is"},{"id":"intro-to-terminal/what-the-terminal-is/s03-03-open-it"},{"id":"intro-to-terminal/what-the-terminal-is/s04-04-which-app"},{"id":"intro-to-terminal/what-the-terminal-is/s05-05-the-prompt"},{"id":"intro-to-terminal/what-the-terminal-is/s06-06-what-a-command-is"},{"id":"intro-to-terminal/what-the-terminal-is/s07-07-checkpoint"}]},{"slug":"moving-around","steps":[{"id":"intro-to-terminal/moving-around/s01-01-the-filesystem-is-a-tree"},{"id":"intro-to-terminal/moving-around/s02-02-pwd-and-ls"},{"id":"intro-to-terminal/moving-around/s03-03-where-am-i"},{"id":"intro-to-terminal/moving-around/s04-04-read-this-listing"},{"id":"intro-to-terminal/moving-around/s05-05-cd"},{"id":"intro-to-terminal/moving-around/s06-06-follow-the-moves"},{"id":"intro-to-terminal/moving-around/s07-07-pick-the-command"},{"id":"intro-to-terminal/moving-around/s08-08-checkpoint"}]},{"slug":"making-things","steps":[{"id":"intro-to-terminal/making-things/s01-01-mkdir-and-touch"},{"id":"intro-to-terminal/making-things/s02-02-folder-or-file"},{"id":"intro-to-terminal/making-things/s03-03-cat-and-echo"},{"id":"intro-to-terminal/making-things/s04-04-reading-a-file"},{"id":"intro-to-terminal/making-things/s05-05-rm-carefully"},{"id":"intro-to-terminal/making-things/s06-06-the-no-undo-rule"},{"id":"intro-to-terminal/making-things/s07-07-string-it-together"},{"id":"intro-to-terminal/making-things/s08-08-checkpoint"}]}]},{"slug":"intro-to-claude-cli","lessons":[{"slug":"what-the-claude-cli-is","steps":[{"id":"intro-to-claude-cli/what-the-claude-cli-is/s01-01-the-glass-is-gone"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s02-02-chat-vs-cli"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s03-03-which-tool"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s04-04-the-agent-loop"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s05-05-read-the-loop"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s06-06-permission-and-limits"},{"id":"intro-to-claude-cli/what-the-claude-cli-is/s07-07-checkpoint"}]},{"slug":"install-and-sign-in","steps":[{"id":"intro-to-claude-cli/install-and-sign-in/s01-01-two-ways-to-install"},{"id":"intro-to-claude-cli/install-and-sign-in/s02-02-run-the-installer"},{"id":"intro-to-claude-cli/install-and-sign-in/s03-03-install-check"},{"id":"intro-to-claude-cli/install-and-sign-in/s04-04-sign-in"},{"id":"intro-to-claude-cli/install-and-sign-in/s05-05-what-sign-in-did"},{"id":"intro-to-claude-cli/install-and-sign-in/s06-06-auth-check"},{"id":"intro-to-claude-cli/install-and-sign-in/s07-07-account-not-api-key"},{"id":"intro-to-claude-cli/install-and-sign-in/s08-08-checkpoint"}]},{"slug":"your-first-session","steps":[{"id":"intro-to-claude-cli/your-first-session/s01-01-start-in-a-folder"},{"id":"intro-to-claude-cli/your-first-session/s02-02-your-first-instruction"},{"id":"intro-to-claude-cli/your-first-session/s03-03-watch-the-loop"},{"id":"intro-to-claude-cli/your-first-session/s04-04-read-what-it-did"},{"id":"intro-to-claude-cli/your-first-session/s05-05-one-shot-or-interactive"},{"id":"intro-to-claude-cli/your-first-session/s06-06-chat-or-cli-decision"},{"id":"intro-to-claude-cli/your-first-session/s07-07-where-this-goes"},{"id":"intro-to-claude-cli/your-first-session/s08-08-checkpoint"}]}]},{"slug":"intro-to-codex-cli","lessons":[{"slug":"another-tool-same-idea","steps":[{"id":"intro-to-codex-cli/another-tool-same-idea/s01-01-why-a-second-tool"},{"id":"intro-to-codex-cli/another-tool-same-idea/s02-02-what-carries-over"},{"id":"intro-to-codex-cli/another-tool-same-idea/s03-03-what-carries-over-check"},{"id":"intro-to-codex-cli/another-tool-same-idea/s04-04-what-is-different"},{"id":"intro-to-codex-cli/another-tool-same-idea/s05-05-spot-the-difference"},{"id":"intro-to-codex-cli/another-tool-same-idea/s06-06-when-to-reach-for-which"},{"id":"intro-to-codex-cli/another-tool-same-idea/s07-07-checkpoint"}]},{"slug":"install-and-sign-in","steps":[{"id":"intro-to-codex-cli/install-and-sign-in/s01-01-the-install-paths"},{"id":"intro-to-codex-cli/install-and-sign-in/s02-02-installing-it"},{"id":"intro-to-codex-cli/install-and-sign-in/s03-03-verify-it"},{"id":"intro-to-codex-cli/install-and-sign-in/s04-04-install-check"},{"id":"intro-to-codex-cli/install-and-sign-in/s05-05-sign-in"},{"id":"intro-to-codex-cli/install-and-sign-in/s06-06-account-vs-api-key"},{"id":"intro-to-codex-cli/install-and-sign-in/s07-07-auth-check"},{"id":"intro-to-codex-cli/install-and-sign-in/s08-08-checkpoint"}]},{"slug":"your-first-session","steps":[{"id":"intro-to-codex-cli/your-first-session/s01-01-start-and-instruct"},{"id":"intro-to-codex-cli/your-first-session/s02-02-the-loop-again"},{"id":"intro-to-codex-cli/your-first-session/s03-03-read-what-it-did"},{"id":"intro-to-codex-cli/your-first-session/s04-04-usage-patterns"},{"id":"intro-to-codex-cli/your-first-session/s05-05-usage-scenario"},{"id":"intro-to-codex-cli/your-first-session/s06-06-running-both"},{"id":"intro-to-codex-cli/your-first-session/s07-07-where-this-goes"},{"id":"intro-to-codex-cli/your-first-session/s08-08-checkpoint"}]}]},{"slug":"team-skills","lessons":[{"slug":"what-a-skill-is","steps":[{"id":"team-skills/what-a-skill-is/s01-01-the-re-explaining-problem"},{"id":"team-skills/what-a-skill-is/s02-02-a-skill-defined"},{"id":"team-skills/what-a-skill-is/s03-03-a-real-skill"},{"id":"team-skills/what-a-skill-is/s04-04-what-a-skill-is-check"},{"id":"team-skills/what-a-skill-is/s05-05-skill-vs-prompt-vs-agent"},{"id":"team-skills/what-a-skill-is/s06-06-skill-or-agent"},{"id":"team-skills/what-a-skill-is/s07-07-checkpoint"}]},{"slug":"skills-as-a-team-thing","steps":[{"id":"team-skills/skills-as-a-team-thing/s01-01-the-team-problem"},{"id":"team-skills/skills-as-a-team-thing/s02-02-org-wide-skills"},{"id":"team-skills/skills-as-a-team-thing/s03-03-sharing-and-the-directory"},{"id":"team-skills/skills-as-a-team-thing/s04-04-deployment-check"},{"id":"team-skills/skills-as-a-team-thing/s05-05-collaborating-through-skills"},{"id":"team-skills/skills-as-a-team-thing/s06-06-sharing-scenario"},{"id":"team-skills/skills-as-a-team-thing/s07-07-checkpoint"}]},{"slug":"role-playbooks","steps":[{"id":"team-skills/role-playbooks/s01-01-hr-skills"},{"id":"team-skills/role-playbooks/s02-02-legal-skills"},{"id":"team-skills/role-playbooks/s03-03-operations-skills"},{"id":"team-skills/role-playbooks/s04-04-match-the-skill"},{"id":"team-skills/role-playbooks/s05-05-skill-or-one-off"},{"id":"team-skills/role-playbooks/s06-06-skill-or-one-off-check"},{"id":"team-skills/role-playbooks/s07-07-checkpoint"}]},{"slug":"governance-and-trust","steps":[{"id":"team-skills/governance-and-trust/s01-01-a-skill-can-do-real-things"},{"id":"team-skills/governance-and-trust/s02-02-treat-it-like-software"},{"id":"team-skills/governance-and-trust/s03-03-evaluate-before-you-deploy"},{"id":"team-skills/governance-and-trust/s04-04-access-control"},{"id":"team-skills/governance-and-trust/s05-05-governance-scenario"},{"id":"team-skills/governance-and-trust/s06-06-a-rollout-checklist"},{"id":"team-skills/governance-and-trust/s07-07-checkpoint"}]}]},{"slug":"dataframes-numpy-pandas","lessons":[{"slug":"ml-package-map","steps":[{"id":"dataframes-numpy-pandas/ml-package-map/s01-01-intro"},{"id":"dataframes-numpy-pandas/ml-package-map/s02-02-match-the-tool"},{"id":"dataframes-numpy-pandas/ml-package-map/s03-03-predict-sys-modules"},{"id":"dataframes-numpy-pandas/ml-package-map/s04-04-fill-the-registry-key"},{"id":"dataframes-numpy-pandas/ml-package-map/s05-05-fix-the-swapped-map"},{"id":"dataframes-numpy-pandas/ml-package-map/s06-06-write-find-unused"},{"id":"dataframes-numpy-pandas/ml-package-map/s07-07-checkpoint"}]},{"slug":"array-and-series-shapes","steps":[{"id":"dataframes-numpy-pandas/array-and-series-shapes/s01-01-intro"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s02-02-choose-the-risk"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s03-03-predict-the-checks"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s04-04-fill-the-return"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s05-05-fix-the-overtrust"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s06-06-write-count-ready"},{"id":"dataframes-numpy-pandas/array-and-series-shapes/s07-07-checkpoint"}]},{"slug":"dataframe-selection-and-cleaning","steps":[{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s01-01-intro"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s02-02-choose-the-risk"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s03-03-predict-the-checks"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s04-04-fill-the-return"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s05-05-fix-the-overtrust"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s06-06-write-count-ready"},{"id":"dataframes-numpy-pandas/dataframe-selection-and-cleaning/s07-07-checkpoint"}]},{"slug":"missing-values-and-types","steps":[{"id":"dataframes-numpy-pandas/missing-values-and-types/s01-01-intro"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s02-02-choose-the-risk"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s03-03-predict-the-checks"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s04-04-fill-the-return"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s05-05-fix-the-overtrust"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s06-06-write-count-ready"},{"id":"dataframes-numpy-pandas/missing-values-and-types/s07-07-checkpoint"}]},{"slug":"groupby-joins-and-features","steps":[{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s01-01-intro"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s02-02-choose-the-risk"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s03-03-predict-the-checks"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s04-04-fill-the-return"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s05-05-fix-the-overtrust"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s06-06-write-count-ready"},{"id":"dataframes-numpy-pandas/groupby-joins-and-features/s07-07-checkpoint"}]},{"slug":"mission-api-to-dataframe","steps":[{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s01-01-intro"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s02-02-choose-the-risk"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s03-03-predict-the-checks"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s04-04-fill-the-return"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s05-05-fix-the-overtrust"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s06-06-write-count-ready"},{"id":"dataframes-numpy-pandas/mission-api-to-dataframe/s07-07-checkpoint"}]}]},{"slug":"sql-for-ml-datasets","lessons":[{"slug":"select-filter-aggregate","steps":[{"id":"sql-for-ml-datasets/select-filter-aggregate/s01-01-intro"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s02-02-choose-the-risk"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s03-03-predict-the-checks"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s04-04-fill-the-return"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s05-05-fix-the-overtrust"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s06-06-write-count-ready"},{"id":"sql-for-ml-datasets/select-filter-aggregate/s07-07-checkpoint"}]},{"slug":"joins-and-label-windows","steps":[{"id":"sql-for-ml-datasets/joins-and-label-windows/s01-01-intro"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s02-02-choose-the-risk"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s03-03-predict-the-checks"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s04-04-fill-the-return"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s05-05-fix-the-overtrust"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s06-06-write-count-ready"},{"id":"sql-for-ml-datasets/joins-and-label-windows/s07-07-checkpoint"}]},{"slug":"sql-quality-checks","steps":[{"id":"sql-for-ml-datasets/sql-quality-checks/s01-01-intro"},{"id":"sql-for-ml-datasets/sql-quality-checks/s02-02-choose-the-risk"},{"id":"sql-for-ml-datasets/sql-quality-checks/s03-03-predict-the-checks"},{"id":"sql-for-ml-datasets/sql-quality-checks/s04-04-fill-the-return"},{"id":"sql-for-ml-datasets/sql-quality-checks/s05-05-fix-the-overtrust"},{"id":"sql-for-ml-datasets/sql-quality-checks/s06-06-write-count-ready"},{"id":"sql-for-ml-datasets/sql-quality-checks/s07-07-checkpoint"}]},{"slug":"mission-sql-feature-query-lab","steps":[{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s01-01-intro"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s02-02-choose-the-risk"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s03-03-predict-the-checks"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s04-04-fill-the-return"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s05-05-fix-the-overtrust"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s06-06-write-count-ready"},{"id":"sql-for-ml-datasets/mission-sql-feature-query-lab/s07-07-checkpoint"}]}]},{"slug":"dataset-pipelines","lessons":[{"slug":"csv-jsonl-parquet-tradeoffs","steps":[{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s01-01-intro"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s02-02-choose-the-risk"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s03-03-predict-the-checks"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s04-04-fill-the-return"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s05-05-fix-the-overtrust"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s06-06-write-count-ready"},{"id":"dataset-pipelines/csv-jsonl-parquet-tradeoffs/s07-07-checkpoint"}]},{"slug":"pagination-retries-checkpoints","steps":[{"id":"dataset-pipelines/pagination-retries-checkpoints/s01-01-intro"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s02-02-choose-the-risk"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s03-03-predict-the-checks"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s04-04-fill-the-return"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s05-05-fix-the-overtrust"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s06-06-write-count-ready"},{"id":"dataset-pipelines/pagination-retries-checkpoints/s07-07-checkpoint"}]},{"slug":"schema-drift-and-contracts","steps":[{"id":"dataset-pipelines/schema-drift-and-contracts/s01-01-intro"},{"id":"dataset-pipelines/schema-drift-and-contracts/s02-02-choose-the-risk"},{"id":"dataset-pipelines/schema-drift-and-contracts/s03-03-predict-the-checks"},{"id":"dataset-pipelines/schema-drift-and-contracts/s04-04-fill-the-return"},{"id":"dataset-pipelines/schema-drift-and-contracts/s05-05-fix-the-overtrust"},{"id":"dataset-pipelines/schema-drift-and-contracts/s06-06-write-count-ready"},{"id":"dataset-pipelines/schema-drift-and-contracts/s07-07-checkpoint"}]},{"slug":"pipeline-orchestration-shape","steps":[{"id":"dataset-pipelines/pipeline-orchestration-shape/s01-01-intro"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s02-02-choose-the-risk"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s03-03-predict-the-checks"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s04-04-fill-the-return"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s05-05-fix-the-overtrust"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s06-06-write-count-ready"},{"id":"dataset-pipelines/pipeline-orchestration-shape/s07-07-checkpoint"}]},{"slug":"mission-api-to-dataset-pipeline","steps":[{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s01-01-intro"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s02-02-choose-the-risk"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s03-03-predict-the-checks"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s04-04-fill-the-return"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s05-05-fix-the-overtrust"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s06-06-write-count-ready"},{"id":"dataset-pipelines/mission-api-to-dataset-pipeline/s07-07-checkpoint"}]}]},{"slug":"ml-math-and-stats","lessons":[{"slug":"vectors-matrices-dot-products","steps":[{"id":"ml-math-and-stats/vectors-matrices-dot-products/s01-01-intro"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s02-02-choose-the-risk"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s03-03-predict-the-checks"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s04-04-fill-the-return"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s05-05-fix-the-overtrust"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s06-06-write-count-ready"},{"id":"ml-math-and-stats/vectors-matrices-dot-products/s07-07-checkpoint"}]},{"slug":"probability-and-baselines","steps":[{"id":"ml-math-and-stats/probability-and-baselines/s01-01-intro"},{"id":"ml-math-and-stats/probability-and-baselines/s02-02-choose-the-risk"},{"id":"ml-math-and-stats/probability-and-baselines/s03-03-predict-the-checks"},{"id":"ml-math-and-stats/probability-and-baselines/s04-04-fill-the-return"},{"id":"ml-math-and-stats/probability-and-baselines/s05-05-fix-the-overtrust"},{"id":"ml-math-and-stats/probability-and-baselines/s06-06-write-count-ready"},{"id":"ml-math-and-stats/probability-and-baselines/s07-07-checkpoint"}]},{"slug":"distributions-sampling-variance","steps":[{"id":"ml-math-and-stats/distributions-sampling-variance/s01-01-intro"},{"id":"ml-math-and-stats/distributions-sampling-variance/s02-02-choose-the-risk"},{"id":"ml-math-and-stats/distributions-sampling-variance/s03-03-predict-the-checks"},{"id":"ml-math-and-stats/distributions-sampling-variance/s04-04-fill-the-return"},{"id":"ml-math-and-stats/distributions-sampling-variance/s05-05-fix-the-overtrust"},{"id":"ml-math-and-stats/distributions-sampling-variance/s06-06-write-variance"},{"id":"ml-math-and-stats/distributions-sampling-variance/s07-07-checkpoint"}]},{"slug":"bias-variance-and-leakage","steps":[{"id":"ml-math-and-stats/bias-variance-and-leakage/s01-01-intro"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s02-02-choose-the-risk"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s03-03-predict-the-checks"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s04-04-fill-the-return"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s05-05-fix-the-overtrust"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s06-06-write-diagnose"},{"id":"ml-math-and-stats/bias-variance-and-leakage/s07-07-checkpoint"}]},{"slug":"mission-statistical-sanity-check","steps":[{"id":"ml-math-and-stats/mission-statistical-sanity-check/s01-01-intro"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s02-02-choose-the-risk"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s03-03-predict-the-checks"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s04-04-fill-the-return"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s05-05-fix-the-overtrust"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s06-06-write-count-ready"},{"id":"ml-math-and-stats/mission-statistical-sanity-check/s07-07-checkpoint"}]}]},{"slug":"supervised-learning-workflows","lessons":[{"slug":"labels-features-splits","steps":[{"id":"supervised-learning-workflows/labels-features-splits/s01-01-intro"},{"id":"supervised-learning-workflows/labels-features-splits/s02-02-choose-the-risk"},{"id":"supervised-learning-workflows/labels-features-splits/s03-03-predict-the-checks"},{"id":"supervised-learning-workflows/labels-features-splits/s04-04-fill-the-return"},{"id":"supervised-learning-workflows/labels-features-splits/s05-05-fix-the-overtrust"},{"id":"supervised-learning-workflows/labels-features-splits/s06-06-write-split-audit"},{"id":"supervised-learning-workflows/labels-features-splits/s07-07-checkpoint"}]},{"slug":"baselines-before-fancy-models","steps":[{"id":"supervised-learning-workflows/baselines-before-fancy-models/s01-01-intro"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s02-02-choose-the-risk"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s03-03-predict-the-checks"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s04-04-fill-the-return"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s05-05-fix-the-overtrust"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s06-06-write-baseline-comparison"},{"id":"supervised-learning-workflows/baselines-before-fancy-models/s07-07-checkpoint"}]},{"slug":"training-and-prediction-loop","steps":[{"id":"supervised-learning-workflows/training-and-prediction-loop/s01-01-intro"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s02-02-choose-the-risk"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s03-03-predict-the-checks"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s04-04-fill-the-return"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s05-05-fix-the-overtrust"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s06-06-write-train"},{"id":"supervised-learning-workflows/training-and-prediction-loop/s07-07-checkpoint"}]},{"slug":"overfitting-and-regularization","steps":[{"id":"supervised-learning-workflows/overfitting-and-regularization/s01-01-intro"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s02-02-choose-the-risk"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s03-03-predict-the-checks"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s04-04-fill-the-return"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s05-05-fix-the-overtrust"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s06-06-write-overfit-audit"},{"id":"supervised-learning-workflows/overfitting-and-regularization/s07-07-checkpoint"}]},{"slug":"mission-baseline-model-showdown","steps":[{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s01-01-intro"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s02-02-choose-the-risk"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s03-03-predict-the-checks"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s04-04-fill-the-return"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s05-05-fix-the-overtrust"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s06-06-write-classifier-brief"},{"id":"supervised-learning-workflows/mission-baseline-model-showdown/s07-07-checkpoint"}]}]},{"slug":"unsupervised-learning-and-embeddings","lessons":[{"slug":"clustering-without-labels","steps":[{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s01-01-intro"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s02-02-choose-the-risk"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s03-03-predict-the-checks"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s04-04-fill-the-return"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s05-05-fix-the-overtrust"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s06-06-write-cluster-summary"},{"id":"unsupervised-learning-and-embeddings/clustering-without-labels/s07-07-checkpoint"}]},{"slug":"dimensionality-and-distance","steps":[{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s01-01-intro"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s02-02-choose-the-risk"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s03-03-predict-the-checks"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s04-04-fill-the-return"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s05-05-fix-the-overtrust"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s06-06-write-euclidean"},{"id":"unsupervised-learning-and-embeddings/dimensionality-and-distance/s07-07-checkpoint"}]},{"slug":"embeddings-as-features","steps":[{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s01-01-intro"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s02-02-choose-the-risk"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s03-03-predict-the-checks"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s04-04-fill-the-return"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s05-05-fix-the-overtrust"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s06-06-write-retrieval-receipt"},{"id":"unsupervised-learning-and-embeddings/embeddings-as-features/s07-07-checkpoint"}]},{"slug":"mission-tiny-recommender","steps":[{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s01-01-intro"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s02-02-choose-the-risk"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s03-03-predict-the-checks"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s04-04-fill-the-return"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s05-05-fix-the-overtrust"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s06-06-write-recommendation-card"},{"id":"unsupervised-learning-and-embeddings/mission-tiny-recommender/s07-07-checkpoint"}]}]},{"slug":"metrics-and-error-analysis","lessons":[{"slug":"confusion-matrix","steps":[{"id":"metrics-and-error-analysis/confusion-matrix/s01-01-intro"},{"id":"metrics-and-error-analysis/confusion-matrix/s02-02-choose-the-risk"},{"id":"metrics-and-error-analysis/confusion-matrix/s03-03-predict-the-checks"},{"id":"metrics-and-error-analysis/confusion-matrix/s04-04-fill-the-return"},{"id":"metrics-and-error-analysis/confusion-matrix/s05-05-fix-the-overtrust"},{"id":"metrics-and-error-analysis/confusion-matrix/s06-06-write-count-ready"},{"id":"metrics-and-error-analysis/confusion-matrix/s07-07-checkpoint"}]},{"slug":"precision-recall-thresholds","steps":[{"id":"metrics-and-error-analysis/precision-recall-thresholds/s01-01-intro"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s02-02-choose-the-risk"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s03-03-predict-the-checks"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s04-04-fill-the-return"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s05-05-fix-the-overtrust"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s06-06-write-count-ready"},{"id":"metrics-and-error-analysis/precision-recall-thresholds/s07-07-checkpoint"}]},{"slug":"slice-based-error-analysis","steps":[{"id":"metrics-and-error-analysis/slice-based-error-analysis/s01-01-intro"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s02-02-choose-the-risk"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s03-03-predict-the-checks"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s04-04-fill-the-return"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s05-05-fix-the-overtrust"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s06-06-write-count-ready"},{"id":"metrics-and-error-analysis/slice-based-error-analysis/s07-07-checkpoint"}]},{"slug":"regression-metrics-and-residuals","steps":[{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s01-01-intro"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s02-02-choose-the-risk"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s03-03-predict-the-checks"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s04-04-fill-the-return"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s05-05-fix-the-overtrust"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s06-06-write-count-ready"},{"id":"metrics-and-error-analysis/regression-metrics-and-residuals/s07-07-checkpoint"}]},{"slug":"mission-confusion-matrix-triage","steps":[{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s01-01-intro"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s02-02-choose-the-risk"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s03-03-predict-the-checks"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s04-04-fill-the-return"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s05-05-fix-the-overtrust"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s06-06-write-count-ready"},{"id":"metrics-and-error-analysis/mission-confusion-matrix-triage/s07-07-checkpoint"}]}]},{"slug":"pytorch-tensors-and-autograd","lessons":[{"slug":"tensors-and-shapes","steps":[{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s01-01-intro"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s02-02-choose-the-risk"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s03-03-predict-the-checks"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s04-04-fill-the-return"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s05-05-fix-the-overtrust"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s06-06-write-count-ready"},{"id":"pytorch-tensors-and-autograd/tensors-and-shapes/s07-07-checkpoint"}]},{"slug":"broadcasting-and-vectorization","steps":[{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s01-01-intro"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s02-02-choose-the-risk"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s03-03-predict-the-checks"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s04-04-fill-the-return"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s05-05-fix-the-overtrust"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s06-06-write-count-ready"},{"id":"pytorch-tensors-and-autograd/broadcasting-and-vectorization/s07-07-checkpoint"}]},{"slug":"autograd-mental-model","steps":[{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s01-01-intro"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s02-02-choose-the-risk"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s03-03-predict-the-checks"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s04-04-fill-the-return"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s05-05-fix-the-overtrust"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s06-06-write-count-ready"},{"id":"pytorch-tensors-and-autograd/autograd-mental-model/s07-07-checkpoint"}]},{"slug":"mission-tensor-playground","steps":[{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s01-01-intro"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s02-02-choose-the-risk"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s03-03-predict-the-checks"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s04-04-fill-the-return"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s05-05-fix-the-overtrust"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s06-06-write-count-ready"},{"id":"pytorch-tensors-and-autograd/mission-tensor-playground/s07-07-checkpoint"}]}]},{"slug":"training-loops-and-optimizers","lessons":[{"slug":"loss-forward-backward-step","steps":[{"id":"training-loops-and-optimizers/loss-forward-backward-step/s01-01-intro"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s02-02-choose-the-risk"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s03-03-predict-the-checks"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s04-04-fill-the-return"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s05-05-fix-the-overtrust"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s06-06-write-count-ready"},{"id":"training-loops-and-optimizers/loss-forward-backward-step/s07-07-checkpoint"}]},{"slug":"gradient-descent-by-hand","steps":[{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s01-01-intro"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s02-02-choose-the-risk"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s03-03-predict-the-checks"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s04-04-fill-the-return"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s05-05-fix-the-overtrust"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s06-06-write-count-ready"},{"id":"training-loops-and-optimizers/gradient-descent-by-hand/s07-07-checkpoint"}]},{"slug":"optimizers-and-schedulers","steps":[{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s01-01-intro"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s02-02-choose-the-risk"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s03-03-predict-the-checks"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s04-04-fill-the-return"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s05-05-fix-the-overtrust"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s06-06-write-count-ready"},{"id":"training-loops-and-optimizers/optimizers-and-schedulers/s07-07-checkpoint"}]},{"slug":"checkpoints-and-reproducibility","steps":[{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s01-01-intro"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s02-02-choose-the-risk"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s03-03-predict-the-checks"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s04-04-fill-the-return"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s05-05-fix-the-overtrust"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s06-06-write-count-ready"},{"id":"training-loops-and-optimizers/checkpoints-and-reproducibility/s07-07-checkpoint"}]},{"slug":"mission-overfit-then-recover","steps":[{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s01-01-intro"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s02-02-choose-the-risk"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s03-03-predict-the-checks"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s04-04-fill-the-return"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s05-05-fix-the-overtrust"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s06-06-write-count-ready"},{"id":"training-loops-and-optimizers/mission-overfit-then-recover/s07-07-checkpoint"}]}]},{"slug":"deep-learning-architectures","lessons":[{"slug":"cnns-and-local-patterns","steps":[{"id":"deep-learning-architectures/cnns-and-local-patterns/s01-01-intro"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s02-02-choose-the-risk"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s03-03-predict-the-checks"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s04-04-fill-the-return"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s05-05-fix-the-overtrust"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s06-06-write-count-ready"},{"id":"deep-learning-architectures/cnns-and-local-patterns/s07-07-checkpoint"}]},{"slug":"tokenizers-and-context-budget","steps":[{"id":"deep-learning-architectures/tokenizers-and-context-budget/s01-01-intro"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s02-02-choose-the-risk"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s03-03-predict-the-checks"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s04-04-fill-the-return"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s05-05-fix-the-overtrust"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s06-06-write-count-ready"},{"id":"deep-learning-architectures/tokenizers-and-context-budget/s07-07-checkpoint"}]},{"slug":"attention-and-transformer-blocks","steps":[{"id":"deep-learning-architectures/attention-and-transformer-blocks/s01-01-intro"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s02-02-choose-the-risk"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s03-03-predict-the-checks"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s04-04-fill-the-return"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s05-05-fix-the-overtrust"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s06-06-write-count-ready"},{"id":"deep-learning-architectures/attention-and-transformer-blocks/s07-07-checkpoint"}]},{"slug":"decoding-kv-cache-quantization","steps":[{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s01-01-intro"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s02-02-choose-the-risk"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s03-03-predict-the-checks"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s04-04-fill-the-return"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s05-05-fix-the-overtrust"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s06-06-write-count-ready"},{"id":"deep-learning-architectures/decoding-kv-cache-quantization/s07-07-checkpoint"}]},{"slug":"mission-architecture-tradeoff-note","steps":[{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s01-01-intro"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s02-02-choose-the-risk"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s03-03-predict-the-checks"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s04-04-fill-the-return"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s05-05-fix-the-overtrust"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s06-06-write-count-ready"},{"id":"deep-learning-architectures/mission-architecture-tradeoff-note/s07-07-checkpoint"}]}]},{"slug":"feature-experiments-registries","lessons":[{"slug":"feature-pipeline-contracts","steps":[{"id":"feature-experiments-registries/feature-pipeline-contracts/s01-01-intro"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s02-02-choose-the-risk"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s03-03-predict-the-checks"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s04-04-fill-the-return"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s05-05-fix-the-overtrust"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s06-06-write-count-ready"},{"id":"feature-experiments-registries/feature-pipeline-contracts/s07-07-checkpoint"}]},{"slug":"train-inference-skew","steps":[{"id":"feature-experiments-registries/train-inference-skew/s01-01-intro"},{"id":"feature-experiments-registries/train-inference-skew/s02-02-choose-the-risk"},{"id":"feature-experiments-registries/train-inference-skew/s03-03-predict-the-checks"},{"id":"feature-experiments-registries/train-inference-skew/s04-04-fill-the-return"},{"id":"feature-experiments-registries/train-inference-skew/s05-05-fix-the-overtrust"},{"id":"feature-experiments-registries/train-inference-skew/s06-06-write-count-ready"},{"id":"feature-experiments-registries/train-inference-skew/s07-07-checkpoint"}]},{"slug":"experiment-tracker-lite","steps":[{"id":"feature-experiments-registries/experiment-tracker-lite/s01-01-intro"},{"id":"feature-experiments-registries/experiment-tracker-lite/s02-02-choose-the-risk"},{"id":"feature-experiments-registries/experiment-tracker-lite/s03-03-predict-the-checks"},{"id":"feature-experiments-registries/experiment-tracker-lite/s04-04-fill-the-return"},{"id":"feature-experiments-registries/experiment-tracker-lite/s05-05-fix-the-overtrust"},{"id":"feature-experiments-registries/experiment-tracker-lite/s06-06-write-count-ready"},{"id":"feature-experiments-registries/experiment-tracker-lite/s07-07-checkpoint"}]},{"slug":"model-registry-shape","steps":[{"id":"feature-experiments-registries/model-registry-shape/s01-01-intro"},{"id":"feature-experiments-registries/model-registry-shape/s02-02-choose-the-risk"},{"id":"feature-experiments-registries/model-registry-shape/s03-03-predict-the-checks"},{"id":"feature-experiments-registries/model-registry-shape/s04-04-fill-the-return"},{"id":"feature-experiments-registries/model-registry-shape/s05-05-fix-the-overtrust"},{"id":"feature-experiments-registries/model-registry-shape/s06-06-write-count-ready"},{"id":"feature-experiments-registries/model-registry-shape/s07-07-checkpoint"}]},{"slug":"mission-feature-pipeline-and-tracker","steps":[{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s01-01-intro"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s02-02-choose-the-risk"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s03-03-predict-the-checks"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s04-04-fill-the-return"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s05-05-fix-the-overtrust"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s06-06-write-count-ready"},{"id":"feature-experiments-registries/mission-feature-pipeline-and-tracker/s07-07-checkpoint"}]}]},{"slug":"model-serving-and-mlops","lessons":[{"slug":"fastapi-inference-shape","steps":[{"id":"model-serving-and-mlops/fastapi-inference-shape/s01-01-intro"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s06-06-write-count-ready"},{"id":"model-serving-and-mlops/fastapi-inference-shape/s07-07-checkpoint"}]},{"slug":"batch-vs-realtime-serving","steps":[{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s01-01-intro"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s06-06-write-count-ready"},{"id":"model-serving-and-mlops/batch-vs-realtime-serving/s07-07-checkpoint"}]},{"slug":"docker-runtime-and-config","steps":[{"id":"model-serving-and-mlops/docker-runtime-and-config/s01-01-intro"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s06-06-write-count-ready"},{"id":"model-serving-and-mlops/docker-runtime-and-config/s07-07-checkpoint"}]},{"slug":"github-actions-validation-gates","steps":[{"id":"model-serving-and-mlops/github-actions-validation-gates/s01-01-intro"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s06-06-write-failing-checks"},{"id":"model-serving-and-mlops/github-actions-validation-gates/s07-07-checkpoint"}]},{"slug":"continuous-training-and-rollout","steps":[{"id":"model-serving-and-mlops/continuous-training-and-rollout/s01-01-intro"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s06-06-write-count-ready"},{"id":"model-serving-and-mlops/continuous-training-and-rollout/s07-07-checkpoint"}]},{"slug":"mission-fastapi-model-server","steps":[{"id":"model-serving-and-mlops/mission-fastapi-model-server/s01-01-intro"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s02-02-choose-the-risk"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s03-03-predict-the-checks"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s04-04-fill-the-return"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s05-05-fix-the-overtrust"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s06-06-write-count-ready"},{"id":"model-serving-and-mlops/mission-fastapi-model-server/s07-07-checkpoint"}]}]},{"slug":"monitoring-cloud-portfolio","lessons":[{"slug":"structured-logs-and-alerts","steps":[{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s01-01-intro"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s06-06-write-count-ready"},{"id":"monitoring-cloud-portfolio/structured-logs-and-alerts/s07-07-checkpoint"}]},{"slug":"data-and-concept-drift","steps":[{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s01-01-intro"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s06-06-write-count-ready"},{"id":"monitoring-cloud-portfolio/data-and-concept-drift/s07-07-checkpoint"}]},{"slug":"retraining-triggers","steps":[{"id":"monitoring-cloud-portfolio/retraining-triggers/s01-01-intro"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s06-06-write-count-ready"},{"id":"monitoring-cloud-portfolio/retraining-triggers/s07-07-checkpoint"}]},{"slug":"cloud-gpu-kubernetes-cost","steps":[{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s01-01-intro"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s06-06-write-count-ready"},{"id":"monitoring-cloud-portfolio/cloud-gpu-kubernetes-cost/s07-07-checkpoint"}]},{"slug":"portfolio-architecture-docs","steps":[{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s01-01-intro"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s06-06-write-count-ready"},{"id":"monitoring-cloud-portfolio/portfolio-architecture-docs/s07-07-checkpoint"}]},{"slug":"mission-final-portfolio-ml-system","steps":[{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s01-01-intro"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s02-02-choose-the-risk"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s03-03-predict-the-checks"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s04-04-fill-the-return"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s05-05-fix-the-overtrust"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s06-06-write-decide"},{"id":"monitoring-cloud-portfolio/mission-final-portfolio-ml-system/s07-07-checkpoint"}]}]},{"slug":"graphic-design-studio","lessons":[{"slug":"prompting-like-an-art-director","steps":[{"id":"graphic-design-studio/prompting-like-an-art-director/s01-01-the-prompt-is-a-creative-brief"},{"id":"graphic-design-studio/prompting-like-an-art-director/s02-02-style-language"},{"id":"graphic-design-studio/prompting-like-an-art-director/s03-03-which-tool-for-the-headline"},{"id":"graphic-design-studio/prompting-like-an-art-director/s04-04-predict-the-scaffold"},{"id":"graphic-design-studio/prompting-like-an-art-director/s05-05-negative-space-and-the-iteration-loop"},{"id":"graphic-design-studio/prompting-like-an-art-director/s06-06-fix-the-mood-word-prompt"},{"id":"graphic-design-studio/prompting-like-an-art-director/s07-07-write-prompt-linter"},{"id":"graphic-design-studio/prompting-like-an-art-director/s08-08-checkpoint"}]},{"slug":"the-design-kit","steps":[{"id":"graphic-design-studio/the-design-kit/s01-01-the-brand-bible-as-data"},{"id":"graphic-design-studio/the-design-kit/s02-02-fill-the-palette-line"},{"id":"graphic-design-studio/the-design-kit/s03-03-banned-looks"},{"id":"graphic-design-studio/the-design-kit/s04-04-predict-the-negatives"},{"id":"graphic-design-studio/the-design-kit/s05-05-which-field-failed"},{"id":"graphic-design-studio/the-design-kit/s06-06-fix-the-crossed-kit"},{"id":"graphic-design-studio/the-design-kit/s07-07-write-contrast-check"},{"id":"graphic-design-studio/the-design-kit/s08-08-tokens-for-the-interface"},{"id":"graphic-design-studio/the-design-kit/s09-09-write-token-audit"},{"id":"graphic-design-studio/the-design-kit/s10-10-checkpoint"}]},{"slug":"fifty-assets-one-brand","steps":[{"id":"graphic-design-studio/fifty-assets-one-brand/s01-01-the-consistency-problem"},{"id":"graphic-design-studio/fifty-assets-one-brand/s02-02-seeds-the-same-dice-twice"},{"id":"graphic-design-studio/fifty-assets-one-brand/s03-03-why-does-the-mascot-drift"},{"id":"graphic-design-studio/fifty-assets-one-brand/s04-04-predict-the-batch-plan"},{"id":"graphic-design-studio/fifty-assets-one-brand/s05-05-the-qa-filter"},{"id":"graphic-design-studio/fifty-assets-one-brand/s06-06-defend-the-pick"},{"id":"graphic-design-studio/fifty-assets-one-brand/s07-07-fix-the-qa-threshold"},{"id":"graphic-design-studio/fifty-assets-one-brand/s08-08-write-palette-distance"},{"id":"graphic-design-studio/fifty-assets-one-brand/s09-09-checkpoint"}]}]},{"slug":"campaign-studio","lessons":[{"slug":"voice-pack-as-data","steps":[{"id":"campaign-studio/voice-pack-as-data/s01-01-the-campaign-and-the-gate"},{"id":"campaign-studio/voice-pack-as-data/s02-02-lint-the-draft"},{"id":"campaign-studio/voice-pack-as-data/s03-03-predict-the-violations"},{"id":"campaign-studio/voice-pack-as-data/s04-04-testable-or-mood"},{"id":"campaign-studio/voice-pack-as-data/s05-05-fill-the-linter"},{"id":"campaign-studio/voice-pack-as-data/s06-06-variants-as-records"},{"id":"campaign-studio/voice-pack-as-data/s07-07-write-lint-copy"},{"id":"campaign-studio/voice-pack-as-data/s08-08-checkpoint"}]},{"slug":"batch-plan-and-calendar","steps":[{"id":"campaign-studio/batch-plan-and-calendar/s01-01-slots-before-prompts"},{"id":"campaign-studio/batch-plan-and-calendar/s02-02-which-field-is-missing"},{"id":"campaign-studio/batch-plan-and-calendar/s03-03-write-batch-plan"},{"id":"campaign-studio/batch-plan-and-calendar/s04-04-the-calendar-is-data"},{"id":"campaign-studio/batch-plan-and-calendar/s05-05-predict-the-queue"},{"id":"campaign-studio/batch-plan-and-calendar/s06-06-fix-the-gate"},{"id":"campaign-studio/batch-plan-and-calendar/s07-07-checkpoint"}]},{"slug":"the-numbers-pass","steps":[{"id":"campaign-studio/the-numbers-pass/s01-01-a-report-is-a-draft-over-data"},{"id":"campaign-studio/the-numbers-pass/s02-02-predict-the-verdict"},{"id":"campaign-studio/the-numbers-pass/s03-03-write-verify-numbers"},{"id":"campaign-studio/the-numbers-pass/s04-04-denominators-and-the-ftc"},{"id":"campaign-studio/the-numbers-pass/s05-05-what-may-not-ship"},{"id":"campaign-studio/the-numbers-pass/s06-06-fill-the-policy-router"},{"id":"campaign-studio/the-numbers-pass/s07-07-checkpoint"}]}]},{"slug":"support-studio","lessons":[{"slug":"the-policy-pack","steps":[{"id":"support-studio/the-policy-pack/s01-01-the-reply-that-binds-the-company"},{"id":"support-studio/the-policy-pack/s02-02-the-pack-as-data"},{"id":"support-studio/the-policy-pack/s03-03-spot-the-banned-promise"},{"id":"support-studio/the-policy-pack/s04-04-predict-the-lint"},{"id":"support-studio/the-policy-pack/s05-05-fix-the-stale-citation"},{"id":"support-studio/the-policy-pack/s06-06-the-review-loop-evidence"},{"id":"support-studio/the-policy-pack/s07-07-write-lint-draft"},{"id":"support-studio/the-policy-pack/s08-08-checkpoint"}]},{"slug":"golden-evals-earn-autonomy","steps":[{"id":"support-studio/golden-evals-earn-autonomy/s01-01-the-klarna-arc"},{"id":"support-studio/golden-evals-earn-autonomy/s02-02-a-macro-is-three-things"},{"id":"support-studio/golden-evals-earn-autonomy/s03-03-inside-the-lane"},{"id":"support-studio/golden-evals-earn-autonomy/s04-04-predict-the-eval-run"},{"id":"support-studio/golden-evals-earn-autonomy/s05-05-fill-the-promotion-rule"},{"id":"support-studio/golden-evals-earn-autonomy/s06-06-write-run-golden-evals"},{"id":"support-studio/golden-evals-earn-autonomy/s07-07-checkpoint"}]},{"slug":"the-escalation-boundary","steps":[{"id":"support-studio/the-escalation-boundary/s01-01-drafting-is-not-spending"},{"id":"support-studio/the-escalation-boundary/s02-02-triggers-as-conditions"},{"id":"support-studio/the-escalation-boundary/s03-03-route-this-ticket"},{"id":"support-studio/the-escalation-boundary/s04-04-predict-the-routing"},{"id":"support-studio/the-escalation-boundary/s05-05-fix-the-refund-hole"},{"id":"support-studio/the-escalation-boundary/s06-06-disclosure-and-the-kill-switch"},{"id":"support-studio/the-escalation-boundary/s07-07-write-route-ticket"},{"id":"support-studio/the-escalation-boundary/s08-08-checkpoint"}]}]},{"slug":"copy-studio","lessons":[{"slug":"the-voice-fingerprint","steps":[{"id":"copy-studio/the-voice-fingerprint/s01-01-the-client-you-inherit"},{"id":"copy-studio/the-voice-fingerprint/s02-02-the-fingerprint"},{"id":"copy-studio/the-voice-fingerprint/s03-03-predict-the-ceiling"},{"id":"copy-studio/the-voice-fingerprint/s04-04-the-samples-say-otherwise"},{"id":"copy-studio/the-voice-fingerprint/s05-05-fill-derive-the-rules"},{"id":"copy-studio/the-voice-fingerprint/s06-06-the-pack-with-provenance"},{"id":"copy-studio/the-voice-fingerprint/s07-07-write-fingerprint"},{"id":"copy-studio/the-voice-fingerprint/s08-08-checkpoint"}]},{"slug":"variants-that-hold-the-line","steps":[{"id":"copy-studio/variants-that-hold-the-line/s01-01-twelve-by-thursday"},{"id":"copy-studio/variants-that-hold-the-line/s02-02-the-drift-problem"},{"id":"copy-studio/variants-that-hold-the-line/s03-03-predict-the-drift-report"},{"id":"copy-studio/variants-that-hold-the-line/s04-04-which-subject-line-ships"},{"id":"copy-studio/variants-that-hold-the-line/s05-05-fix-the-channel-check"},{"id":"copy-studio/variants-that-hold-the-line/s06-06-the-kill-log"},{"id":"copy-studio/variants-that-hold-the-line/s07-07-write-voice-gate"},{"id":"copy-studio/variants-that-hold-the-line/s08-08-checkpoint"}]},{"slug":"the-claim-check","steps":[{"id":"copy-studio/the-claim-check/s01-01-forty-one-of-seventy-seven"},{"id":"copy-studio/the-claim-check/s02-02-what-counts-as-checkable"},{"id":"copy-studio/the-claim-check/s03-03-predict-the-claim-list"},{"id":"copy-studio/the-claim-check/s04-04-what-closes-a-row"},{"id":"copy-studio/the-claim-check/s05-05-fill-the-ship-decision"},{"id":"copy-studio/the-claim-check/s06-06-the-corrections-log"},{"id":"copy-studio/the-claim-check/s07-07-write-ship-gate"},{"id":"copy-studio/the-claim-check/s08-08-checkpoint"}]}]},{"slug":"analytics-studio","lessons":[{"slug":"the-cleaning-log","steps":[{"id":"analytics-studio/the-cleaning-log/s01-01-where-did-those-rows-go"},{"id":"analytics-studio/the-cleaning-log/s02-02-the-log-as-data"},{"id":"analytics-studio/the-cleaning-log/s03-03-refuse-the-silent-step"},{"id":"analytics-studio/the-cleaning-log/s04-04-predict-the-receipts"},{"id":"analytics-studio/the-cleaning-log/s05-05-fill-the-null-decision"},{"id":"analytics-studio/the-cleaning-log/s06-06-fix-the-vanishing-rows"},{"id":"analytics-studio/the-cleaning-log/s07-07-write-the-null-ledger"},{"id":"analytics-studio/the-cleaning-log/s08-08-checkpoint"}]},{"slug":"every-number-has-an-address","steps":[{"id":"analytics-studio/every-number-has-an-address/s01-01-the-plausible-wrong-number"},{"id":"analytics-studio/every-number-has-an-address/s02-02-sample-first-then-full"},{"id":"analytics-studio/every-number-has-an-address/s03-03-which-check-catches-it"},{"id":"analytics-studio/every-number-has-an-address/s04-04-predict-the-fan-out"},{"id":"analytics-studio/every-number-has-an-address/s05-05-fill-the-trace"},{"id":"analytics-studio/every-number-has-an-address/s06-06-write-audit-the-memo"},{"id":"analytics-studio/every-number-has-an-address/s07-07-checkpoint"}]},{"slug":"challenge-then-rerun","steps":[{"id":"analytics-studio/challenge-then-rerun/s01-01-the-confident-recommendation"},{"id":"analytics-studio/challenge-then-rerun/s02-02-the-triad-as-a-checklist"},{"id":"analytics-studio/challenge-then-rerun/s03-03-name-the-leg"},{"id":"analytics-studio/challenge-then-rerun/s04-04-predict-the-rerun"},{"id":"analytics-studio/challenge-then-rerun/s05-05-fix-the-drifting-memo"},{"id":"analytics-studio/challenge-then-rerun/s06-06-write-score-the-challenge"},{"id":"analytics-studio/challenge-then-rerun/s07-07-checkpoint"}]}]},{"slug":"delivery-studio","lessons":[{"slug":"the-spec-is-the-build","steps":[{"id":"delivery-studio/the-spec-is-the-build/s01-01-the-clarifying-question-is-gone"},{"id":"delivery-studio/the-spec-is-the-build/s02-02-the-ticket-as-data"},{"id":"delivery-studio/the-spec-is-the-build/s03-03-spot-the-untestable-check"},{"id":"delivery-studio/the-spec-is-the-build/s04-04-predict-the-spec-lint"},{"id":"delivery-studio/the-spec-is-the-build/s05-05-fill-the-scope-fence"},{"id":"delivery-studio/the-spec-is-the-build/s06-06-inside-or-outside-the-frontier"},{"id":"delivery-studio/the-spec-is-the-build/s07-07-write-lint-spec"},{"id":"delivery-studio/the-spec-is-the-build/s08-08-checkpoint"}]},{"slug":"acceptance-before-the-build","steps":[{"id":"delivery-studio/acceptance-before-the-build/s01-01-the-ninety-seven-thousand-dollar-skip"},{"id":"delivery-studio/acceptance-before-the-build/s02-02-evidence-against-the-checks"},{"id":"delivery-studio/acceptance-before-the-build/s03-03-which-check-catches-it"},{"id":"delivery-studio/acceptance-before-the-build/s04-04-fix-the-vibes-review"},{"id":"delivery-studio/acceptance-before-the-build/s05-05-predict-the-review-run"},{"id":"delivery-studio/acceptance-before-the-build/s06-06-rework-is-a-list-not-a-feeling"},{"id":"delivery-studio/acceptance-before-the-build/s07-07-write-review-deliverable"},{"id":"delivery-studio/acceptance-before-the-build/s08-08-checkpoint"}]},{"slug":"status-from-artifacts","steps":[{"id":"delivery-studio/status-from-artifacts/s01-01-watermelon-status"},{"id":"delivery-studio/status-from-artifacts/s02-02-derive-the-rag"},{"id":"delivery-studio/status-from-artifacts/s03-03-spot-the-artifact"},{"id":"delivery-studio/status-from-artifacts/s04-04-fill-the-worst-of"},{"id":"delivery-studio/status-from-artifacts/s05-05-predict-the-rollup"},{"id":"delivery-studio/status-from-artifacts/s06-06-the-not-ready-call"},{"id":"delivery-studio/status-from-artifacts/s07-07-write-readiness-gate"},{"id":"delivery-studio/status-from-artifacts/s08-08-checkpoint"}]}]},{"slug":"people-studio","lessons":[{"slug":"the-data-firewall","steps":[{"id":"people-studio/the-data-firewall/s01-01-the-rule-that-cannot-live-in-memory"},{"id":"people-studio/the-data-firewall/s02-02-the-firewall-as-data"},{"id":"people-studio/the-data-firewall/s03-03-which-payload-passes"},{"id":"people-studio/the-data-firewall/s04-04-predict-the-scan"},{"id":"people-studio/the-data-firewall/s05-05-fix-the-leaky-gate"},{"id":"people-studio/the-data-firewall/s06-06-employee-relations-takes-nothing"},{"id":"people-studio/the-data-firewall/s07-07-write-prompt-gate"},{"id":"people-studio/the-data-firewall/s08-08-checkpoint"}]},{"slug":"briefs-with-receipts","steps":[{"id":"people-studio/briefs-with-receipts/s01-01-the-most-automated-desk"},{"id":"people-studio/briefs-with-receipts/s02-02-a-claim-with-a-receipt"},{"id":"people-studio/briefs-with-receipts/s03-03-sourced-or-inference"},{"id":"people-studio/briefs-with-receipts/s04-04-predict-the-receipt-audit"},{"id":"people-studio/briefs-with-receipts/s05-05-fill-the-lookup"},{"id":"people-studio/briefs-with-receipts/s06-06-the-screening-line"},{"id":"people-studio/briefs-with-receipts/s07-07-write-audit-brief"},{"id":"people-studio/briefs-with-receipts/s08-08-checkpoint"}]},{"slug":"the-paper-trail","steps":[{"id":"people-studio/the-paper-trail/s01-01-policy-is-legal-surface"},{"id":"people-studio/the-paper-trail/s02-02-the-boundary-as-a-state-machine"},{"id":"people-studio/the-paper-trail/s03-03-who-signs-off"},{"id":"people-studio/the-paper-trail/s04-04-predict-the-version-log"},{"id":"people-studio/the-paper-trail/s05-05-fix-the-shortcut"},{"id":"people-studio/the-paper-trail/s06-06-the-evidence-trail"},{"id":"people-studio/the-paper-trail/s07-07-write-decision-record"},{"id":"people-studio/the-paper-trail/s08-08-checkpoint"}]}]},{"slug":"ops-studio","lessons":[{"slug":"the-sop-is-data","steps":[{"id":"ops-studio/the-sop-is-data/s01-01-the-two-weeks-notice-problem"},{"id":"ops-studio/the-sop-is-data/s02-02-the-sop-as-data"},{"id":"ops-studio/the-sop-is-data/s03-03-spot-the-invented-step"},{"id":"ops-studio/the-sop-is-data/s04-04-fill-the-sop-lint"},{"id":"ops-studio/the-sop-is-data/s05-05-predict-the-diff"},{"id":"ops-studio/the-sop-is-data/s06-06-the-diff-is-the-change-control"},{"id":"ops-studio/the-sop-is-data/s07-07-write-lint-sop"},{"id":"ops-studio/the-sop-is-data/s08-08-checkpoint"}]},{"slug":"auto-run-or-named-approval","steps":[{"id":"ops-studio/auto-run-or-named-approval/s01-01-the-checklist-that-lied"},{"id":"ops-studio/auto-run-or-named-approval/s02-02-the-schedule-is-data"},{"id":"ops-studio/auto-run-or-named-approval/s03-03-which-item-may-auto-run"},{"id":"ops-studio/auto-run-or-named-approval/s04-04-fix-the-silent-auto-complete"},{"id":"ops-studio/auto-run-or-named-approval/s05-05-predict-the-exception-queue"},{"id":"ops-studio/auto-run-or-named-approval/s06-06-exceptions-are-a-queue-not-a-shrug"},{"id":"ops-studio/auto-run-or-named-approval/s07-07-write-run-checklist"},{"id":"ops-studio/auto-run-or-named-approval/s08-08-checkpoint"}]},{"slug":"numbers-with-receipts","steps":[{"id":"ops-studio/numbers-with-receipts/s01-01-the-number-that-moves-money"},{"id":"ops-studio/numbers-with-receipts/s02-02-recompute-dont-trust"},{"id":"ops-studio/numbers-with-receipts/s03-03-which-claim-has-no-receipt"},{"id":"ops-studio/numbers-with-receipts/s04-04-predict-the-verification"},{"id":"ops-studio/numbers-with-receipts/s05-05-fill-the-receipt-check"},{"id":"ops-studio/numbers-with-receipts/s06-06-the-anomaly-is-a-question"},{"id":"ops-studio/numbers-with-receipts/s07-07-write-verify-report"},{"id":"ops-studio/numbers-with-receipts/s08-08-checkpoint"}]}]},{"slug":"legal-studio","lessons":[{"slug":"the-cite-check-gate","steps":[{"id":"legal-studio/the-cite-check-gate/s01-01-the-brief-that-reads-perfectly"},{"id":"legal-studio/the-cite-check-gate/s02-02-three-verdicts-as-data"},{"id":"legal-studio/the-cite-check-gate/s03-03-which-cite-needs-no-check"},{"id":"legal-studio/the-cite-check-gate/s04-04-predict-the-verification-run"},{"id":"legal-studio/the-cite-check-gate/s05-05-fix-the-verifier-that-asks-again"},{"id":"legal-studio/the-cite-check-gate/s06-06-write-verify-authorities"},{"id":"legal-studio/the-cite-check-gate/s07-07-checkpoint"}]},{"slug":"clause-extraction-with-pinpoints","steps":[{"id":"legal-studio/clause-extraction-with-pinpoints/s01-01-a-claim-without-a-pinpoint-is-a-rumor"},{"id":"legal-studio/clause-extraction-with-pinpoints/s02-02-the-playbook-as-data"},{"id":"legal-studio/clause-extraction-with-pinpoints/s03-03-which-row-is-unusable"},{"id":"legal-studio/clause-extraction-with-pinpoints/s04-04-predict-the-triage"},{"id":"legal-studio/clause-extraction-with-pinpoints/s05-05-fill-the-walk-away-route"},{"id":"legal-studio/clause-extraction-with-pinpoints/s06-06-write-triage-extraction"},{"id":"legal-studio/clause-extraction-with-pinpoints/s07-07-checkpoint"}]},{"slug":"privilege-rules-and-the-review-log","steps":[{"id":"legal-studio/privilege-rules-and-the-review-log/s01-01-privilege-is-a-routing-question"},{"id":"legal-studio/privilege-rules-and-the-review-log/s02-02-the-routing-table"},{"id":"legal-studio/privilege-rules-and-the-review-log/s03-03-which-request-is-the-waiver-risk"},{"id":"legal-studio/privilege-rules-and-the-review-log/s04-04-predict-the-router"},{"id":"legal-studio/privilege-rules-and-the-review-log/s05-05-fix-the-privilege-hole"},{"id":"legal-studio/privilege-rules-and-the-review-log/s06-06-the-log-that-survives-a-motion"},{"id":"legal-studio/privilege-rules-and-the-review-log/s07-07-write-log-ready"},{"id":"legal-studio/privilege-rules-and-the-review-log/s08-08-checkpoint"}]}]},{"slug":"agent-studio","lessons":[{"slug":"the-tool-contract","steps":[{"id":"agent-studio/the-tool-contract/s01-01-the-feature-you-ship-today"},{"id":"agent-studio/the-tool-contract/s02-02-the-schema-is-the-blast-radius"},{"id":"agent-studio/the-tool-contract/s03-03-spot-the-loose-tool"},{"id":"agent-studio/the-tool-contract/s04-04-predict-the-validation"},{"id":"agent-studio/the-tool-contract/s05-05-fix-the-yolo-rerun"},{"id":"agent-studio/the-tool-contract/s06-06-write-validate-call"},{"id":"agent-studio/the-tool-contract/s07-07-checkpoint"}]},{"slug":"the-loop-with-a-meter","steps":[{"id":"agent-studio/the-loop-with-a-meter/s01-01-a-loop-that-cannot-stop-is-an-invoice"},{"id":"agent-studio/the-loop-with-a-meter/s02-02-caps-are-part-of-the-loop"},{"id":"agent-studio/the-loop-with-a-meter/s03-03-why-did-it-stop"},{"id":"agent-studio/the-loop-with-a-meter/s04-04-predict-the-stop-reason"},{"id":"agent-studio/the-loop-with-a-meter/s05-05-fill-the-budget-check"},{"id":"agent-studio/the-loop-with-a-meter/s06-06-the-trace-is-the-receipt"},{"id":"agent-studio/the-loop-with-a-meter/s07-07-write-run-loop"},{"id":"agent-studio/the-loop-with-a-meter/s08-08-checkpoint"}]},{"slug":"evals-before-it-works","steps":[{"id":"agent-studio/evals-before-it-works/s01-01-green-before-it-runs"},{"id":"agent-studio/evals-before-it-works/s02-02-a-golden-case-is-a-past-incident"},{"id":"agent-studio/evals-before-it-works/s03-03-which-case-catches-it"},{"id":"agent-studio/evals-before-it-works/s04-04-predict-the-eval-run"},{"id":"agent-studio/evals-before-it-works/s05-05-fix-the-router-to-green"},{"id":"agent-studio/evals-before-it-works/s06-06-the-ratchet-entry"},{"id":"agent-studio/evals-before-it-works/s07-07-write-lint-the-ratchet"},{"id":"agent-studio/evals-before-it-works/s08-08-checkpoint"}]}]}]},"detail":{"number":24,"slug":"debugging-output","title":"debugging broken ai output","blurb":"when the model lies to your customer. the methodology for narrowing down what went wrong, the four most-common breakage classes, and the discipline that separates 'we shipped a fix' from 'we blamed the model and shrugged'.","overview":"$20","status":"live","lessons":[{"slug":"read-the-trace-not-the-chat","title":"Read the trace, not the chat — find the broken turn before reading the user's complaint","estMinutes":11,"prerequisites":["orientation"],"status":"live","steps":[{"type":"read","id":"debugging-output/read-the-trace-not-the-chat/s01-01-intro","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":110,"concept":"trace-first-debugging","body":"$21","cta":"Got it","code":"# a trace dump from a broken agent run.\n# the user's complaint: \"it gave me the wrong city.\"\n# before reading the chat, find the bad turn in the trace.\n\ntrace = [\n {\"turn\": 1, \"stop_reason\": \"tool_use\", \"tools_called\": [\"search\"], \"tokens\": 412, \"validation_errors\": 0},\n {\"turn\": 2, \"stop_reason\": \"tool_use\", \"tools_called\": [\"read_page\"], \"tokens\": 880, \"validation_errors\": 0},\n {\"turn\": 3, \"stop_reason\": \"tool_use\", \"tools_called\": [\"search\"], \"tokens\": 920, \"validation_errors\": 1}, # ← something here\n {\"turn\": 4, \"stop_reason\": \"end_turn\", \"tools_called\": [], \"tokens\": 145, \"validation_errors\": 0},\n]\n\n# find the turn with a non-zero validation_errors count.\nfor t in trace:\n if t[\"validation_errors\"] > 0:\n print(f\"suspect turn: {t['turn']}, tools_called: {t['tools_called']}\")\n","runnable":true},{"type":"mc","id":"debugging-output/read-the-trace-not-the-chat/s02-02-where-to-look-first","xp":2,"hint":[{"level":1,"body":"The trace is the structured record. The chat is the surface. Always read the structured record first — it's sorted and indexable; the chat is unstructured prose.","cost":0}],"personalize":false,"phase":"warmup","estSeconds":55,"concept":"triage-first-look","prompt":"A user reports a bad agent response. Where do you look FIRST?\n","options":[{"id":"b_trace_first","label":"B — open the trace and scan for the broken turn"},{"id":"a_chat_first","label":"A — re-read the chat to spot the issue","explain":"The chat shows the symptom; the trace shows the cause. Reading the chat first is how you waste 30 minutes before finding the actual broken turn."},{"id":"c_rerun","label":"C — re-run the agent and see if it reproduces","explain":"Re-running BEFORE understanding the trace is debugging by superstition. You want to know what happened *that* time, not what happens *this* time."},{"id":"d_trycatch","label":"D — add a try/except to catch errors next time","explain":"This silences future symptoms without finding the cause. You'll catch the error 100 more times and still not know why it fires."}],"answerIds":["b_trace_first"],"shuffle":false,"code":"# a complaint comes in. four places you could look first.\nplaces = {\n \"A\": \"Re-read the user's complaint and the agent's answer.\",\n \"B\": \"Open the trace and scan for non-zero validation_errors / non-end_turn stop_reason.\",\n \"C\": \"Re-run the agent on the same input and see what happens.\",\n \"D\": \"Add a try/except around the call site to catch errors.\",\n}\nfor k, v in places.items():\n print(f\"{k}: {v}\")\n","runnable":true},{"type":"read","id":"debugging-output/read-the-trace-not-the-chat/s03-03-the-four-failure-classes","xp":2,"hint":[],"personalize":false,"phase":"build","estSeconds":100,"concept":"failure-class-taxonomy","body":"$22","cta":"Got it","code":"# four failure classes. each has a different fix layer.\n\nCLASSES = {\n \"retrieval\": \"Tool/RAG returned wrong or missing data.\",\n \"prompt\": \"Model interpreted instructions wrong.\",\n \"true_hallucination\": \"No tool ran; model invented from nothing.\",\n \"downstream_mangling\": \"Trace fine; post-processing broke the output.\",\n}\n\nFIXES = {\n \"retrieval\": \"Fix the retriever — chunking, top-k, embedding model.\",\n \"prompt\": \"Fix the prompt — clearer instructions, few-shot example.\",\n \"true_hallucination\": \"Constrain output — schema, citations, refuse on uncertainty.\",\n \"downstream_mangling\": \"Fix the JSON parse / regex / display layer.\",\n}\n\nfor name in CLASSES:\n print(f\"{name}: {CLASSES[name]}\")\n print(f\" fix: {FIXES[name]}\")\n","runnable":true},{"type":"predict","id":"debugging-output/read-the-trace-not-the-chat/s04-04-predict-the-bad-turn","xp":3,"hint":[{"level":1,"body":"The loop iterates turns 1, 2, 3, 4. The break fires when `validation_errors > 0` — first true on turn 3. `suspect` is set to 3 and the loop exits.","cost":0}],"personalize":false,"phase":"build","estSeconds":75,"concept":"scan-trace-for-anomaly","code":"trace = [\n {\"turn\": 1, \"stop_reason\": \"tool_use\", \"tools_called\": [\"search\"], \"tokens\": 412, \"validation_errors\": 0},\n {\"turn\": 2, \"stop_reason\": \"tool_use\", \"tools_called\": [\"read_page\"], \"tokens\": 880, \"validation_errors\": 0},\n {\"turn\": 3, \"stop_reason\": \"tool_use\", \"tools_called\": [\"search\"], \"tokens\": 920, \"validation_errors\": 1},\n {\"turn\": 4, \"stop_reason\": \"end_turn\", \"tools_called\": [], \"tokens\": 145, \"validation_errors\": 0},\n]\n\n# find the FIRST turn with a non-zero validation_errors count.\nsuspect = None\nfor t in trace:\n if t[\"validation_errors\"] > 0:\n suspect = t[\"turn\"]\n break\n\nprint(f\"suspect turn: {suspect}\")\n","prompt":"Read the trace in the editor. Four turns. The third turn has\n`validation_errors: 1` — the rest are zero. The script scans\nfor the first turn with non-zero validation_errors and prints\nits turn number. What's the output?\n","grader":{"kind":"stdout-equality","expected":"suspect turn: 3","normalize":"collapse-trailing-newline"}},{"type":"fill","id":"debugging-output/read-the-trace-not-the-chat/s05-05-fill-the-find-bad-turn","xp":3,"hint":[{"level":1,"body":"Pass two arguments: the trace, and a lambda that returns True when `t[\"tokens\"] > 500`.","cost":0},{"level":2,"body":"`find_first(trace, lambda t: t[\"tokens\"] > 500)` — predicate as a single-line lambda. Lambdas are how production code parameterizes \"find the bad turn\" without writing a new function for each criterion.","cost":0}],"personalize":false,"phase":"build","estSeconds":65,"concept":"predicate-based-trace-search","prompt":"The function `find_first(trace, predicate)` should return the\nfirst turn where `predicate(turn)` returns `True`, or `None` if\nno turn matches. Fill in the call so the example finds the first\nturn with `tokens > 500`.\n\nExpected output:\n```\nbad turn: 2\n```\n","code":"trace = [\n {\"turn\": 1, \"tokens\": 412},\n {\"turn\": 2, \"tokens\": 880},\n {\"turn\": 3, \"tokens\": 920},\n {\"turn\": 4, \"tokens\": 145},\n]\n\ndef find_first(trace, predicate):\n for t in trace:\n if predicate(t):\n return t\n return None\n\nbad = ___\nprint(f\"bad turn: {bad['turn']}\")\n","blanks":[{"id":"find_first_call","accept":["find_first(trace, lambda t: t['tokens'] > 500)","find_first(trace, lambda t: t[\"tokens\"] > 500)"],"caseSensitive":true,"normalize":"trim"}]},{"type":"fix","id":"debugging-output/read-the-trace-not-the-chat/s06-06-fix-the-fix-the-symptom","xp":4,"hint":[{"level":1,"body":"Replace `return \"unknown\"` with a return of the diagnostic dict including the raw text. The caller then knows WHICH layer to fix.","cost":0},{"level":2,"body":"Change line 7 to:\n\n```python\nreturn {\"class\": \"prompt\", \"fix_layer\": \"the prompt\", \"raw\": raw_text}\n```\n","cost":0}],"personalize":false,"phase":"build","estSeconds":100,"concept":"fix-upstream-not-symptom","brokenCode":"import json\n\ndef parse_model_output(raw_text):\n try:\n return json.loads(raw_text)\n except json.JSONDecodeError:\n # bug: hides the failure class. caller sees \"unknown\" forever.\n return \"unknown\"\n\nraw = \"I think the answer is 42\"\nresult = parse_model_output(raw)\nprint(result)\n","prompt":"The agent returned malformed JSON. The downstream parser is\ncatching the error and returning `\"unknown\"`. This *hides* the\nsymptom but doesn't fix the bug — the next user gets the same\nwrong answer, packaged as `\"unknown\"`.\n\nThe trace shows the bug is upstream: the prompt didn't tell the\nmodel to return JSON. The fix is to return a `dict` describing\nthe upstream class so the caller can route to the right fix.\n\nFix the parser to RETURN\n`{\"class\": \"prompt\", \"fix_layer\": \"the prompt\", \"raw\": }`\ninstead of swallowing the error with `\"unknown\"`.\n\nExpected output:\n```\n{'class': 'prompt', 'fix_layer': 'the prompt', 'raw': 'I think the answer is 42'}\n```\n","grader":{"kind":"stdout-equality","expected":"{'class': 'prompt', 'fix_layer': 'the prompt', 'raw': 'I think the answer is 42'}","normalize":"collapse-trailing-newline"},"bugLines":[7],"revealAfter":4},{"type":"fix","id":"debugging-output/read-the-trace-not-the-chat/s07-07-fix-the-blame-the-model","xp":4,"hint":[{"level":1,"body":"Read `trace[-1][\"tools_called\"]`. If it has any items, return `\"retrieval\"`. Otherwise return `\"true_hallucination\"`.","cost":0},{"level":2,"body":"Replace the body of `classify` with:\n\n```python\nif trace[-1][\"tools_called\"]:\n return \"retrieval\"\nreturn \"true_hallucination\"\n```\n","cost":0}],"personalize":false,"phase":"build","estSeconds":100,"concept":"classify-not-blame","brokenCode":"trace = [\n {\"turn\": 1, \"tools_called\": [\"search\"], \"tool_result_snippet\": \"wrong data here\"},\n]\n\ndef classify(trace):\n # bug: ignores the trace. always blames hallucination.\n return \"true_hallucination\"\n\nprint(f\"class: {classify(trace)}\")\n","prompt":"The triage code is hardcoded to blame \"true_hallucination\" no\nmatter what the trace shows. But the trace says the agent's\nsearch tool ran AND returned data — so this is a **retrieval**\nproblem (wrong data came back), not a hallucination (model\ninventing from nothing).\n\nFix the classifier to check `trace[-1][\"tools_called\"]`. If it's\nnon-empty, this is `\"retrieval\"`. Otherwise it's\n`\"true_hallucination\"`.\n\nExpected output:\n```\nclass: retrieval\n```\n","grader":{"kind":"stdout-equality","expected":"class: retrieval","normalize":"collapse-trailing-newline"},"bugLines":[6],"revealAfter":4},{"type":"write","id":"debugging-output/read-the-trace-not-the-chat/s08-08-write-the-trace-summary","xp":5,"hint":[{"level":1,"body":"Compute each summary number with a comprehension over `trace`: `len(trace)`, `sum(len(t[\"tools_called\"]) for t in trace)`, `sum(t[\"validation_errors\"] for t in trace)`, `trace[-1][\"stop_reason\"]`. Then format as the f-string.","cost":0},{"level":2,"body":"`f\"{n_turns} turns, {n_tools} tools, {n_validation} validation_errors, final stop={final_stop}\"` — exact phrasing matters for grading.","cost":0}],"personalize":false,"phase":"build","estSeconds":200,"concept":"trace-summary-for-bug-ticket","prompt":"Write `summarize_trace(trace)` that produces a one-line bug-ticket\nsummary. The function:\n\n- Takes a trace (list of turn dicts).\n- Returns a string formatted exactly:\n `\" turns, tools, validation_errors, final stop=\"`.\n Where:\n - `` is the number of turns.\n - `` is the total number of tool calls across all turns\n (sum of `len(t[\"tools_called\"])`).\n - `` is the total `validation_errors` summed across turns.\n - `` is `trace[-1][\"stop_reason\"]`.\n\nThree traces run for you. Expected output:\n```\n4 turns, 3 tools, 0 validation_errors, final stop=end_turn\n3 turns, 2 tools, 1 validation_errors, final stop=end_turn\n2 turns, 0 tools, 0 validation_errors, final stop=max_tokens\n```\n","starter":"$23","grader":{"kind":"stdout-equality","expected":"4 turns, 3 tools, 0 validation_errors, final stop=end_turn\n3 turns, 2 tools, 1 validation_errors, final stop=end_turn\n2 turns, 0 tools, 0 validation_errors, final stop=max_tokens\n","normalize":"collapse-trailing-newline"},"solution":"$24","hiddenTests":[]},{"type":"checkpoint","id":"debugging-output/read-the-trace-not-the-chat/s09-09-checkpoint","xp":8,"hint":[],"personalize":false,"phase":"check","estSeconds":220,"concept":"full-triage-pipeline","prompt":"$25","starter":"trace_retrieval = [\n {\"turn\": 1, \"stop_reason\": \"tool_use\", \"tools_called\": [\"search\"], \"validation_errors\": 1},\n {\"turn\": 2, \"stop_reason\": \"tool_use\", \"tools_called\": [\"search\"], \"validation_errors\": 0},\n {\"turn\": 3, \"stop_reason\": \"end_turn\", \"tools_called\": [], \"validation_errors\": 0},\n]\n\ntrace_hallucination = [\n {\"turn\": 1, \"stop_reason\": \"tool_use\", \"tools_called\": [], \"validation_errors\": 0},\n {\"turn\": 2, \"stop_reason\": \"end_turn\", \"tools_called\": [], \"validation_errors\": 0},\n]\n\ntrace_truncated = [\n {\"turn\": 1, \"stop_reason\": \"tool_use\", \"tools_called\": [\"search\"], \"validation_errors\": 0},\n {\"turn\": 2, \"stop_reason\": \"max_tokens\", \"tools_called\": [], \"validation_errors\": 0},\n]\n\n# define triage(trace) below\n\nfor tr in [trace_retrieval, trace_hallucination, trace_truncated]:\n r = triage(tr)\n print(f\"class={r['class']} fix_layer={r['fix_layer']} summary={r['summary']}\")\n","grader":{"kind":"stdout-equality","expected":"class=retrieval fix_layer=retriever / tool inputs summary=3 turns, 2 tools, 1 validation_errors, final stop=end_turn\nclass=true_hallucination fix_layer=constrain output (force tool use, require citations) summary=2 turns, 0 tools, 0 validation_errors, final stop=end_turn\nclass=prompt fix_layer=prompt (output too long; cap or split task) summary=2 turns, 1 tools, 0 validation_errors, final stop=max_tokens\n","normalize":"collapse-trailing-newline"},"solution":"$26","passThreshold":0.66}],"xpTotal":32},{"slug":"the-four-breakage-classes","title":"The four breakage classes — sort any LLM failure before you touch the prompt","estMinutes":14,"prerequisites":["read-the-trace-not-the-chat"],"status":"live","steps":[{"type":"read","id":"debugging-output/the-four-breakage-classes/s01-01-intro","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":130,"concept":"four-classes-taxonomy","body":"$27","cta":"Got it","code":"$28","runnable":true},{"type":"mc","id":"debugging-output/the-four-breakage-classes/s02-02-classify-the-refund-bug","xp":2,"hint":[{"level":1,"body":"The model's output is consistent with what retrieval handed it. The retrieved chunk is from the wrong account. That points at one layer.","cost":0}],"personalize":false,"phase":"warmup","estSeconds":65,"concept":"classify-from-symptom","prompt":"A chatbot told a customer their refund was processed. It wasn't.\nThe trace shows retrieval pulled ticket `ALT-CUST-04` instead of\nthe customer's actual account `CUST-04` — different customer,\nsimilar ID. The model read the ticket and answered honestly from it.\n\nWhich class is this?\n","options":[{"id":"retrieval","label":"Class 1 — retrieval gave wrong context"},{"id":"prompt","label":"Class 2 — prompt was ambiguous","explain":"The prompt was clear: 'look up the refund status'. The model did exactly that. The data it received was wrong, not the instructions."},{"id":"hallucination","label":"Class 3 — true hallucination","explain":"The model didn't invent anything. It read a real ticket from retrieval and reported what it said. The ticket was just the wrong one."},{"id":"parse","label":"Class 4 — downstream code mangled the output","explain":"`raw_output` and `output_after_postprocess` match exactly. Nothing mangled the model's output. The bug is upstream."}],"answerIds":["retrieval"],"shuffle":false,"code":"# the support ticket\nticket = \"Your AI assistant told me my refund was processed. It hasn't been.\"\n\n# the trace says:\ntrace_summary = {\n \"rendered_prompt\": \"Look up the refund status for the user's most recent ticket and answer.\",\n \"tools_called\": [\"search_tickets\"],\n \"retrieved_chunks\": [\n \"ticket_id=84211, account=ALT-CUST-04, status=refunded, amount=$48.20\"\n ],\n \"raw_output\": \"Yes — your refund of $48.20 was processed.\",\n \"output_after_postprocess\": \"Yes — your refund of $48.20 was processed.\",\n}\n\n# the customer's actual account is CUST-04. retrieval pulled ALT-CUST-04.\nprint(trace_summary)\n","runnable":true},{"type":"read","id":"debugging-output/the-four-breakage-classes/s03-03-the-trace-anatomy","xp":2,"hint":[],"personalize":false,"phase":"build","estSeconds":120,"concept":"trace-fields-by-class","body":"$29","cta":"Got it","code":"$2a","runnable":true},{"type":"predict","id":"debugging-output/the-four-breakage-classes/s04-04-predict-the-class","xp":3,"hint":[{"level":1,"body":"Walk the priority order. retrieved_chunks is empty (no class-1 evidence). prompt is unambiguous (no class-2 evidence). raw matches postprocessed (no class-4 evidence). The model still answered with a specific date. That last fact is the signature of class 3 — the model invented an answer because retrieval gave it nothing to ground in.","cost":0}],"personalize":false,"phase":"build","estSeconds":90,"concept":"walk-a-trace","code":"# the trace from a real failure.\n# rendered_prompt is clear. retrieval returned nothing.\n# raw_output and output_after_postprocess are identical.\n# the model still answered, confidently.\n\ntrace = {\n \"rendered_prompt\": \"When did Acme sign their enterprise contract with us?\",\n \"tools_called\": [\"search_contracts\"],\n \"retrieved_chunks\": [],\n \"raw_output\": \"Acme signed their enterprise contract on March 14, 2024.\",\n \"output_after_postprocess\": \"Acme signed their enterprise contract on March 14, 2024.\",\n}\n\n# walk it.\nprint(f\"retrieved_chunks empty? {trace['retrieved_chunks'] == []}\")\nprint(f\"prompt clear? True\")\nprint(f\"raw == postprocessed? {trace['raw_output'] == trace['output_after_postprocess']}\")\nprint(f\"model answered anyway? True\")\n\n# which class is this?\nprint(\"class: hallucination\")\n","prompt":"Read the trace in the editor. The prompt is clear. The retriever\nreturned zero chunks. The raw output and post-processed output\nare identical. The model still produced a confident, specific\ndate answer.\n\nPredict the output of the script — specifically the final\n`class:` line.\n\nExpected output:\n```\nretrieved_chunks empty? True\nprompt clear? True\nraw == postprocessed? True\nmodel answered anyway? True\nclass: hallucination\n```\n","grader":{"kind":"stdout-equality","expected":"retrieved_chunks empty? True\nprompt clear? True\nraw == postprocessed? True\nmodel answered anyway? True\nclass: hallucination\n","normalize":"collapse-trailing-newline"}},{"type":"fill","id":"debugging-output/the-four-breakage-classes/s05-05-fill-the-class-name","xp":3,"hint":[{"level":1,"body":"Empty retrieval + confident answer means the model had nothing to ground in and answered anyway — that's the hallucination class. Raw differs from post-processed means the bug is in the code between the model and the user — that's the parse class.","cost":0},{"level":2,"body":"First blank: `\"hallucination\"`. Second blank: `\"parse\"`. Both as Python strings.","cost":0}],"personalize":false,"phase":"build","estSeconds":70,"concept":"class-naming","prompt":"Two trace signatures. Fill in the class name each one describes.\n\nUse the exact class names from the taxonomy:\n`\"retrieval\"`, `\"prompt\"`, `\"hallucination\"`, `\"parse\"`.\n\nExpected output:\n```\nempty retrieval, confident answer: hallucination\nraw_output != output_after_postprocess: parse\n```\n","code":"# signature 1: retrieved_chunks is empty, but the model still\n# answered confidently with a specific fact. no grounding, no\n# citation, just made up.\nclass_for_empty_retrieval = ___\n\n# signature 2: raw_output is correct JSON, but\n# output_after_postprocess has a different value. something in\n# the post-processing layer mangled it.\nclass_for_postprocess_diff = ___\n\nprint(f\"empty retrieval, confident answer: {class_for_empty_retrieval}\")\nprint(f\"raw_output != output_after_postprocess: {class_for_postprocess_diff}\")\n","blanks":[{"id":"class_for_empty_retrieval","accept":["\"hallucination\"","'hallucination'"],"caseSensitive":true,"normalize":"trim"},{"id":"class_for_postprocess_diff","accept":["\"parse\"","'parse'"],"caseSensitive":true,"normalize":"trim"}]},{"type":"fix","id":"debugging-output/the-four-breakage-classes/s06-06-fix-the-misclassification","xp":4,"hint":[{"level":1,"body":"Replace the single-line body with four if-checks in priority order. Check retrieved_chunks first (class 1), then raw vs postprocessed (class 4), then empty retrieved_chunks (class 3), then fall through to prompt (class 2).","cost":0},{"level":2,"body":"Replace the body of `classify_failure` with:\n\n```python\nif trace[\"retrieved_chunks\"] and not trace[\"retrieved_chunks_match_query\"]:\n return \"retrieval\"\nif trace[\"raw_output\"] != trace[\"output_after_postprocess\"]:\n return \"parse\"\nif not trace[\"retrieved_chunks\"]:\n return \"hallucination\"\nreturn \"prompt\"\n```\n","cost":0}],"personalize":false,"phase":"build","estSeconds":110,"concept":"dont-blame-the-model","brokenCode":"trace = {\n \"rendered_prompt\": \"Look up the user's refund status.\",\n \"retrieved_chunks\": [\"ticket from wrong account\"],\n \"retrieved_chunks_match_query\": False,\n \"raw_output\": \"Your refund was processed.\",\n \"output_after_postprocess\": \"Your refund was processed.\",\n}\n\ndef classify_failure(trace):\n # bug: every failure gets blamed on the model. lazy.\n return \"hallucination\"\n\nprint(f\"class: {classify_failure(trace)}\")\n","prompt":"The `classify_failure(trace)` below is the lazy default: it\nreturns `\"hallucination\"` for every failure. That's the worst\nhabit on a debugging team — \"the model hallucinated\" is what you\nsay when you don't want to read the trace.\n\nFix the function to check the trace fields in priority order:\n\n1. If `retrieved_chunks` is non-empty AND\n `retrieved_chunks_match_query` is `False`, return `\"retrieval\"`.\n2. Else if `raw_output != output_after_postprocess`, return\n `\"parse\"`.\n3. Else if `retrieved_chunks` is empty, return `\"hallucination\"`.\n4. Else return `\"prompt\"`.\n\nThe trace in the editor is a class-1 retrieval bug (retrieval\nreturned the wrong account's ticket). The fixed function should\nreturn `\"retrieval\"`.\n\nExpected output:\n```\nclass: retrieval\n```\n","grader":{"kind":"stdout-equality","expected":"class: retrieval","normalize":"collapse-trailing-newline"},"bugLines":[10],"revealAfter":4},{"type":"fix","id":"debugging-output/the-four-breakage-classes/s07-07-fix-the-parse-mangle","xp":4,"hint":[{"level":1,"body":"The intent was \"strip backticks\" — the backtick character is `` ` ``. The pattern `r\"\\$\"` matches a literal dollar sign. Replace the pattern with one that matches a literal backtick.","cost":0},{"level":2,"body":"Change line 8 to:\n\n```python\noutput_after_postprocess = re.sub(r\"`\", \"\", raw_output)\n```\n\nSince the raw output contains no backticks, the substitution is a no-op and the dollar sign is preserved.\n","cost":0}],"personalize":false,"phase":"build","estSeconds":110,"concept":"class-4-parse-bug","brokenCode":"import re\n\nraw_output = '{\"refund\": \"$48.20\"}'\n\n# bug: this regex strips the dollar sign. it was supposed to strip\n# backticks. someone wrote `\\$` thinking it meant \"the literal\n# backtick character\" — it doesn't.\noutput_after_postprocess = re.sub(r\"\\$\", \"\", raw_output)\n\nprint(f\"raw: {raw_output}\")\nprint(f\"postprocessed: {output_after_postprocess}\")\nprint(\"class: parse (bug was in regex, not model)\")\n","prompt":"The model returned correct JSON: `{\"refund\": \"$48.20\"}`. A\ndownstream \"currency cleaner\" regex was supposed to strip junk\ncharacters but instead strips the `$` — turning `$48.20` into\n`48.20`. The user reads `48.20` and asks support what currency.\n\nThe model was right. The trace's `raw_output` is correct. The\nbug is in the regex on the post-processing line.\n\nFix the regex so it strips literal backticks (`` ` ``) — which\nis what the cleaner was meant to do — and leaves the `$`\nuntouched.\n\nExpected output:\n```\nraw: {\"refund\": \"$48.20\"}\npostprocessed: {\"refund\": \"$48.20\"}\nclass: parse (bug was in regex, not model)\n```\n","grader":{"kind":"stdout-equality","expected":"raw: {\"refund\": \"$48.20\"}\npostprocessed: {\"refund\": \"$48.20\"}\nclass: parse (bug was in regex, not model)\n","normalize":"collapse-trailing-newline"},"bugLines":[8],"revealAfter":4},{"type":"write","id":"debugging-output/the-four-breakage-classes/s08-08-write-the-classifier","xp":5,"hint":[{"level":1,"body":"Four branches, checked in priority order. The order matters — checking hallucination before parse would mislabel a parse bug as a hallucination when retrieval happened to be empty.","cost":0},{"level":2,"body":"```python\ndef classify_failure(trace):\n if trace[\"retrieved_chunks\"] and not trace[\"retrieved_chunks_match_query\"]:\n return \"retrieval\"\n if trace[\"raw_output\"] != trace[\"output_after_postprocess\"]:\n return \"parse\"\n if not trace[\"retrieved_chunks\"]:\n return \"hallucination\"\n return \"prompt\"\n```\n","cost":0}],"personalize":false,"phase":"build","estSeconds":220,"concept":"write-classify-failure","prompt":"Write `classify_failure(trace)` that returns one of\n`\"retrieval\"`, `\"prompt\"`, `\"hallucination\"`, or `\"parse\"`.\n\nApply the priority order from the chapter:\n\n1. If `trace[\"retrieved_chunks\"]` is non-empty AND\n `trace[\"retrieved_chunks_match_query\"]` is `False` →\n `\"retrieval\"`.\n2. Else if `trace[\"raw_output\"] != trace[\"output_after_postprocess\"]`\n → `\"parse\"`.\n3. Else if `trace[\"retrieved_chunks\"]` is empty (the list `[]`) →\n `\"hallucination\"`.\n4. Else → `\"prompt\"`.\n\nThree traces run for you, one per branch. Expected output:\n```\nretrieval\nparse\nhallucination\n```\n","starter":"trace_retrieval = {\n \"rendered_prompt\": \"Look up the user's refund.\",\n \"retrieved_chunks\": [\"wrong-account ticket\"],\n \"retrieved_chunks_match_query\": False,\n \"raw_output\": \"Your refund was processed.\",\n \"output_after_postprocess\": \"Your refund was processed.\",\n}\n\ntrace_parse = {\n \"rendered_prompt\": \"Return refund amount as JSON.\",\n \"retrieved_chunks\": [\"right ticket\"],\n \"retrieved_chunks_match_query\": True,\n \"raw_output\": '{\"refund\": \"$48.20\"}',\n \"output_after_postprocess\": '{\"refund\": \"48.20\"}',\n}\n\ntrace_hallucination = {\n \"rendered_prompt\": \"When did Acme sign their contract?\",\n \"retrieved_chunks\": [],\n \"retrieved_chunks_match_query\": None,\n \"raw_output\": \"Acme signed on March 14, 2024.\",\n \"output_after_postprocess\": \"Acme signed on March 14, 2024.\",\n}\n\n# define classify_failure(trace) below\n\nfor tr in [trace_retrieval, trace_parse, trace_hallucination]:\n print(classify_failure(tr))\n","grader":{"kind":"stdout-equality","expected":"retrieval\nparse\nhallucination\n","normalize":"collapse-trailing-newline"},"solution":"$2b","hiddenTests":[]},{"type":"checkpoint","id":"debugging-output/the-four-breakage-classes/s09-09-checkpoint","xp":8,"hint":[],"personalize":false,"phase":"check","estSeconds":240,"concept":"classify-three-traces","prompt":"Final drill. You got three user-reported failures from the support\ninbox this morning. Each has a trace attached. Classify them so\nthe right team gets the right ticket — retrieval team, prompt\nteam, ML team (hallucination), or backend team (parse).\n\nWrite `classify_failure(trace)` using the same priority order\nfrom the previous step:\n\n1. retrieved_chunks non-empty + retrieved_chunks_match_query False\n → `\"retrieval\"`\n2. raw_output != output_after_postprocess → `\"parse\"`\n3. retrieved_chunks empty → `\"hallucination\"`\n4. else → `\"prompt\"`\n\nThen print one line per trace in this format:\n`ticket_: `\n\nExpected output:\n```\nticket_8821: retrieval\nticket_8822: prompt\nticket_8823: parse\n```\n","starter":"$2c","grader":{"kind":"stdout-equality","expected":"ticket_8821: retrieval\nticket_8822: prompt\nticket_8823: parse\n","normalize":"collapse-trailing-newline"},"solution":"$2d","passThreshold":0.66}],"xpTotal":32},{"slug":"five-real-postmortems","title":"Five postmortems — three public, two composite — and the one fix that would have caught all of them","estMinutes":22,"prerequisites":["the-four-breakage-classes"],"status":"live","steps":[{"type":"read","id":"debugging-output/five-real-postmortems/s01-01-the-postmortem-template","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":200,"concept":"postmortem-template","body":"$2e","cta":"Got it","runnable":true},{"type":"read","id":"debugging-output/five-real-postmortems/s02-02-air-canada-refund-lawsuit","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":220,"concept":"case-air-canada","body":"$2f","cta":"Got it","runnable":true},{"type":"read","id":"debugging-output/five-real-postmortems/s03-03-dpd-and-nyc-mycity","xp":1,"hint":[],"personalize":false,"phase":"build","estSeconds":220,"concept":"case-dpd-mycity","body":"$30","cta":"Got it","runnable":true},{"type":"mc","id":"debugging-output/five-real-postmortems/s04-04-classify-the-five","xp":2,"hint":[{"level":1,"body":"A: 'pulled a stale doc' = wrong context retrieved = class 1. B: 'generated a policy that does not exist' = class 3. C: 'raw output right, post-processing dropped the symbol' = class 4. D: 'vague instruction, model picked a reasonable interpretation' = class 2. E: 'no retrieval, generated from training' = class 3.","cost":0}],"personalize":false,"phase":"build","estSeconds":80,"concept":"classify-five-failures","prompt":"Five short failure descriptions from real (or real-shaped) AI\nincidents. Apply the four-class taxonomy from the previous lesson.\n\nWhich mapping is correct?\n","options":[{"id":"a1_b3_c4_d2_e3","label":"A=1, B=3, C=4, D=2, E=3"},{"id":"a3_b1_c4_d2_e1","label":"A=3, B=1, C=4, D=2, E=1","explain":"A is retrieval pulling the wrong doc (stale = wrong context, class 1), not a hallucination. B is the Air Canada shape — invented policy with no source, class 3 (or class 2 if you frame it as missing prompt constraint; either way it is not class 1)."},{"id":"a1_b2_c4_d3_e3","label":"A=1, B=2, C=4, D=3, E=3","explain":"Close but D is wrong. The summarizer got correct context and a vague instruction; that is the textbook class 2 ambiguity case, not a hallucination."},{"id":"a1_b3_c2_d2_e4","label":"A=1, B=3, C=2, D=2, E=4","explain":"C and E are swapped. JSON mangled by downstream code is class 4 (parse). A bot generating from training data with no retrieval is class 3 (hallucination)."}],"answerIds":["a1_b3_c4_d2_e3"],"shuffle":false,"code":"# five real-ish failure descriptions. apply the chapter 3 taxonomy:\n# 1 = retrieval gave wrong context\n# 2 = prompt was ambiguous\n# 3 = true hallucination (no grounding)\n# 4 = downstream code mangled the output\nfailures = [\n \"A: RAG bot pulled a 2021 policy doc when the 2024 update existed; answered from the stale doc.\",\n \"B: Chatbot generated a confident bereavement-refund policy that does not exist anywhere in the company's docs.\",\n \"C: Model returned valid JSON with the right field names; downstream parser dropped the currency symbol from the value.\",\n \"D: Summarizer was told 'give me the highlights'; returned 14 bullets when the UI had room for 5.\",\n \"E: Support bot answered a municipal-law question by generating from training data with no retrieval grounding.\",\n]\nfor f in failures:\n print(f)\n","runnable":true},{"type":"read","id":"debugging-output/five-real-postmortems/s05-05-recruiter-json-mangle","xp":1,"hint":[],"personalize":false,"phase":"build","estSeconds":200,"concept":"case-recruiter-json","body":"$31","cta":"Got it","runnable":true},{"type":"read","id":"debugging-output/five-real-postmortems/s06-06-glean-style-retrieval-miss","xp":1,"hint":[],"personalize":false,"phase":"build","estSeconds":200,"concept":"case-retrieval-staleness","body":"$32","cta":"Got it","runnable":true},{"type":"mc","id":"debugging-output/five-real-postmortems/s07-07-which-fix-catches-the-most","xp":2,"hint":[{"level":1,"body":"Notice that every other answer is downstream of 'did we test for this?' A better prompt only helps if you have a way to verify it helped. Schema validation only helps if you have tests that prove it catches the bug. Retrieval-with-citations only helps if you have evals that confirm the citation is current. The eval suite is the meta-fix — every other fix is one of the *outcomes* of having one. The five teams that shipped these failures all had teams who could have written the fixes. None of them had the eval that would have shown them they needed to.","cost":0}],"personalize":false,"phase":"build","estSeconds":70,"concept":"meta-fix-eval-suite","prompt":"Look back at the five cases:\n\n1. Air Canada — invented a bereavement-refund policy\n2. DPD — chatbot swore at customers, generated insults of own company\n3. NYC MyCity — gave illegal small-business advice\n4. Recruiter agent — swapped email into phone-number slot\n5. Enterprise RAG — cited a superseded 2022 policy as current\n\nYou can pick exactly one fix to apply across all five teams. Which\nfix would have caught the MOST of these cases before they reached a\ncustomer?\n","options":[{"id":"better_prompt","label":"A better-engineered system prompt with stricter rules","explain":"A better prompt helps Air Canada and arguably DPD. It does nothing for the recruiter's JSON-slot swap (the prompt was clear), nothing for the enterprise RAG staleness (retrieval problem), and only weakly addresses MyCity (the bot needed retrieval grounding, not a stricter tone). One or two cases at best."},{"id":"eval_suite","label":"A regression eval suite run in CI on every model/prompt/retriever change"},{"id":"schema_validation","label":"Pydantic-style schema validation at every agent output boundary","explain":"Schema validation catches the recruiter case cleanly. It would not have caught Air Canada (the output was a fluent string, schema-valid by any measure), DPD (no schema), MyCity (no schema), or the enterprise RAG stale-citation (the cited URL was valid). One case caught."},{"id":"retrieval_citations","label":"Mandatory retrieval-with-citations on every response","explain":"Retrieval with citations catches DPD, MyCity, and the enterprise RAG case (citation-with-date is a sub-fix of citation discipline). It does not catch Air Canada cleanly (the bot was making up policy whether or not it had retrieval) and does nothing for the recruiter's JSON-slot swap. Three cases caught — strong, but not the strongest."}],"answerIds":["eval_suite"],"shuffle":false,"runnable":true},{"type":"write","id":"debugging-output/five-real-postmortems/s08-08-write-the-postmortem","xp":5,"hint":[{"level":1,"body":"Five `if` branches, checked in priority order, each returning a dict literal with `class` first and `fix` second. The order matters — Python preserves insertion order in dicts since 3.7, so building the dict with `class` set first guarantees the printed repr matches the expected output.","cost":0},{"level":2,"body":"```python\ndef classify_failure(case):\n snippet = case[\"trace_snippet\"]\n if \"no retrieval\" in snippet:\n return {\"class\": 3, \"fix\": \"add retrieval-with-citations + out-of-domain refusal\"}\n if \"schema mismatch\" in snippet:\n return {\"class\": 4, \"fix\": \"add Pydantic validation at the consumer boundary\"}\n if \"stale chunk\" in snippet:\n return {\"class\": 1, \"fix\": \"add freshness metadata + superseded_by filter\"}\n if \"ambiguous\" in snippet:\n return {\"class\": 2, \"fix\": \"tighten prompt with explicit constraint + negative example\"}\n return {\"class\": 0, \"fix\": \"unclassified — read the full trace\"}\n```\n","cost":0}],"personalize":false,"phase":"build","estSeconds":220,"concept":"heuristic-classify-from-postmortem","prompt":"$33","starter":"air_canada = {\n \"user_saw\": \"Chatbot told me I could book now and claim a bereavement refund retroactively.\",\n \"trace_snippet\": \"no retrieval grounding; model generated policy from training data\",\n \"system_state\": \"RAG layer present but bypassed on policy questions\",\n}\n\nrecruiter = {\n \"user_saw\": \"My SMS went to jane.doe@gmail.com instead of a phone number.\",\n \"trace_snippet\": \"schema mismatch: candidate_email value found in phone_number slot\",\n \"system_state\": \"no validation at queue-worker boundary\",\n}\n\n# define classify_failure(case) below\n\nprint(f\"air_canada: {classify_failure(air_canada)}\")\nprint(f\"recruiter: {classify_failure(recruiter)}\")\n","grader":{"kind":"stdout-equality","expected":"air_canada: {'class': 3, 'fix': 'add retrieval-with-citations + out-of-domain refusal'}\nrecruiter: {'class': 4, 'fix': 'add Pydantic validation at the consumer boundary'}\n","normalize":"collapse-trailing-newline"},"solution":"$34","hiddenTests":[]},{"type":"checkpoint","id":"debugging-output/five-real-postmortems/s09-09-checkpoint","xp":8,"hint":[],"personalize":false,"phase":"check","estSeconds":240,"concept":"triage-five-postmortems","prompt":"Final drill. You have five postmortem records from the inbox. Each\none already carries a `class` field (1-4) and a `severity` field\n(1-10, where 10 is \"front-page lawsuit\").\n\nWrite `triage_failures(cases)` that returns a dict with two keys,\nin this exact order:\n\n- `counts`: a dict mapping class number (int) to count (int), with\n keys inserted in ascending class order (1, 2, 3, 4). Classes\n with zero count are omitted.\n- `urgent`: the `name` (str) of the case with the highest severity.\n On a tie, return the FIRST one encountered (stable order).\n\nFive cases run for you. Expected output:\n```\n{'counts': {1: 1, 2: 1, 3: 2, 4: 1}, 'urgent': 'air_canada'}\n```\n","starter":"cases = [\n {\"name\": \"air_canada\", \"class\": 2, \"severity\": 10},\n {\"name\": \"dpd\", \"class\": 3, \"severity\": 7},\n {\"name\": \"nyc_mycity\", \"class\": 3, \"severity\": 9},\n {\"name\": \"recruiter_json\", \"class\": 4, \"severity\": 6},\n {\"name\": \"enterprise_rag\", \"class\": 1, \"severity\": 8},\n]\n\n# define triage_failures(cases) below\n\nprint(triage_failures(cases))\n","grader":{"kind":"stdout-equality","expected":"{'counts': {1: 1, 2: 1, 3: 2, 4: 1}, 'urgent': 'air_canada'}","normalize":"collapse-trailing-newline"},"solution":"cases = [\n {\"name\": \"air_canada\", \"class\": 2, \"severity\": 10},\n {\"name\": \"dpd\", \"class\": 3, \"severity\": 7},\n {\"name\": \"nyc_mycity\", \"class\": 3, \"severity\": 9},\n {\"name\": \"recruiter_json\", \"class\": 4, \"severity\": 6},\n {\"name\": \"enterprise_rag\", \"class\": 1, \"severity\": 8},\n]\n\ndef triage_failures(cases):\n counts = {}\n for c in cases:\n counts[c[\"class\"]] = counts.get(c[\"class\"], 0) + 1\n ordered_counts = {k: counts[k] for k in sorted(counts)}\n urgent = max(cases, key=lambda c: c[\"severity\"])[\"name\"]\n return {\"counts\": ordered_counts, \"urgent\": urgent}\n\nprint(triage_failures(cases))\n","passThreshold":0.66}],"xpTotal":22},{"slug":"prediction-failure-triage","title":"Triage the bad prediction — reproduce with the logged input, then route by class","estMinutes":14,"prerequisites":["the-four-breakage-classes"],"status":"live","steps":[{"type":"read","id":"debugging-output/prediction-failure-triage/s01-01-reproduce-classify-route","xp":1,"hint":[],"personalize":false,"phase":"warmup","estSeconds":150,"concept":"triage-runbook","body":"$35","cta":"Got it","code":"# one bad prediction, walked through the whole runbook.\nreport = {\n \"complaint\": \"the assistant said my invoice total was $1,240. it's $940.\",\n \"logged_input\": \"total for INV-8814, exclude the voided line\",\n \"reproduced\": True, # re-ran the pipeline on the logged input\n \"failure_class\": \"retrieval\", # from classify_failure(), lesson 02\n}\n\nROUTES = {\n \"retrieval\": \"retrieval pipeline (data in)\",\n \"prompt\": \"prompt + spec (what you asked for)\",\n \"hallucination\": \"grounding + guardrails (the model itself)\",\n \"parse\": \"post-processing code (data out)\",\n}\n\nif not report[\"reproduced\"]:\n print(\"triage blocked: reproduce with the logged input first\")\nelse:\n print(f\"class: {report['failure_class']}\")\n print(f\"route: {ROUTES[report['failure_class']]}\")\n","runnable":true},{"type":"mc","id":"debugging-output/prediction-failure-triage/s02-02-first-move","xp":2,"hint":[{"level":1,"body":"Two of the options act before knowing what broke, and one acts on\na different input than the one that failed. Only one re-creates\nthe actual failure, byte for byte.\n","cost":0}],"personalize":false,"phase":"build","estSeconds":70,"concept":"reproduce-before-touching","prompt":"A bad-prediction report just landed. Support pasted their own\none-line summary of what the customer asked. The trace has the\ncustomer's exact input. What is your first move?\n","options":[{"id":"paraphrase","label":"Re-type support's summary into the pipeline and see if it fails.","explain":"Compare the two strings in the editor — the summary drops the\n\"exclude the voided line\" clause, which is exactly where the bug\nlives. The paraphrase will likely produce a correct answer, and\nyou'll close a bug that's still in production.\n"},{"id":"prompt-edit","label":"Add \"be careful with voided line items\" to the system prompt.","explain":"A fix before a diagnosis. You don't yet know whether this is a\nretrieval, prompt, hallucination, or parse failure — a prompt\npatch only helps one of those four, and you'd ship it blind.\n"},{"id":"logged","label":"Re-run the pipeline on the exact `logged_input` from the trace."},{"id":"blame","label":"File it as \"model hallucination\" and move on.","explain":"That's the shrug, not a category. The trace shows a specific\ninput and a specific wrong output — until you reproduce and\nclassify, \"hallucination\" is an accusation, not a diagnosis.\n"}],"answerIds":["logged"],"shuffle":false,"code":"# the report, as it arrived. note what support typed vs what was logged.\nreport = {\n \"complaint\": \"AI told the customer a voided line item still counted\",\n \"typed_by_support\": \"what's the total on invoice 8814?\",\n}\ntrace = {\n \"logged_input\": \"total for INV-8814, exclude the voided line\",\n \"raw_output\": \"$1,240\",\n}\nprint(report[\"typed_by_support\"])\nprint(trace[\"logged_input\"])\n","runnable":true},{"type":"predict","id":"debugging-output/prediction-failure-triage/s03-03-predict-the-route","xp":3,"hint":[{"level":1,"body":"The gate runs before the routing lookup. Ticket 512 never reaches\nthe `routes` dict — its `\"retrieval\"` label is dead weight until\nsomeone reproduces the failure. Ticket 511 passes the gate and\nlooks up `\"parse\"`.\n","cost":0}],"personalize":false,"phase":"build","estSeconds":90,"concept":"triage-tree-as-code","code":"# the triage tree as code. gate first, then route by class.\ndef triage(report):\n if not report[\"reproduced_with_logged_input\"]:\n return \"blocked: reproduce first\"\n routes = {\n \"retrieval\": \"retrieval pipeline\",\n \"prompt\": \"prompt + spec\",\n \"hallucination\": \"grounding + guardrails\",\n \"parse\": \"post-processing code\",\n }\n return \"route to \" + routes[report[\"failure_class\"]]\n\nreports = [\n {\"id\": 511, \"reproduced_with_logged_input\": True, \"failure_class\": \"parse\"},\n {\"id\": 512, \"reproduced_with_logged_input\": False, \"failure_class\": \"retrieval\"},\n]\n\nfor r in reports:\n print(f\"ticket_{r['id']}: {triage(r)}\")\n","prompt":"Two tickets hit the triage function. Ticket 511 was reproduced\nfrom its logged input and classified as a parse failure. Ticket\n512 has a class attached — but nobody reproduced it yet.\n\nPredict both output lines. Watch the gate: does 512's\n`failure_class` matter at all?\n","grader":{"kind":"stdout-equality","expected":"ticket_511: route to post-processing code\nticket_512: blocked: reproduce first\n","normalize":"collapse-trailing-newline"}},{"type":"fill","id":"debugging-output/prediction-failure-triage/s04-04-fill-the-owners","xp":3,"hint":[{"level":1,"body":"Symptom 1 is a data problem — the wrong context went in, so it\nroutes to the layer that feeds the model. Symptom 2 is an\nexpectation problem — the priority order from lesson 02 falls\nthrough to it once retrieval, parse, and empty-chunks are ruled\nout.\n","cost":0},{"level":2,"body":"First blank: `\"retrieval\"`. Second blank: `\"prompt\"`. Both as\nquoted Python strings — they're dict keys into `ROUTES`.\n","cost":0}],"personalize":false,"phase":"build","estSeconds":70,"concept":"route-the-class","prompt":"Two symptoms, straight from this morning's traces. Fill each\nblank with the class name (as a Python string) that routes the\nticket to the right owner.\n\nClass names: `\"retrieval\"`, `\"prompt\"`, `\"hallucination\"`,\n`\"parse\"`.\n\nExpected output:\n```\nwrong chunk -> retrieval pipeline (data in)\nambiguous ask -> prompt + spec (what you asked)\n```\n","code":"ROUTES = {\n \"retrieval\": \"retrieval pipeline (data in)\",\n \"prompt\": \"prompt + spec (what you asked)\",\n \"hallucination\": \"grounding + guardrails (the model)\",\n \"parse\": \"post-processing code (data out)\",\n}\n\n# symptom 1: the chunk in the trace is from the wrong account.\n# the model answered honestly from bad data.\nwrong_chunk_class = ___\n\n# symptom 2: clean context, output untouched by post-processing —\n# but the model picked a reasonable-but-wrong reading of your ask.\nambiguous_ask_class = ___\n\nprint(f\"wrong chunk -> {ROUTES[wrong_chunk_class]}\")\nprint(f\"ambiguous ask -> {ROUTES[ambiguous_ask_class]}\")\n","blanks":[{"id":"wrong_chunk_class","accept":["\"retrieval\"","'retrieval'"],"caseSensitive":true,"normalize":"trim"},{"id":"ambiguous_ask_class","accept":["\"prompt\"","'prompt'"],"caseSensitive":true,"normalize":"trim"}]},{"type":"fix","id":"debugging-output/prediction-failure-triage/s05-05-fix-the-paraphrase-repro","xp":4,"hint":[{"level":1,"body":"Compare the two input strings. Support's paraphrase never mentions\nthe voided line, so `rerun_pipeline` takes the happy branch and\nreturns the correct total. The failing input is sitting one dict\nover, in `trace`.\n","cost":0},{"level":2,"body":"Change line 18 to:\n\n```python\noutput = rerun_pipeline(trace[\"logged_input\"])\n```\n\nNow the input contains \"voided\", the stand-in pipeline returns the\nwrong `$1,240` again, and the failure is reproduced.\n","cost":0}],"personalize":false,"phase":"build","estSeconds":110,"concept":"repro-with-logged-input","brokenCode":"report = {\n \"complaint\": \"your AI says my invoice total is wrong\",\n \"typed_by_support\": \"what's the total on my invoice?\",\n}\n\ntrace = {\n \"logged_input\": \"total for INV-8814, exclude the voided line\",\n}\n\ndef rerun_pipeline(user_input):\n # deterministic stand-in for the real pipeline. the bug only\n # fires when the voided-line instruction is in the input.\n if \"voided\" in user_input:\n return \"$1,240\" # still counts the voided line. wrong.\n return \"$940\"\n\n# reproduce the failure before touching anything\noutput = rerun_pipeline(report[\"typed_by_support\"])\n\nif output == \"$1,240\":\n print(\"reproduced: yes - now classify it\")\nelse:\n print(\"reproduced: no - do not touch the prompt yet\")\n","prompt":"This repro script feeds the pipeline support's paraphrase of the\ncustomer's question — and the paraphrase drops the clause that\ntriggers the bug. The script cheerfully reports the failure as\nunreproducible while the bug sits live in production.\n\nFix the reproduction line so it re-runs the pipeline on the\nexact `logged_input` from the trace.\n\nExpected output:\n```\nreproduced: yes - now classify it\n```\n","grader":{"kind":"stdout-equality","expected":"reproduced: yes - now classify it","normalize":"collapse-trailing-newline"},"bugLines":[18],"revealAfter":3},{"type":"write","id":"debugging-output/prediction-failure-triage/s06-06-write-the-router","xp":5,"hint":[{"level":1,"body":"One membership check (`failure_class in ROUTES`), one f-string on\nthe hit, one fixed rejection string on the miss. The router is\ndeliberately strict — an unknown label means someone skipped the\nclassify step, and the function sends them back to it.\n","cost":0}],"personalize":false,"phase":"build","estSeconds":150,"concept":"write-the-router","prompt":"Write `route_failure(failure_class)`:\n\n- If the class is a key in `ROUTES`, return\n `\"route to \"` using the dict value.\n- Anything else — including the eternal favorite,\n `\"the model is dumb\"` — returns\n `\"not a triage category - go classify the trace\"`.\n\nThree inputs run, one of them the shrug. Expected output:\n```\nroute to post-processing code\nnot a triage category - go classify the trace\nroute to retrieval pipeline\n```\n","starter":"ROUTES = {\n \"retrieval\": \"retrieval pipeline\",\n \"prompt\": \"prompt + spec\",\n \"hallucination\": \"grounding + guardrails\",\n \"parse\": \"post-processing code\",\n}\n\n# define route_failure(failure_class) below\n\nfor c in [\"parse\", \"the model is dumb\", \"retrieval\"]:\n print(route_failure(c))\n","grader":{"kind":"stdout-equality","expected":"route to post-processing code\nnot a triage category - go classify the trace\nroute to retrieval pipeline\n","normalize":"collapse-trailing-newline"},"solution":"ROUTES = {\n \"retrieval\": \"retrieval pipeline\",\n \"prompt\": \"prompt + spec\",\n \"hallucination\": \"grounding + guardrails\",\n \"parse\": \"post-processing code\",\n}\n\ndef route_failure(failure_class):\n if failure_class in ROUTES:\n return f\"route to {ROUTES[failure_class]}\"\n return \"not a triage category - go classify the trace\"\n\nfor c in [\"parse\", \"the model is dumb\", \"retrieval\"]:\n print(route_failure(c))\n","hiddenTests":[]},{"type":"checkpoint","id":"debugging-output/prediction-failure-triage/s07-07-checkpoint","xp":8,"hint":[{"level":1,"body":"Structure it as gate, then classify, then route. The classify\nblock is `classify_failure` from lesson 02's checkpoint, verbatim\n— same four branches, same priority order. The only new parts are\nthe repro gate up front and the `ROUTES` lookup at the end.\n","cost":0},{"level":2,"body":"```python\ndef triage(report):\n if not report[\"reproduced_with_logged_input\"]:\n return \"blocked: reproduce with the logged input\"\n trace = report[\"trace\"]\n if trace[\"retrieved_chunks\"] and not trace[\"retrieved_chunks_match_query\"]:\n cls = \"retrieval\"\n elif trace[\"raw_output\"] != trace[\"output_after_postprocess\"]:\n cls = \"parse\"\n elif not trace[\"retrieved_chunks\"]:\n cls = \"hallucination\"\n else:\n cls = \"prompt\"\n return f\"route to {ROUTES[cls]}\"\n```\n","cost":0}],"personalize":false,"phase":"check","estSeconds":240,"concept":"triage-end-to-end","prompt":"Final drill: the whole runbook in one function. Three tickets\nfrom this morning, each with a trace and a repro flag. Write\n`triage(report)`:\n\n1. **Gate.** If `report[\"reproduced_with_logged_input\"]` is\n falsy, return `\"blocked: reproduce with the logged input\"` —\n no classification, no routing.\n2. **Classify.** Apply the lesson-02 priority order to\n `report[\"trace\"]`:\n - `retrieved_chunks` non-empty AND\n `retrieved_chunks_match_query` is `False` → `\"retrieval\"`\n - `raw_output != output_after_postprocess` → `\"parse\"`\n - `retrieved_chunks` empty → `\"hallucination\"`\n - otherwise → `\"prompt\"`\n3. **Route.** Return `\"route to \"` via `ROUTES`.\n\nPrint one line per ticket as `ticket_: `.\n\nExpected output:\n```\nticket_9101: blocked: reproduce with the logged input\nticket_9102: route to retrieval pipeline\nticket_9103: route to post-processing code\n```\n","starter":"$36","grader":{"kind":"stdout-equality","expected":"ticket_9101: blocked: reproduce with the logged input\nticket_9102: route to retrieval pipeline\nticket_9103: route to post-processing code\n","normalize":"collapse-trailing-newline"},"solution":"$37","passThreshold":0.66}],"xpTotal":26}],"liveLessonCount":4,"liveStepCount":34,"xpTotal":112}},"chapter":"$16:1:props:tree:detail","lesson":"$16:1:props:tree:detail:lessons:3","step":"$16:1:props:tree:detail:lessons:3:steps:1","stepIndex":1,"next":{"chapterSlug":"debugging-output","lessonSlug":"prediction-failure-triage","stepIndex":2}}]]

Triage the bad prediction — reproduce with the logged input, then route by class — step 2 of 7