analytics studio

The number is plausible. That's the problem.

Your memo says churn improved 1.8 points in July. The CFO liked it so much it's now slide four of the board deck — and this morning she wants to know whether that's before or after the billing-migration backfill. The AI drafted the pull, the notebook has since been renamed final_v2_REAL.ipynb, and when you rerun it just now, it says 1.6.

Nothing crashed. No traceback, no red cell. Every step looked reasonable when it ran. That's the signature failure of AI-assisted analysis: not wrong numbers that look wrong — you'd catch those — but wrong numbers that look exactly like right ones. Plausible-but-wrong aggregations. Nulls handled silently, whichever way the model felt like. Joins that quietly double a revenue line. A confident narrative wrapped around a result nobody validated. You are the person whose name is on the memo, so you are the person this chapter arms.

The pipeline here is the one you already run — messy export in, memo out — with five habits bolted on where the bodies are buried: a cleaning log that accounts for every dropped row, a sample-first check on every AI-drafted aggregation, a trace from every number in the writeup back to the result that produced it, a scored challenge of the AI's recommendation before it ships, and a rerun test at the end. All of it as Python you execute, not advice you nod at.

Why it matters

• Data work is where AI use actually is. Anthropic's Economic Index puts computer and mathematical tasks as the single largest occupational category of observed Claude use — roughly a third of Claude.ai conversations and nearly half of API traffic (exact figures move between index editions; the direction doesn't). Secondary analyses of Eloundou et al.'s "GPTs are GPTs" report mathematicians as the single 100%-exposed occupation. Your task list is the bullseye.
• And yet the analysts aren't scared — they're re-arming. Alteryx's 2025 survey (vendor survey, so treat direction, not precision): 7 in 10 analysts say AI makes them more effective, and only 17% are deeply concerned about displacement — down from 65% a year earlier. The reconciliation is simple: the drafting moved to the machine, the accountability didn't move an inch.
• Nobody teaches the checking. Courses teach you to prompt a copilot into writing SQL faster; almost nothing on the market grades whether you can tell the SQL is wrong. And the tool itself stopped being a differentiator: ODSC's 2025 practitioner survey has 90% using conversational AI platforms and 81% working with agents, LLMs, or RAG. When everyone drafts with the same machine, verification is the skill that separates analysts — so verification is what gets graded here.

What AI specifically gets wrong

• It picks an aggregation for you, plausibly. "Average conversion rate" — average of the segment rates, or pooled signups over pooled visits? Those differ by double digits on real traffic mixes, both fit the phrase, and the drafter commits to one without telling you it chose.
• It handles nulls silently. dropna() is one token. It's also an undocumented decision about every column at once — and the three customers it deleted were the churn story.
• Its joins fan out. One order, two payment rows, and your revenue is suddenly 20% up. The number is bigger, the query ran clean, and the memo writes itself. Row counts before and after the join catch it in one line; nothing else does.
• It narrates unvalidated results with total confidence. And when you push back, it apologizes and agrees with you — which means asking "are you sure?" is not a check. Recomputation is a check.

What you'll be able to do at the end

• Clean an export with a log where rows_in − rows_out is exactly accounted for by named steps with named reasons — and a per-column null ledger instead of a silent dropna().
• Run every AI-drafted aggregation on a hand-checkable sample first: expectation written down before the query runs, full run only after they match.
• Audit a memo the way a hostile CFO would: every number carries a ref, every ref resolves to a result, every mismatch gets flagged before someone else flags it.
• Challenge the AI's recommendation with banking's SR 11-7 validation triad — conceptual soundness, ongoing monitoring, outcomes analysis — as a scored record with a non-author challenger, not a vibe.
• Prove reproducibility mechanically: build the memo twice from the same data and require the same bytes. Rerun drift means a mutation is hiding somewhere, and you'll know how to find it.

No warehouse login required — every drill is plain Python on fixture data, which is also the rule at work: row-level customer data never goes into a prompt. The failure stories are pandas-shaped on purpose (dropna(), join fan-out); chapter 35 is where you build these same receipts on real DataFrames. What you leave with isn't a badge. It's a memo you can defend line by line while the CFO watches, and the habit of never shipping one you can't.