Three real breakages — what happens when nobody mans the border
The schema-at-the-boundary pattern sounds abstract until you see the bills. Below are three illustrative breakages — composites built from patterns that recur across production AI postmortems in the last 18 months. Specific company names, dollar figures, and incident dates are representative, not literal — the shape of the failure (and the fix) is what to internalize. Treat these the way you'd treat case studies in a textbook, not direct quotations.
Breakage 1: The receipt extractor that swapped fields
A small-business expense app uses Claude to extract vendor and amount from a photographed receipt. Schema (in the developer's head, not in code):
{
"vendor": "string",
"amount": "float (USD)",
}
The team prompted the model carefully: Return only valid JSON with
vendor and amount. In dev, on 50 test receipts, it always worked.
In prod, on a Saturday morning, the model started returning:
{"vendor": "12.99", "amount": "Whole Foods Market"}
The fields were swapped. The downstream code took whatever was in
amount, parsed it as a float (failed silently on the string), fell
through to a try/except, and charged the customer's stored card
the value in vendor — except that value was now the vendor name,
which the float() had thrown a ValueError on, which the catch-all
handler interpreted as "use the last successful amount."
Customers got charged the wrong amount. Some got charged twice. The support queue exploded. Two engineers spent a weekend rolling back.
What would have stopped it: a schema-aware extraction mode plus a
Pydantic validator at the boundary. The moment the model returned
vendor as a numeric-looking string and amount as text, the
validator would have raised ValidationError and the charge would
never have been attempted.
Cost: ~$40k in refunds and ~$15k in support hours. Root cause in the postmortem: "No validation between model response and billing code." Translation: no customs officer at the boundary.
Breakage 2: The router that hallucinated an enum
A team built an agent to route inbound requests to the right team. The
agent classified each request with a priority
field. The intended values were low, medium, high.
Alert rules: anything with priority="high" woke the on-call
owner. The team did not define critical as a valid value. They
just didn't page on it, because it didn't exist.
The model, in production, started occasionally returning
priority="critical" on requests that were unusually
urgent. Downstream code did a string-equality check against "high",
correctly skipped the page — but the ops dashboard
counted critical as high for the urgent bucket, marking thousands
of requests as breached and firing automated notices
to users whose requests
were actually fine.
What would have stopped it: a Pydantic Literal["low", "medium", "high"]
or an Enum. The model would not have been able to return critical
under a schema-aware mode; if it had under a non-enforced mode, the
validator would have rejected the response on arrival.
Cost: ~120 wrongly-issued notices, ~30 angry follow-ups. Root cause: "Free-string priority field, no enum validation at the API boundary."
Breakage 3: The eval harness that lost ground truth
An ML team built an eval harness to score Claude's answers to
customer-service questions. The expected-answer field was typed
loosely: expected: str. They scored the model's output with a
string-equality check against expected.
The expected answers came from a CSV. A junior engineer accidentally saved one row with the answer wrapped in a list:
expected
"yes"
"no"
["yes"]
The CSV parser loaded ["yes"] as the literal string '["yes"]'.
The model returned "yes". String equality returned False. The
eval scored that case as a regression — even though the model was
right. A week of "the model got worse" debugging followed. The team
rolled back a perfectly fine prompt change to fix a problem that
wasn't there.
What would have stopped it: a schema on the eval CSV itself. The
expected-answer field, defined as str with a Pydantic validator,
would have rejected the ["yes"] row on ingest. The harness would
have refused to run with bad ground truth.
Cost: 6 engineer-days of false-alarm debugging, one good prompt change reverted for no reason. Root cause: "No schema validation on eval inputs."
The pattern across all three
| Breakage | Trust boundary that wasn't guarded |
|---|---|
| Receipt swap | Model output → billing logic |
| Enum classifier | Model output → dashboard urgency logic |
| Eval CSV | File input → eval scoring logic |
Every one of these would have been caught — in seconds, on the first bad input — if the boundary had a schema and a validator on it. In all three cases, the team had Pydantic in their codebase. They just didn't put it where the foreign data came in.
That's the lesson worth memorizing. The presence of Pydantic in
requirements.txt doesn't protect you. A schema at every boundary
where untrusted data enters is what protects you.
Next: a quick check that you can distinguish a boundary failure from a logic failure.
Three real breakages — what happens when nobody mans the border
The schema-at-the-boundary pattern sounds abstract until you see the bills. Below are three illustrative breakages — composites built from patterns that recur across production AI postmortems in the last 18 months. Specific company names, dollar figures, and incident dates are representative, not literal — the shape of the failure (and the fix) is what to internalize. Treat these the way you'd treat case studies in a textbook, not direct quotations.
Breakage 1: The receipt extractor that swapped fields
A small-business expense app uses Claude to extract vendor and amount from a photographed receipt. Schema (in the developer's head, not in code):
{
"vendor": "string",
"amount": "float (USD)",
}
The team prompted the model carefully: Return only valid JSON with
vendor and amount. In dev, on 50 test receipts, it always worked.
In prod, on a Saturday morning, the model started returning:
{"vendor": "12.99", "amount": "Whole Foods Market"}
The fields were swapped. The downstream code took whatever was in
amount, parsed it as a float (failed silently on the string), fell
through to a try/except, and charged the customer's stored card
the value in vendor — except that value was now the vendor name,
which the float() had thrown a ValueError on, which the catch-all
handler interpreted as "use the last successful amount."
Customers got charged the wrong amount. Some got charged twice. The support queue exploded. Two engineers spent a weekend rolling back.
What would have stopped it: a schema-aware extraction mode plus a
Pydantic validator at the boundary. The moment the model returned
vendor as a numeric-looking string and amount as text, the
validator would have raised ValidationError and the charge would
never have been attempted.
Cost: ~$40k in refunds and ~$15k in support hours. Root cause in the postmortem: "No validation between model response and billing code." Translation: no customs officer at the boundary.
Breakage 2: The router that hallucinated an enum
A team built an agent to route inbound incident notes to the right on-call engineer. The
agent classified each note with a priority
field. The intended values were low, medium, high.
Alert rules: anything with priority="high" woke the on-call
owner. The team did not define critical as a valid value. They
just didn't page on it, because it didn't exist.
The model, in production, started occasionally returning
priority="critical" on incident notes that were unusually
urgent. Downstream code did a string-equality check against "high",
correctly skipped the page — but the incident dashboard
counted critical as high for the urgent bucket, marking thousands
of incident notes as breached and firing automated notices
to on-call engineers whose incident notes
were actually fine.
What would have stopped it: a Pydantic Literal["low", "medium", "high"]
or an Enum. The model would not have been able to return critical
under a schema-aware mode; if it had under a non-enforced mode, the
validator would have rejected the response on arrival.
Cost: ~120 wrongly-issued notices, ~30 angry follow-ups. Root cause: "Free-string priority field, no enum validation at the API boundary."
Breakage 3: The eval harness that lost ground truth
An ML team built an eval harness to score Claude's answers to
customer-service questions. The expected-answer field was typed
loosely: expected: str. They scored the model's output with a
string-equality check against expected.
The expected answers came from a CSV. A junior engineer accidentally saved one row with the answer wrapped in a list:
expected
"yes"
"no"
["yes"]
The CSV parser loaded ["yes"] as the literal string '["yes"]'.
The model returned "yes". String equality returned False. The
eval scored that case as a regression — even though the model was
right. A week of "the model got worse" debugging followed. The team
rolled back a perfectly fine prompt change to fix a problem that
wasn't there.
What would have stopped it: a schema on the eval CSV itself. The
expected-answer field, defined as str with a Pydantic validator,
would have rejected the ["yes"] row on ingest. The harness would
have refused to run with bad ground truth.
Cost: 6 engineer-days of false-alarm debugging, one good prompt change reverted for no reason. Root cause: "No schema validation on eval inputs."
The pattern across all three
| Breakage | Trust boundary that wasn't guarded |
|---|---|
| Receipt swap | Model output → billing logic |
| Enum classifier | Model output → dashboard urgency logic |
| Eval CSV | File input → eval scoring logic |
Every one of these would have been caught — in seconds, on the first bad input — if the boundary had a schema and a validator on it. In all three cases, the team had Pydantic in their codebase. They just didn't put it where the foreign data came in.
That's the lesson worth memorizing. The presence of Pydantic in
requirements.txt doesn't protect you. A schema at every boundary
where untrusted data enters is what protects you.
Next: a quick check that you can distinguish a boundary failure from a logic failure.
Three real breakages — what happens when nobody mans the border
The schema-at-the-boundary pattern sounds abstract until you see the bills. Below are three illustrative breakages — composites built from patterns that recur across production AI postmortems in the last 18 months. Specific company names, dollar figures, and incident dates are representative, not literal — the shape of the failure (and the fix) is what to internalize. Treat these the way you'd treat case studies in a textbook, not direct quotations.
Breakage 1: The receipt extractor that swapped fields
A small-business expense app uses Claude to extract vendor and amount from a photographed receipt. Schema (in the developer's head, not in code):
{
"vendor": "string",
"amount": "float (USD)",
}
The team prompted the model carefully: Return only valid JSON with
vendor and amount. In dev, on 50 test receipts, it always worked.
In prod, on a Saturday morning, the model started returning:
{"vendor": "12.99", "amount": "Whole Foods Market"}
The fields were swapped. The downstream code took whatever was in
amount, parsed it as a float (failed silently on the string), fell
through to a try/except, and charged the customer's stored card
the value in vendor — except that value was now the vendor name,
which the float() had thrown a ValueError on, which the catch-all
handler interpreted as "use the last successful amount."
Customers got charged the wrong amount. Some got charged twice. The support queue exploded. Two engineers spent a weekend rolling back.
What would have stopped it: a schema-aware extraction mode plus a
Pydantic validator at the boundary. The moment the model returned
vendor as a numeric-looking string and amount as text, the
validator would have raised ValidationError and the charge would
never have been attempted.
Cost: ~$40k in refunds and ~$15k in support hours. Root cause in the postmortem: "No validation between model response and billing code." Translation: no customs officer at the boundary.
Breakage 2: The router that hallucinated an enum
A team built an agent to route inbound campaign assets to the right channel owner. The
agent classified each asset with a priority
field. The intended values were low, medium, high.
Alert rules: anything with priority="high" woke the on-call
owner. The team did not define critical as a valid value. They
just didn't page on it, because it didn't exist.
The model, in production, started occasionally returning
priority="critical" on campaign assets that were unusually
urgent. Downstream code did a string-equality check against "high",
correctly skipped the page — but the ship-list dashboard
counted critical as high for the urgent bucket, marking thousands
of campaign assets as breached and firing automated notices
to channel owners whose campaign assets
were actually fine.
What would have stopped it: a Pydantic Literal["low", "medium", "high"]
or an Enum. The model would not have been able to return critical
under a schema-aware mode; if it had under a non-enforced mode, the
validator would have rejected the response on arrival.
Cost: ~120 wrongly-issued notices, ~30 angry follow-ups. Root cause: "Free-string priority field, no enum validation at the API boundary."
Breakage 3: The eval harness that lost ground truth
An ML team built an eval harness to score Claude's answers to
customer-service questions. The expected-answer field was typed
loosely: expected: str. They scored the model's output with a
string-equality check against expected.
The expected answers came from a CSV. A junior engineer accidentally saved one row with the answer wrapped in a list:
expected
"yes"
"no"
["yes"]
The CSV parser loaded ["yes"] as the literal string '["yes"]'.
The model returned "yes". String equality returned False. The
eval scored that case as a regression — even though the model was
right. A week of "the model got worse" debugging followed. The team
rolled back a perfectly fine prompt change to fix a problem that
wasn't there.
What would have stopped it: a schema on the eval CSV itself. The
expected-answer field, defined as str with a Pydantic validator,
would have rejected the ["yes"] row on ingest. The harness would
have refused to run with bad ground truth.
Cost: 6 engineer-days of false-alarm debugging, one good prompt change reverted for no reason. Root cause: "No schema validation on eval inputs."
The pattern across all three
| Breakage | Trust boundary that wasn't guarded |
|---|---|
| Receipt swap | Model output → billing logic |
| Enum classifier | Model output → dashboard urgency logic |
| Eval CSV | File input → eval scoring logic |
Every one of these would have been caught — in seconds, on the first bad input — if the boundary had a schema and a validator on it. In all three cases, the team had Pydantic in their codebase. They just didn't put it where the foreign data came in.
That's the lesson worth memorizing. The presence of Pydantic in
requirements.txt doesn't protect you. A schema at every boundary
where untrusted data enters is what protects you.
Next: a quick check that you can distinguish a boundary failure from a logic failure.
Three real breakages — what happens when nobody mans the border
The schema-at-the-boundary pattern sounds abstract until you see the bills. Below are three illustrative breakages — composites built from patterns that recur across production AI postmortems in the last 18 months. Specific company names, dollar figures, and incident dates are representative, not literal — the shape of the failure (and the fix) is what to internalize. Treat these the way you'd treat case studies in a textbook, not direct quotations.
Breakage 1: The receipt extractor that swapped fields
A small-business expense app uses Claude to extract vendor and amount from a photographed receipt. Schema (in the developer's head, not in code):
{
"vendor": "string",
"amount": "float (USD)",
}
The team prompted the model carefully: Return only valid JSON with
vendor and amount. In dev, on 50 test receipts, it always worked.
In prod, on a Saturday morning, the model started returning:
{"vendor": "12.99", "amount": "Whole Foods Market"}
The fields were swapped. The downstream code took whatever was in
amount, parsed it as a float (failed silently on the string), fell
through to a try/except, and charged the customer's stored card
the value in vendor — except that value was now the vendor name,
which the float() had thrown a ValueError on, which the catch-all
handler interpreted as "use the last successful amount."
Customers got charged the wrong amount. Some got charged twice. The support queue exploded. Two engineers spent a weekend rolling back.
What would have stopped it: a schema-aware extraction mode plus a
Pydantic validator at the boundary. The moment the model returned
vendor as a numeric-looking string and amount as text, the
validator would have raised ValidationError and the charge would
never have been attempted.
Cost: ~$40k in refunds and ~$15k in support hours. Root cause in the postmortem: "No validation between model response and billing code." Translation: no customs officer at the boundary.
Breakage 2: The router that hallucinated an enum
A team built an agent to route inbound brand-review notes to the right reviewer. The
agent classified each note with a priority
field. The intended values were low, medium, high.
Alert rules: anything with priority="high" woke the on-call
owner. The team did not define critical as a valid value. They
just didn't page on it, because it didn't exist.
The model, in production, started occasionally returning
priority="critical" on brand-review notes that were unusually
urgent. Downstream code did a string-equality check against "high",
correctly skipped the page — but the kit-review dashboard
counted critical as high for the urgent bucket, marking thousands
of brand-review notes as breached and firing automated notices
to reviewers whose brand-review notes
were actually fine.
What would have stopped it: a Pydantic Literal["low", "medium", "high"]
or an Enum. The model would not have been able to return critical
under a schema-aware mode; if it had under a non-enforced mode, the
validator would have rejected the response on arrival.
Cost: ~120 wrongly-issued notices, ~30 angry follow-ups. Root cause: "Free-string priority field, no enum validation at the API boundary."
Breakage 3: The eval harness that lost ground truth
An ML team built an eval harness to score Claude's answers to
customer-service questions. The expected-answer field was typed
loosely: expected: str. They scored the model's output with a
string-equality check against expected.
The expected answers came from a CSV. A junior engineer accidentally saved one row with the answer wrapped in a list:
expected
"yes"
"no"
["yes"]
The CSV parser loaded ["yes"] as the literal string '["yes"]'.
The model returned "yes". String equality returned False. The
eval scored that case as a regression — even though the model was
right. A week of "the model got worse" debugging followed. The team
rolled back a perfectly fine prompt change to fix a problem that
wasn't there.
What would have stopped it: a schema on the eval CSV itself. The
expected-answer field, defined as str with a Pydantic validator,
would have rejected the ["yes"] row on ingest. The harness would
have refused to run with bad ground truth.
Cost: 6 engineer-days of false-alarm debugging, one good prompt change reverted for no reason. Root cause: "No schema validation on eval inputs."
The pattern across all three
| Breakage | Trust boundary that wasn't guarded |
|---|---|
| Receipt swap | Model output → billing logic |
| Enum classifier | Model output → dashboard urgency logic |
| Eval CSV | File input → eval scoring logic |
Every one of these would have been caught — in seconds, on the first bad input — if the boundary had a schema and a validator on it. In all three cases, the team had Pydantic in their codebase. They just didn't put it where the foreign data came in.
That's the lesson worth memorizing. The presence of Pydantic in
requirements.txt doesn't protect you. A schema at every boundary
where untrusted data enters is what protects you.
Next: a quick check that you can distinguish a boundary failure from a logic failure.
Three real breakages — what happens when nobody mans the border
The schema-at-the-boundary pattern sounds abstract until you see the bills. Below are three illustrative breakages — composites built from patterns that recur across production AI postmortems in the last 18 months. Specific company names, dollar figures, and incident dates are representative, not literal — the shape of the failure (and the fix) is what to internalize. Treat these the way you'd treat case studies in a textbook, not direct quotations.
Breakage 1: The receipt extractor that swapped fields
A small-business expense app uses Claude to extract vendor and amount from a photographed receipt. Schema (in the developer's head, not in code):
{
"vendor": "string",
"amount": "float (USD)",
}
The team prompted the model carefully: Return only valid JSON with
vendor and amount. In dev, on 50 test receipts, it always worked.
In prod, on a Saturday morning, the model started returning:
{"vendor": "12.99", "amount": "Whole Foods Market"}
The fields were swapped. The downstream code took whatever was in
amount, parsed it as a float (failed silently on the string), fell
through to a try/except, and charged the customer's stored card
the value in vendor — except that value was now the vendor name,
which the float() had thrown a ValueError on, which the catch-all
handler interpreted as "use the last successful amount."
Customers got charged the wrong amount. Some got charged twice. The support queue exploded. Two engineers spent a weekend rolling back.
What would have stopped it: a schema-aware extraction mode plus a
Pydantic validator at the boundary. The moment the model returned
vendor as a numeric-looking string and amount as text, the
validator would have raised ValidationError and the charge would
never have been attempted.
Cost: ~$40k in refunds and ~$15k in support hours. Root cause in the postmortem: "No validation between model response and billing code." Translation: no customs officer at the boundary.
Breakage 2: The router that hallucinated an enum
A team built an agent to route inbound support tickets to the right team. The
agent classified each ticket with a priority
field. The intended values were low, medium, high.
Alert rules: anything with priority="high" woke the on-call
owner. The team did not define critical as a valid value. They
just didn't page on it, because it didn't exist.
The model, in production, started occasionally returning
priority="critical" on tickets that were unusually
urgent. Downstream code did a string-equality check against "high",
correctly skipped the page — but the support dashboard
counted critical as high for the urgent bucket, marking thousands
of tickets as breached and firing automated notices
to customers whose tickets
were actually fine.
What would have stopped it: a Pydantic Literal["low", "medium", "high"]
or an Enum. The model would not have been able to return critical
under a schema-aware mode; if it had under a non-enforced mode, the
validator would have rejected the response on arrival.
Cost: ~120 wrongly-issued notices, ~30 angry follow-ups. Root cause: "Free-string priority field, no enum validation at the API boundary."
Breakage 3: The eval harness that lost ground truth
An ML team built an eval harness to score Claude's answers to
customer-service questions. The expected-answer field was typed
loosely: expected: str. They scored the model's output with a
string-equality check against expected.
The expected answers came from a CSV. A junior engineer accidentally saved one row with the answer wrapped in a list:
expected
"yes"
"no"
["yes"]
The CSV parser loaded ["yes"] as the literal string '["yes"]'.
The model returned "yes". String equality returned False. The
eval scored that case as a regression — even though the model was
right. A week of "the model got worse" debugging followed. The team
rolled back a perfectly fine prompt change to fix a problem that
wasn't there.
What would have stopped it: a schema on the eval CSV itself. The
expected-answer field, defined as str with a Pydantic validator,
would have rejected the ["yes"] row on ingest. The harness would
have refused to run with bad ground truth.
Cost: 6 engineer-days of false-alarm debugging, one good prompt change reverted for no reason. Root cause: "No schema validation on eval inputs."
The pattern across all three
| Breakage | Trust boundary that wasn't guarded |
|---|---|
| Receipt swap | Model output → billing logic |
| Enum classifier | Model output → dashboard urgency logic |
| Eval CSV | File input → eval scoring logic |
Every one of these would have been caught — in seconds, on the first bad input — if the boundary had a schema and a validator on it. In all three cases, the team had Pydantic in their codebase. They just didn't put it where the foreign data came in.
That's the lesson worth memorizing. The presence of Pydantic in
requirements.txt doesn't protect you. A schema at every boundary
where untrusted data enters is what protects you.
Next: a quick check that you can distinguish a boundary failure from a logic failure.
Three real breakages — what happens when nobody mans the border
The schema-at-the-boundary pattern sounds abstract until you see the bills. Below are three illustrative breakages — composites built from patterns that recur across production AI postmortems in the last 18 months. Specific company names, dollar figures, and incident dates are representative, not literal — the shape of the failure (and the fix) is what to internalize. Treat these the way you'd treat case studies in a textbook, not direct quotations.
Breakage 1: The receipt extractor that swapped fields
A small-business expense app uses Claude to extract vendor and amount from a photographed receipt. Schema (in the developer's head, not in code):
{
"vendor": "string",
"amount": "float (USD)",
}
The team prompted the model carefully: Return only valid JSON with
vendor and amount. In dev, on 50 test receipts, it always worked.
In prod, on a Saturday morning, the model started returning:
{"vendor": "12.99", "amount": "Whole Foods Market"}
The fields were swapped. The downstream code took whatever was in
amount, parsed it as a float (failed silently on the string), fell
through to a try/except, and charged the customer's stored card
the value in vendor — except that value was now the vendor name,
which the float() had thrown a ValueError on, which the catch-all
handler interpreted as "use the last successful amount."
Customers got charged the wrong amount. Some got charged twice. The support queue exploded. Two engineers spent a weekend rolling back.
What would have stopped it: a schema-aware extraction mode plus a
Pydantic validator at the boundary. The moment the model returned
vendor as a numeric-looking string and amount as text, the
validator would have raised ValidationError and the charge would
never have been attempted.
Cost: ~$40k in refunds and ~$15k in support hours. Root cause in the postmortem: "No validation between model response and billing code." Translation: no customs officer at the boundary.
Breakage 2: The router that hallucinated an enum
A team built an agent to route inbound claim lines to the right editor. The
agent classified each claim with a priority
field. The intended values were low, medium, high.
Alert rules: anything with priority="high" woke the on-call
owner. The team did not define critical as a valid value. They
just didn't page on it, because it didn't exist.
The model, in production, started occasionally returning
priority="critical" on claim lines that were unusually
urgent. Downstream code did a string-equality check against "high",
correctly skipped the page — but the clearance dashboard
counted critical as high for the urgent bucket, marking thousands
of claim lines as breached and firing automated notices
to editors whose claim lines
were actually fine.
What would have stopped it: a Pydantic Literal["low", "medium", "high"]
or an Enum. The model would not have been able to return critical
under a schema-aware mode; if it had under a non-enforced mode, the
validator would have rejected the response on arrival.
Cost: ~120 wrongly-issued notices, ~30 angry follow-ups. Root cause: "Free-string priority field, no enum validation at the API boundary."
Breakage 3: The eval harness that lost ground truth
An ML team built an eval harness to score Claude's answers to
customer-service questions. The expected-answer field was typed
loosely: expected: str. They scored the model's output with a
string-equality check against expected.
The expected answers came from a CSV. A junior engineer accidentally saved one row with the answer wrapped in a list:
expected
"yes"
"no"
["yes"]
The CSV parser loaded ["yes"] as the literal string '["yes"]'.
The model returned "yes". String equality returned False. The
eval scored that case as a regression — even though the model was
right. A week of "the model got worse" debugging followed. The team
rolled back a perfectly fine prompt change to fix a problem that
wasn't there.
What would have stopped it: a schema on the eval CSV itself. The
expected-answer field, defined as str with a Pydantic validator,
would have rejected the ["yes"] row on ingest. The harness would
have refused to run with bad ground truth.
Cost: 6 engineer-days of false-alarm debugging, one good prompt change reverted for no reason. Root cause: "No schema validation on eval inputs."
The pattern across all three
| Breakage | Trust boundary that wasn't guarded |
|---|---|
| Receipt swap | Model output → billing logic |
| Enum classifier | Model output → dashboard urgency logic |
| Eval CSV | File input → eval scoring logic |
Every one of these would have been caught — in seconds, on the first bad input — if the boundary had a schema and a validator on it. In all three cases, the team had Pydantic in their codebase. They just didn't put it where the foreign data came in.
That's the lesson worth memorizing. The presence of Pydantic in
requirements.txt doesn't protect you. A schema at every boundary
where untrusted data enters is what protects you.
Next: a quick check that you can distinguish a boundary failure from a logic failure.
Three real breakages — what happens when nobody mans the border
The schema-at-the-boundary pattern sounds abstract until you see the bills. Below are three illustrative breakages — composites built from patterns that recur across production AI postmortems in the last 18 months. Specific company names, dollar figures, and incident dates are representative, not literal — the shape of the failure (and the fix) is what to internalize. Treat these the way you'd treat case studies in a textbook, not direct quotations.
Breakage 1: The receipt extractor that swapped fields
A small-business expense app uses Claude to extract vendor and amount from a photographed receipt. Schema (in the developer's head, not in code):
{
"vendor": "string",
"amount": "float (USD)",
}
The team prompted the model carefully: Return only valid JSON with
vendor and amount. In dev, on 50 test receipts, it always worked.
In prod, on a Saturday morning, the model started returning:
{"vendor": "12.99", "amount": "Whole Foods Market"}
The fields were swapped. The downstream code took whatever was in
amount, parsed it as a float (failed silently on the string), fell
through to a try/except, and charged the customer's stored card
the value in vendor — except that value was now the vendor name,
which the float() had thrown a ValueError on, which the catch-all
handler interpreted as "use the last successful amount."
Customers got charged the wrong amount. Some got charged twice. The support queue exploded. Two engineers spent a weekend rolling back.
What would have stopped it: a schema-aware extraction mode plus a
Pydantic validator at the boundary. The moment the model returned
vendor as a numeric-looking string and amount as text, the
validator would have raised ValidationError and the charge would
never have been attempted.
Cost: ~$40k in refunds and ~$15k in support hours. Root cause in the postmortem: "No validation between model response and billing code." Translation: no customs officer at the boundary.
Breakage 2: The router that hallucinated an enum
A team built an agent to route inbound research cuts to the right decision-maker. The
agent classified each cut with a priority
field. The intended values were low, medium, high.
Alert rules: anything with priority="high" woke the on-call
owner. The team did not define critical as a valid value. They
just didn't page on it, because it didn't exist.
The model, in production, started occasionally returning
priority="critical" on research cuts that were unusually
urgent. Downstream code did a string-equality check against "high",
correctly skipped the page — but the findings dashboard
counted critical as high for the urgent bucket, marking thousands
of research cuts as breached and firing automated notices
to decision-makers whose research cuts
were actually fine.
What would have stopped it: a Pydantic Literal["low", "medium", "high"]
or an Enum. The model would not have been able to return critical
under a schema-aware mode; if it had under a non-enforced mode, the
validator would have rejected the response on arrival.
Cost: ~120 wrongly-issued notices, ~30 angry follow-ups. Root cause: "Free-string priority field, no enum validation at the API boundary."
Breakage 3: The eval harness that lost ground truth
An ML team built an eval harness to score Claude's answers to
customer-service questions. The expected-answer field was typed
loosely: expected: str. They scored the model's output with a
string-equality check against expected.
The expected answers came from a CSV. A junior engineer accidentally saved one row with the answer wrapped in a list:
expected
"yes"
"no"
["yes"]
The CSV parser loaded ["yes"] as the literal string '["yes"]'.
The model returned "yes". String equality returned False. The
eval scored that case as a regression — even though the model was
right. A week of "the model got worse" debugging followed. The team
rolled back a perfectly fine prompt change to fix a problem that
wasn't there.
What would have stopped it: a schema on the eval CSV itself. The
expected-answer field, defined as str with a Pydantic validator,
would have rejected the ["yes"] row on ingest. The harness would
have refused to run with bad ground truth.
Cost: 6 engineer-days of false-alarm debugging, one good prompt change reverted for no reason. Root cause: "No schema validation on eval inputs."
The pattern across all three
| Breakage | Trust boundary that wasn't guarded |
|---|---|
| Receipt swap | Model output → billing logic |
| Enum classifier | Model output → dashboard urgency logic |
| Eval CSV | File input → eval scoring logic |
Every one of these would have been caught — in seconds, on the first bad input — if the boundary had a schema and a validator on it. In all three cases, the team had Pydantic in their codebase. They just didn't put it where the foreign data came in.
That's the lesson worth memorizing. The presence of Pydantic in
requirements.txt doesn't protect you. A schema at every boundary
where untrusted data enters is what protects you.
Next: a quick check that you can distinguish a boundary failure from a logic failure.
Three real breakages — what happens when nobody mans the border
The schema-at-the-boundary pattern sounds abstract until you see the bills. Below are three illustrative breakages — composites built from patterns that recur across production AI postmortems in the last 18 months. Specific company names, dollar figures, and incident dates are representative, not literal — the shape of the failure (and the fix) is what to internalize. Treat these the way you'd treat case studies in a textbook, not direct quotations.
Breakage 1: The receipt extractor that swapped fields
A small-business expense app uses Claude to extract vendor and amount from a photographed receipt. Schema (in the developer's head, not in code):
{
"vendor": "string",
"amount": "float (USD)",
}
The team prompted the model carefully: Return only valid JSON with
vendor and amount. In dev, on 50 test receipts, it always worked.
In prod, on a Saturday morning, the model started returning:
{"vendor": "12.99", "amount": "Whole Foods Market"}
The fields were swapped. The downstream code took whatever was in
amount, parsed it as a float (failed silently on the string), fell
through to a try/except, and charged the customer's stored card
the value in vendor — except that value was now the vendor name,
which the float() had thrown a ValueError on, which the catch-all
handler interpreted as "use the last successful amount."
Customers got charged the wrong amount. Some got charged twice. The support queue exploded. Two engineers spent a weekend rolling back.
What would have stopped it: a schema-aware extraction mode plus a
Pydantic validator at the boundary. The moment the model returned
vendor as a numeric-looking string and amount as text, the
validator would have raised ValidationError and the charge would
never have been attempted.
Cost: ~$40k in refunds and ~$15k in support hours. Root cause in the postmortem: "No validation between model response and billing code." Translation: no customs officer at the boundary.
Breakage 2: The router that hallucinated an enum
A team built an agent to route inbound risk notes to the right delivery lead. The
agent classified each note with a priority
field. The intended values were low, medium, high.
Alert rules: anything with priority="high" woke the on-call
owner. The team did not define critical as a valid value. They
just didn't page on it, because it didn't exist.
The model, in production, started occasionally returning
priority="critical" on risk notes that were unusually
urgent. Downstream code did a string-equality check against "high",
correctly skipped the page — but the delivery dashboard
counted critical as high for the urgent bucket, marking thousands
of risk notes as breached and firing automated notices
to leads whose risk notes
were actually fine.
What would have stopped it: a Pydantic Literal["low", "medium", "high"]
or an Enum. The model would not have been able to return critical
under a schema-aware mode; if it had under a non-enforced mode, the
validator would have rejected the response on arrival.
Cost: ~120 wrongly-issued notices, ~30 angry follow-ups. Root cause: "Free-string priority field, no enum validation at the API boundary."
Breakage 3: The eval harness that lost ground truth
An ML team built an eval harness to score Claude's answers to
customer-service questions. The expected-answer field was typed
loosely: expected: str. They scored the model's output with a
string-equality check against expected.
The expected answers came from a CSV. A junior engineer accidentally saved one row with the answer wrapped in a list:
expected
"yes"
"no"
["yes"]
The CSV parser loaded ["yes"] as the literal string '["yes"]'.
The model returned "yes". String equality returned False. The
eval scored that case as a regression — even though the model was
right. A week of "the model got worse" debugging followed. The team
rolled back a perfectly fine prompt change to fix a problem that
wasn't there.
What would have stopped it: a schema on the eval CSV itself. The
expected-answer field, defined as str with a Pydantic validator,
would have rejected the ["yes"] row on ingest. The harness would
have refused to run with bad ground truth.
Cost: 6 engineer-days of false-alarm debugging, one good prompt change reverted for no reason. Root cause: "No schema validation on eval inputs."
The pattern across all three
| Breakage | Trust boundary that wasn't guarded |
|---|---|
| Receipt swap | Model output → billing logic |
| Enum classifier | Model output → dashboard urgency logic |
| Eval CSV | File input → eval scoring logic |
Every one of these would have been caught — in seconds, on the first bad input — if the boundary had a schema and a validator on it. In all three cases, the team had Pydantic in their codebase. They just didn't put it where the foreign data came in.
That's the lesson worth memorizing. The presence of Pydantic in
requirements.txt doesn't protect you. A schema at every boundary
where untrusted data enters is what protects you.
Next: a quick check that you can distinguish a boundary failure from a logic failure.
Three real breakages — what happens when nobody mans the border
The schema-at-the-boundary pattern sounds abstract until you see the bills. Below are three illustrative breakages — composites built from patterns that recur across production AI postmortems in the last 18 months. Specific company names, dollar figures, and incident dates are representative, not literal — the shape of the failure (and the fix) is what to internalize. Treat these the way you'd treat case studies in a textbook, not direct quotations.
Breakage 1: The receipt extractor that swapped fields
A small-business expense app uses Claude to extract vendor and amount from a photographed receipt. Schema (in the developer's head, not in code):
{
"vendor": "string",
"amount": "float (USD)",
}
The team prompted the model carefully: Return only valid JSON with
vendor and amount. In dev, on 50 test receipts, it always worked.
In prod, on a Saturday morning, the model started returning:
{"vendor": "12.99", "amount": "Whole Foods Market"}
The fields were swapped. The downstream code took whatever was in
amount, parsed it as a float (failed silently on the string), fell
through to a try/except, and charged the customer's stored card
the value in vendor — except that value was now the vendor name,
which the float() had thrown a ValueError on, which the catch-all
handler interpreted as "use the last successful amount."
Customers got charged the wrong amount. Some got charged twice. The support queue exploded. Two engineers spent a weekend rolling back.
What would have stopped it: a schema-aware extraction mode plus a
Pydantic validator at the boundary. The moment the model returned
vendor as a numeric-looking string and amount as text, the
validator would have raised ValidationError and the charge would
never have been attempted.
Cost: ~$40k in refunds and ~$15k in support hours. Root cause in the postmortem: "No validation between model response and billing code." Translation: no customs officer at the boundary.
Breakage 2: The router that hallucinated an enum
A team built an agent to route inbound candidate packets to the right panel. The
agent classified each packet with a priority
field. The intended values were low, medium, high.
Alert rules: anything with priority="high" woke the on-call
owner. The team did not define critical as a valid value. They
just didn't page on it, because it didn't exist.
The model, in production, started occasionally returning
priority="critical" on candidate packets that were unusually
urgent. Downstream code did a string-equality check against "high",
correctly skipped the page — but the panel dashboard
counted critical as high for the urgent bucket, marking thousands
of candidate packets as breached and firing automated notices
to candidates whose candidate packets
were actually fine.
What would have stopped it: a Pydantic Literal["low", "medium", "high"]
or an Enum. The model would not have been able to return critical
under a schema-aware mode; if it had under a non-enforced mode, the
validator would have rejected the response on arrival.
Cost: ~120 wrongly-issued notices, ~30 angry follow-ups. Root cause: "Free-string priority field, no enum validation at the API boundary."
Breakage 3: The eval harness that lost ground truth
An ML team built an eval harness to score Claude's answers to
customer-service questions. The expected-answer field was typed
loosely: expected: str. They scored the model's output with a
string-equality check against expected.
The expected answers came from a CSV. A junior engineer accidentally saved one row with the answer wrapped in a list:
expected
"yes"
"no"
["yes"]
The CSV parser loaded ["yes"] as the literal string '["yes"]'.
The model returned "yes". String equality returned False. The
eval scored that case as a regression — even though the model was
right. A week of "the model got worse" debugging followed. The team
rolled back a perfectly fine prompt change to fix a problem that
wasn't there.
What would have stopped it: a schema on the eval CSV itself. The
expected-answer field, defined as str with a Pydantic validator,
would have rejected the ["yes"] row on ingest. The harness would
have refused to run with bad ground truth.
Cost: 6 engineer-days of false-alarm debugging, one good prompt change reverted for no reason. Root cause: "No schema validation on eval inputs."
The pattern across all three
| Breakage | Trust boundary that wasn't guarded |
|---|---|
| Receipt swap | Model output → billing logic |
| Enum classifier | Model output → dashboard urgency logic |
| Eval CSV | File input → eval scoring logic |
Every one of these would have been caught — in seconds, on the first bad input — if the boundary had a schema and a validator on it. In all three cases, the team had Pydantic in their codebase. They just didn't put it where the foreign data came in.
That's the lesson worth memorizing. The presence of Pydantic in
requirements.txt doesn't protect you. A schema at every boundary
where untrusted data enters is what protects you.
Next: a quick check that you can distinguish a boundary failure from a logic failure.
Three real breakages — what happens when nobody mans the border
The schema-at-the-boundary pattern sounds abstract until you see the bills. Below are three illustrative breakages — composites built from patterns that recur across production AI postmortems in the last 18 months. Specific company names, dollar figures, and incident dates are representative, not literal — the shape of the failure (and the fix) is what to internalize. Treat these the way you'd treat case studies in a textbook, not direct quotations.
Breakage 1: The receipt extractor that swapped fields
A small-business expense app uses Claude to extract vendor and amount from a photographed receipt. Schema (in the developer's head, not in code):
{
"vendor": "string",
"amount": "float (USD)",
}
The team prompted the model carefully: Return only valid JSON with
vendor and amount. In dev, on 50 test receipts, it always worked.
In prod, on a Saturday morning, the model started returning:
{"vendor": "12.99", "amount": "Whole Foods Market"}
The fields were swapped. The downstream code took whatever was in
amount, parsed it as a float (failed silently on the string), fell
through to a try/except, and charged the customer's stored card
the value in vendor — except that value was now the vendor name,
which the float() had thrown a ValueError on, which the catch-all
handler interpreted as "use the last successful amount."
Customers got charged the wrong amount. Some got charged twice. The support queue exploded. Two engineers spent a weekend rolling back.
What would have stopped it: a schema-aware extraction mode plus a
Pydantic validator at the boundary. The moment the model returned
vendor as a numeric-looking string and amount as text, the
validator would have raised ValidationError and the charge would
never have been attempted.
Cost: ~$40k in refunds and ~$15k in support hours. Root cause in the postmortem: "No validation between model response and billing code." Translation: no customs officer at the boundary.
Breakage 2: The router that hallucinated an enum
A team built an agent to route inbound handoff exceptions to the right on-call lead. The
agent classified each step with a priority
field. The intended values were low, medium, high.
Alert rules: anything with priority="high" woke the on-call
owner. The team did not define critical as a valid value. They
just didn't page on it, because it didn't exist.
The model, in production, started occasionally returning
priority="critical" on handoff steps that were unusually
urgent. Downstream code did a string-equality check against "high",
correctly skipped the page — but the handoff dashboard
counted critical as high for the urgent bucket, marking thousands
of handoff steps as breached and firing automated notices
to shift leads whose handoff steps
were actually fine.
What would have stopped it: a Pydantic Literal["low", "medium", "high"]
or an Enum. The model would not have been able to return critical
under a schema-aware mode; if it had under a non-enforced mode, the
validator would have rejected the response on arrival.
Cost: ~120 wrongly-issued notices, ~30 angry follow-ups. Root cause: "Free-string priority field, no enum validation at the API boundary."
Breakage 3: The eval harness that lost ground truth
An ML team built an eval harness to score Claude's answers to
customer-service questions. The expected-answer field was typed
loosely: expected: str. They scored the model's output with a
string-equality check against expected.
The expected answers came from a CSV. A junior engineer accidentally saved one row with the answer wrapped in a list:
expected
"yes"
"no"
["yes"]
The CSV parser loaded ["yes"] as the literal string '["yes"]'.
The model returned "yes". String equality returned False. The
eval scored that case as a regression — even though the model was
right. A week of "the model got worse" debugging followed. The team
rolled back a perfectly fine prompt change to fix a problem that
wasn't there.
What would have stopped it: a schema on the eval CSV itself. The
expected-answer field, defined as str with a Pydantic validator,
would have rejected the ["yes"] row on ingest. The harness would
have refused to run with bad ground truth.
Cost: 6 engineer-days of false-alarm debugging, one good prompt change reverted for no reason. Root cause: "No schema validation on eval inputs."
The pattern across all three
| Breakage | Trust boundary that wasn't guarded |
|---|---|
| Receipt swap | Model output → billing logic |
| Enum classifier | Model output → dashboard urgency logic |
| Eval CSV | File input → eval scoring logic |
Every one of these would have been caught — in seconds, on the first bad input — if the boundary had a schema and a validator on it. In all three cases, the team had Pydantic in their codebase. They just didn't put it where the foreign data came in.
That's the lesson worth memorizing. The presence of Pydantic in
requirements.txt doesn't protect you. A schema at every boundary
where untrusted data enters is what protects you.
Next: a quick check that you can distinguish a boundary failure from a logic failure.
Three real breakages — what happens when nobody mans the border
The schema-at-the-boundary pattern sounds abstract until you see the bills. Below are three illustrative breakages — composites built from patterns that recur across production AI postmortems in the last 18 months. Specific company names, dollar figures, and incident dates are representative, not literal — the shape of the failure (and the fix) is what to internalize. Treat these the way you'd treat case studies in a textbook, not direct quotations.
Breakage 1: The receipt extractor that swapped fields
A small-business expense app uses Claude to extract vendor and amount from a photographed receipt. Schema (in the developer's head, not in code):
{
"vendor": "string",
"amount": "float (USD)",
}
The team prompted the model carefully: Return only valid JSON with
vendor and amount. In dev, on 50 test receipts, it always worked.
In prod, on a Saturday morning, the model started returning:
{"vendor": "12.99", "amount": "Whole Foods Market"}
The fields were swapped. The downstream code took whatever was in
amount, parsed it as a float (failed silently on the string), fell
through to a try/except, and charged the customer's stored card
the value in vendor — except that value was now the vendor name,
which the float() had thrown a ValueError on, which the catch-all
handler interpreted as "use the last successful amount."
Customers got charged the wrong amount. Some got charged twice. The support queue exploded. Two engineers spent a weekend rolling back.
What would have stopped it: a schema-aware extraction mode plus a
Pydantic validator at the boundary. The moment the model returned
vendor as a numeric-looking string and amount as text, the
validator would have raised ValidationError and the charge would
never have been attempted.
Cost: ~$40k in refunds and ~$15k in support hours. Root cause in the postmortem: "No validation between model response and billing code." Translation: no customs officer at the boundary.
Breakage 2: The router that hallucinated an enum
A team built an agent to route inbound privilege questions to the right partner. The
agent classified each question with a priority
field. The intended values were low, medium, high.
Alert rules: anything with priority="high" woke the on-call
owner. The team did not define critical as a valid value. They
just didn't page on it, because it didn't exist.
The model, in production, started occasionally returning
priority="critical" on privilege questions that were unusually
urgent. Downstream code did a string-equality check against "high",
correctly skipped the page — but the matter dashboard
counted critical as high for the urgent bucket, marking thousands
of privilege questions as breached and firing automated notices
to clients whose privilege questions
were actually fine.
What would have stopped it: a Pydantic Literal["low", "medium", "high"]
or an Enum. The model would not have been able to return critical
under a schema-aware mode; if it had under a non-enforced mode, the
validator would have rejected the response on arrival.
Cost: ~120 wrongly-issued notices, ~30 angry follow-ups. Root cause: "Free-string priority field, no enum validation at the API boundary."
Breakage 3: The eval harness that lost ground truth
An ML team built an eval harness to score Claude's answers to
customer-service questions. The expected-answer field was typed
loosely: expected: str. They scored the model's output with a
string-equality check against expected.
The expected answers came from a CSV. A junior engineer accidentally saved one row with the answer wrapped in a list:
expected
"yes"
"no"
["yes"]
The CSV parser loaded ["yes"] as the literal string '["yes"]'.
The model returned "yes". String equality returned False. The
eval scored that case as a regression — even though the model was
right. A week of "the model got worse" debugging followed. The team
rolled back a perfectly fine prompt change to fix a problem that
wasn't there.
What would have stopped it: a schema on the eval CSV itself. The
expected-answer field, defined as str with a Pydantic validator,
would have rejected the ["yes"] row on ingest. The harness would
have refused to run with bad ground truth.
Cost: 6 engineer-days of false-alarm debugging, one good prompt change reverted for no reason. Root cause: "No schema validation on eval inputs."
The pattern across all three
| Breakage | Trust boundary that wasn't guarded |
|---|---|
| Receipt swap | Model output → billing logic |
| Enum classifier | Model output → dashboard urgency logic |
| Eval CSV | File input → eval scoring logic |
Every one of these would have been caught — in seconds, on the first bad input — if the boundary had a schema and a validator on it. In all three cases, the team had Pydantic in their codebase. They just didn't put it where the foreign data came in.
That's the lesson worth memorizing. The presence of Pydantic in
requirements.txt doesn't protect you. A schema at every boundary
where untrusted data enters is what protects you.
Next: a quick check that you can distinguish a boundary failure from a logic failure.