Row-level random split leaks
A random 80/20 on rows looks clean in a notebook: 400 train, 100 eval, no duplicate line. It still leaks. Logs from acme-pay share job names, stack shapes, and the same flaky npm 503. If six Acme rows land in train and two in eval, the adapter is graded on a company it already saw.
The split unit is the company (or the source document, if you do not have companies). Every row from acme-pay is train, or every row is eval. splits.json lists the IDs so a reviewer can see the partition without grepping jsonl.
{
"unit": "company",
"train": ["acme-pay", "northwind-ci", "helix-mobile"],
"eval": ["fieldnote-web", "lumen-billing"]
}
Those two lists must not overlap. A doc_id that appears under two companies is a data bug, not a clever split.
Why this is the hold that people skip: loss goes down, eval accuracy looks great, and the "held-out" set is the same flake phrasing with a different timestamp. The eval receipt later will refuse a contaminated split even if the FT numbers beat prompt-only. Baking the leak into the receipt is how a bad adapter ships.
If you only have one company, split by doc_id (one failing job, one document) and say so in splits.json. Do not pretend a row shuffle is a company split.