promptdojo_

files without fear — open(), pathlib, and the receipt for your data — step 3 of 8

ch 04 · read the pipeline3/8

pathlib — and the receipt that catches silent data swaps

Reading AI-generated Python, you'll see two completely different ways of handling files. The old one strings together os.path.join(...), os.path.exists(...), and an open/with block. The new one is one object:

from pathlib import Path

log_path = Path("data") / "logs" / "today.log"
if log_path.exists():
    text = log_path.read_text()

A Path is a value that represents a filesystem location — it doesn't matter whether the file exists yet. The / operator joins segments correctly on every platform (no "a" + "/" + "b" string mashing, which breaks on Windows). .read_text() opens, reads, and closes in one call; .write_text("...") is its mirror (it overwrites — no append, no warning); .read_bytes() gives you the raw bytes for binary files. The with block still matters when you're streaming line by line — but for "load the whole thing," the Path methods are the move.

Inspecting a path without touching disk

Four attributes AI code leans on constantly:

  • p.name — the last segment: "q1.csv"
  • p.suffix — the dot-extension: ".csv"
  • p.stem — the name minus the extension: "q1"
  • p.parent — everything before the last /

None of these touch the filesystem; they're string surgery on the path itself. p.exists(), by contrast, actually checks disk.

The manifest — a receipt for your dataset

Now the part AI never writes on its own. Here's a failure mode that produces no error message anywhere: a teammate regenerates eval.csv with a "small cleanup", your evaluation script happily reads the new file under the old name, and your model's accuracy jumps four points. Two weeks later someone asks which rows you evaluated on, and nobody knows. The file changed and nothing noticed. Files don't version themselves.

The fix is old, boring, and beautiful: a manifest — a small record, kept next to the data, that says this exact file, with this many rows and this checksum, is what we mean by eval.csv. Three fields do most of the work:

  1. file — which file this entry describes (path.name).
  2. rows — how many records. For a CSV that's the newline count minus one, because the first line is the header, not data. For JSONL — one JSON object per line, the format fine-tuning data and eval logs ship in — there's no header: newline count is the record count.
  3. sha256 — a checksum of the raw bytes, via the stdlib's hashlib. hashlib.sha256(data) returns a hash object; .hexdigest() turns it into a hex fingerprint string you can store and compare. Change one character anywhere in the file and the fingerprint changes completely — the first 8-12 characters are plenty for catching swaps.

The editor on the right builds one real entry: read_bytes() gets the contents, sha256(...).hexdigest()[:12] fingerprints them, and the entry itself is dumped as JSON — the config-shaped blob json.dumps (and its file-writing sibling json.dump) exists for.

Where AI specifically gets this wrong

One: old-style path mashing. os.path.join("a", "b") and f"{folder}/{file}" both work until they don't. Every one is a Path("a") / "b" waiting to happen. Replace them.

Two: hashing the wrong thing. hashlib.sha256(str(path).encode()) hashes the filename, not the contents. The check passes forever, no matter what's in the file — a guardrail made of paint. Hash read_bytes(), always.

Three: writing the manifest and never reading it. Cursor will happily generate build_manifest.py and stop. A manifest you never verify against is a diary, not an alarm. The verify step — re-measure, compare, complain — is the entire point, and it's the checkpoint at the end of this lesson.

Run the editor. One file in, one receipt out — twelve hex characters that will tattle on any future edit.

read, then continue.