promptdojo_

Missing values, dtypes, and the silent model bug — step 1 of 7

ch 35 · dataframes with numpy and pandas1/7

Missing values, dtypes, and the silent model bug

Real data has holes. A user without an age, a sensor that skipped a reading, an API field that arrived null. Pandas represents most of those holes as NaN — and NaN follows rules that surprise everyone once.

The NaN rules

Run the editor. Three facts to internalize:

  1. NaN never equals anything, including itself. nan == nan is False. So df[df["age"] == np.nan] silently matches nothing. The real checks are df["age"].isna() and .notna().
  2. NaN poisons arithmetic. Any plain sum or product touching NaN becomes NaN. (Pandas aggregations like .mean() skip NaN by default — convenient, but know that it's skipping rows, which changes your denominator.)
  3. NaN is a float. An integer column that gains even one missing value gets promoted to float64 — your clean user_id ints suddenly print as 7.0. (Pandas has nullable integer dtypes like Int64 for exactly this, but you must opt in.)

The standard triage

df.isna().sum()              # missing count per column — look first
df = df.dropna(subset=["label"])        # rows useless without a label
df["age"] = df["age"].fillna(df["age"].median())   # impute a feature

Which move is right is a modeling decision, not a syntax one: dropping rows shrinks and can bias your dataset; filling values invents data. Either way, do it explicitly and in one visible place — not scattered wherever the crash happened.

Dtypes: the other half of the bug

df.dtypes right after loading, every time. The classics:

  • Numbers loaded as object (strings) because one row contained "N/A" or a currency symbol — chapter 03's "5" + 10 TypeError, at column scale. Fix at the boundary: pd.to_numeric(col, errors="coerce") turns junk into NaN you can then handle.
  • Dates loaded as strings — pd.to_datetime before any time math.
  • Booleans loaded as "true"/"false" strings — truthiness bugs identical to chapter 18's DEBUG env-var trap.

Where AI specifically gets this wrong

First, the reflex-level one: == np.nan is always wrong — it matches nothing, ever. isna() is the only check that works.

  • Silent NaN-skipping means. A column that's 60% missing still yields a confident .mean() — computed on the 40%. Print isna().sum() next to any aggregate you plan to trust.
  • Imputing before splitting. Filling with the whole dataset's median leaks test-set information into training. Impute after the split, fit on train only — chapter 38 makes this precise.