promptdojo_promptdojo_promptdojo_promptdojo_promptdojo_promptdojo_promptdojo_promptdojo_promptdojo_promptdojo_promptdojo_

Attention and Transformer blocks — step 5 of 7

ch 44 · cnns, transformers, and useful llm internalsch 44 · cnns, transformers, and useful llm internalsch 44 · cnns, transformers, and useful llm internalsch 44 · cnns, transformers, and useful llm internalsch 44 · cnns, transformers, and useful llm internalsch 44 · cnns, transformers, and useful llm internalsch 44 · cnns, transformers, and useful llm internalsch 44 · cnns, transformers, and useful llm internalsch 44 · cnns, transformers, and useful llm internalsch 44 · cnns, transformers, and useful llm internalsch 44 · cnns, transformers, and useful llm internals5/7

Two keys that are 95% identical get attention weights of [0.96, 0.04] — practically a hard argmax over near-twins. The bug: raw dot products grow with DIMENSION (16 terms of ~4 each ≈ 64), so tiny relative differences become huge absolute gaps and softmax saturates. That's exactly why attention divides scores by √dim. Add the scaling.

The break is on line 13 — but read the whole snippet first.

full-screen editor opens — close anytime to keep reading.

open the editor to run it.