promptdojo_

Decoding, KV cache, and quantization — step 1 of 7

ch 44 · cnns, transformers, and useful llm internals1/7

Decoding, the KV cache, and quantization

Training gets the glory; inference pays the bills (chapter 23). Three ideas dominate how transformer inference actually runs — and they explain three knobs you've already used through APIs.

Decoding: how the next token gets picked

A language model's forward pass ends in a probability distribution over the vocabulary. Run the editor's first half:

  • Greedy decoding takes the argmax every time — deterministic, sometimes repetitive.
  • Sampling with temperature reshapes the distribution before drawing: low temperature (0.2) concentrates mass on the favorite — 20 draws, nearly all the; temperature 1.0 samples the distribution as-is and the alternatives appear. That is the temperature parameter you've been setting since chapter 13, mechanically. (Top-k / top-p are cousins: truncate the candidate set before sampling.)

The KV cache: why generation doesn't restart every token

Generating token 501 needs attention over tokens 1–500 — but their keys and values (last lesson) haven't changed. So serving stacks cache them, and each new token computes only its own Q/K/V plus one attention row against the cache. Without the cache you'd redo the whole prefix every step; with it, generation does incremental work per token but must hold K and V for every position in GPU memory — which is why long contexts cost memory even when they're cheap to re-read, and it's the mechanism behind prompt caching economics in chapter 23: a stable prefix is a reusable cache.

Quantization: fewer bits, nearly the same model

Weights train in 32- or 16-bit floats, but storing and multiplying them at 8-bit or 4-bit integers makes models dramatically smaller and faster to serve. Run the editor's second half: rounding weights to coarse steps introduces small per-weight error — that's quantization in one line. Across billions of weights the errors mostly wash out, which is why quantized variants ship everywhere serving cost matters; aggressive 4-bit settings trade a little quality for a lot of feasibility, and the honest way to pick is chapter 21's answer — run your evals on the quantized model, not vibes.

Where AI specifically gets this wrong

  • Temperature superstition. Generated configs copy magic values. You've now watched the knob work on a 3-word vocabulary; reason from the mechanism (deterministic tasks → low or greedy; creative variety → higher).
  • "Just increase max_tokens/context" without the memory story. The KV cache makes context a memory budget at serving time — capacity planning, not a config tweak.
  • Quantizing without re-evaluating. The whole tradeoff is quality-vs-cost; skipping the eval half of it means shipping an unmeasured model — the anti-pattern chapter 21 exists to stop.