promptdojo_

Mission: tensor playground — step 1 of 7

ch 42 · pytorch tensors and autograd1/7

Mission: the tensor playground

First mission of the deep-learning arc, and it runs on machinery you already own: start from the twenty-line autograd, prove every addition against a referee, then push it until it does real work.

The job: grow the twenty-line autograd from lesson 03 into a one-neuron learner — and prove every gradient you add is correct with a numeric check (the referee in the starter: nudge the input by ±ε, measure the slope, compare to what your backward claims).

The stages:

  1. Gradient check harness — the starter runs it on w*w + w. Note it already exercises a subtlety: w appears twice, and accumulation (+=) is what makes the gradient come out 5, not 4 or 1. You get chapter 42's accumulate rule for free.
  2. New ops — add relu() (gradient: pass-through if positive, zero otherwise) and subtraction. Numeric-check each.
  3. A neuron — pred = relu(w*x + b), squared-error loss against a target, .backward(), and a few hand-stepped w.data -= lr * w.grad updates (zero the grads between steps — you know why now). Watch the loss fall: chapter 39's training loop, running on machinery you built.
  4. Stress case — check relu's gradient exactly at 0 input, and a chain where one Value feeds two branches; the numeric referee settles both.
  5. One improvement — a zero_grad() method, a tanh op, or a two-input neuron: pick one, note why.

What goes in is you; what comes out is someone who has implemented forward, backward, accumulation, and gradient checking. After this, loss.backward() in torch is a bigger version of code you wrote.