Neural network explorations
Why does a net need a nonlinearity at all?
A hidden layer with no activation can only draw a straight boundary, so it can’t separate two interleaved rings. Add an activation and it can.
Two noisy rings: the inner disc is class 0, the outer band is class 1. No straight line separates them. Both panels use the same network, and the only thing that changes is whether the hidden layer has an activation. Without one it can only draw a straight line and stalls near chance; with one it wraps the ring. Switch the activation to see how each one bends the boundary.
No activation—
With activation—
Training loss
Does stacking layers actually add power?
Several stacked linear layers collapse to a single linear map, so a deep linear network is no more expressive than one layer. A nonlinearity between the layers changes that.
Left: a single linear layer.
Middle: several linear layers with no activations.
Right: the same stack with a nonlinearity between each pair.
Change the depth, width, or activation: the two all-linear panels stay a straight line, only the nonlinear one bends.
1 linear layer—
Deep · all linear—
Deep · nonlinear—
Each linear layer just multiplies its input by a weight matrix. Stacking several with no activation between them chains those multiplications, and a chain of matrix multiplications is itself a single matrix. So multiplying the deep-linear panel’s trained weight matrices together collapses its whole stack into one matrix, the same 1×2 shape as the single linear layer. Here are the single layer’s weights next to the deep stack’s collapsed product, both computed live from the current weights:
The two rows are the same numbers. The deep stack computes the identical linear map, which is why its panel draws the same boundary as the single layer.
Can a model learn similarity from next-token alone?
Trained only to predict the next token in a tiny grammar, an embedding table starts clustering related tokens, even though similarity was never part of the training signal.
A toy language with a single template:
the <animal> <verb> the <fruit>. The
categories are animals (cat, dog, cow), verbs (eat, chase, see), and
fruits (apple, mango, banana), plus “the”. Because every animal is
followed by a verb, all animals share the same next-token distribution,
and the same holds within the verb and fruit categories.
One embedding table feeds a softmax over the next token. Tokens that share a next-token distribution end up with similar embeddings, so same-category tokens land close together. After training, the embeddings are projected to 2D and the categories separate.
Embeddings → 2D
Next-token loss
When does a model memorize instead of learn?
A high-capacity model on tiny data drives its training loss to almost zero while held-out performance stays bad. More data closes the gap.
An over-parameterized network (two hidden layers) is trained on two overlapping blobs, with a fraction of the training labels flipped to add noise. The same network is trained three times: on 20, 200, and 2000 points.
Accuracy is scored against the true labels for both the training points and a held-out set, so the gap between the two is a real generalization gap. The training loss (bottom chart) is scored against the noisy labels the network actually fits: with 20 points it drives that loss to zero by memorizing the noise, with 2000 it cannot, so it settles on the real boundary. The decision boundary goes from jagged and overfit to smooth as the data grows.