Zach Furman

For wrong answers, they point in different directions and cancel.

This isn't a loose interpretive gloss.

Each piece, the circular embedding, the trig identities, the interference pattern, is concretely present in the weights and can be verified by ablations.

So here's the picture that emerges.

During the memorization phase, the network solves the task some other way, presumably something like a lookup table distributed across its parameters.

It fits the training data, but the solution doesn't extend.

Then, over continued training, a different solution forms.

This trigonometric algorithm.

As the algorithm assembles, generalization happens.

The two are not merely correlated.

Tracing the structure in the weights and the performance on held-out data, they move together.

What should we make of this?

Here's one reading.

The difference between a network that memorizes and a network that generalizes is not just quantitative, but qualitative.

The two networks have learned different kinds of things.

One has stored associations.

The other has found a method, a mechanistic procedure that happens to work on inputs beyond those it was trained on because it captures something about the structure of the problem.

This is a single example and a toy one.

But it raises a question worth taking seriously.

When networks generalize, is it because they've found something like an algorithm?