Zach Furman

The field's response was pragmatic.

Scale the methods that work.

Stop trying to understand why they work.

This attitude was partly earned.

For decades, hand-engineered systems encoding human knowledge about vision or language had lost to generic architectures trained on data.

Human intuitions about what mattered kept being wrong.

But the pragmatic stance hardened into something stronger, a tacit assumption that trained networks were intrinsically opaque, that asking what the weights meant was a category error.

At first glance, this assumption seemed to have some theoretical basis.

If neural networks were best understood as just curve-fitting function approximators, then there was no obvious reason to expect the learned parameters to mean anything in particular.

They were solutions to an optimization problem, not representations.

And when researchers did look inside, they found dense matrices of floating point numbers with no obvious organization.

But a lens that predicts opacity makes the same prediction whether structure is absent or merely invisible.

Some researchers kept looking.

PowerIT OWL 2022, train a small transformer on modular edition.

Given two numbers, output their sum mod 113.

Only a fraction of the possible input pairs are used for training, say, 30%, with the rest held out for testing.

The network memorized the training pairs quickly, getting them all correct.

But on pairs it hasn't seen, it does no better than chance.

This is unsurprising.

With enough parameters, a network can simply store input output associations without extracting any rule.