Nicholas Andresen

At some point the juggling act gets too hard.

There's nowhere to put things.

The solution, as every school child knows, is scratch paper.

When you write down 17,468, that number becomes the paper's responsibility.

You can clear your working memory and start fresh on the next step.

Language models face an analogous constraint.

There's a limit to how much reasoning they can do in a single pass.

The 4chan discovery was that a model's own output could serve as scratch paper.

Get it to write down step 1, and now step 1 is just sitting there in the context, available for computing step 2.

Chain enough steps together and the model can solve problems that would otherwise have been impossible for it.

Researchers formalized this.

Nayatal called it Scratchpad 2021.

Wei Yital called it Chain of Thought, 2022, which is the name that stuck.

The AI safety community called it maybe our last hope for understanding what these things are doing before they turn into completely opaque superintelligence, please God let it keep working.

We got unbelievably, preposterously, cosmically lucky with Chain of Thought.

For years, the trajectory of AI pointed one direction.

Bigger models, more capabilities, more opacity.

GPT-3 was more capable than GPT-2 and harder to understand.

GPT-4 was more capable than GPT-3 and harder to understand, and so on.

Every step forward in intelligence was a step backward in interpretability.