Zach Furman

Compositionality, Hierarchy, and Modularity

My informal notion of programs tier is quite closely related to compositionality.

It is a fairly well-known hypothesis that supervised learning performs well due to compositional, hierarchical or modular structure in the model and or the target task.

This is particularly prominent within approximation theory, especially the literature on depth separations, as an explanation for the issues I highlighted in the Paradox of Approximation section.

Mechanistic Interpretability

The implicit, underlying premise of the field of mechanistic interpretability is that one can understand the internal mechanistic, read, program-like structure responsible for a network's outputs.

Mechanistic interpretability is responsible for discovering a significant number of examples of this type of structure, which I believe constitutes the single strongest evidence for the program synthesis hypothesis.

I discuss a few case studies of this structure in the post, but there are possibly hundreds more examples which I did not cover from the many papers within the field.

A recent review can be found here.

Singular Learning Theory In the Path Forward section, I highlight a possible role of degeneracies in controlling some kind of effective program structure.

In some way, which I have gestured at but not elaborated on, the ideas presented in this post can be seen as motivating singular learning theory as a means to formally ground these ideas and produce practical tools to operationalize them.

This is most explicit within a line of work within singular learning theory that attempts to precisely connect program synthesis with the singular geometry of a toy learning machine.

This article was narrated by Type 3 Audio for Less Wrong.

It was published on January 20, 2026.

The original text contained 14 footnotes which were omitted from the narration.

Images are included in the podcast episode description.