Zach Furman

Compositional relationships between programs might correspond to some notion of path adjacency defined by the parameter function map.

If programs sharing structure are nearby, reachable from each other via direct paths, and if simpler programs lie along paths to more complex ones, then efficiency, simplicity bias, and empirically observed stagewise learning would follow naturally.

Gradient descent would build incrementally rather than search randomly.

The enumeration problem that dooms Solomonoff would dissolve into traversal.

This is speculative and imprecise.

But there's something about the shape of what's needed that feels mathematically natural.

The representation problem asks for a correspondence at the level of objects.

Strata in parameter space corresponding to programs.

The search problem asks for something stronger that this correspondence extends to paths.

Paths in parameter space, what gradient descent traverses, should correspond to some notion of relationship or transition between programs.

This is a familiar move in higher mathematics, sometimes formalized by category theory.

Once you have a correspondence between two kinds of objects, you ask whether it extends to the relationships between those objects.

It is especially familiar, in fields like higher category theory, to ask these kinds of questions when the relationships between objects take the form of paths in particular.

I don't claim that existing machinery from these fields applies directly and certainly not given the lack of detail I've provided in this post.

But the question is suggestive enough to investigate.

What should adjacency between programs mean?

Does the parameter function map induce or preserve such structure?

And if so, what does this predict about learning dynamics that we could check empirically?

Heading Appendix Subheading

Related work.