Francois Chollet

They cannot adapt on the fly to what they're seeing.

And he's actually trying to learn.

So what he's doing is effectively a form of program synthesis.

because the LLM contains a lot of useful building blocks, like programming building blocks, and by fine-tuning it on the task at test time, you are trying to assemble these building blocks into the right pattern that matches the task.

This is exactly what program synthesis is about.

And the way we contrast this approach with discrete program search is that in discrete program search,

So you're trying to assemble a program from a set of primitives.

You have very few primitives.

So people working on discrete program search on Arc, for instance, they tend to work with DSLs that have like 100 to 200 primitive programs.

So very small DSL, but then they're trying to combine these primitives into very complex programs.

So there's a very deep...

depths of search.

And on the other hand, if you look at what Jack Cole is doing with LLMs, is that he's got this sort of like vector program database, DSL, of millions of building blocks in the LLM that are mined by pre-training the LLM, not just on a ton of programming problems, but also on millions of generated arc-like tasks.

So you have an extraordinarily large DSL.

And then the fine-tuning is very, very shallow recombination of these primitives.

So discrete program search, very deep recombination, very small set of primitive programs.

And the LLM approach is the same, but on the complete opposite end of that spectrum, where you scale up the memorization by a massive factor and you're doing very, very shallow search.

But they are the same thing, just different ends of the spectrum.

And I think where you're going to get the most value for your compute cycles is going to be somewhere in between.

You want to leverage memorization to build up a richer, more useful bank of alternative programs.