Francois Chollet

But the big limitation is that it's extremely computing efficient, because you're running into combinatorial explosion, of course.

And so you can sort of see here how

deep learning and discrete program search, they have very complementary strengths and limitations as well.

Like every limitation of deep learning has a strength, a corresponding strength in program synthesis and inversely.

And I think the path forward is going to be to merge the two.

to basically start doing.

So another way you can think about it is, so these parametric curves trained with ground descent, they're a great fit for everything that's system one type thinking, like pattern cognition, intuition, memorization, and so on.

And discrete program search is a great fit for a type two thinking, system two thinking.

For instance, planning, reasoning, quickly figuring out a generalizable model that matches just one or two examples, like for an arc puzzle, for instance.

And I think humans are never doing pure system 1 or pure system 2.

They're always mixing and matching both.

and right now we have all the tools for system one we have almost nothing for system two the way forward is to create a hybrid system and i think the the form it's going to take is it's going to be uh mostly system two so the the outer structure is going to be a discrete program search system but you're going to fix the fundamental limitation of discrete program search which is commander explosion you're going to fix it with deep learning you're going to leverage deep learning to guide to provide intuition

in program space to guide the program search.

And I think that's very similar to what you see, for instance, when you're playing chess or when you're trying to prove a theorem, is that it's mostly a reasoning thing, but you start out with some intuition about the shape of the solution.

And that's very much something you can get via a deep learning model.

Deep learning models, they're very much like intuition machines.

their pattern matching machines.

So you start from this shape of the solution, and then you're going to do actual explicit discrete program search.

But you're not going to do it via brute force.

You're not going to try things kind of like randomly.