Francois Chollet

And when you give them a new puzzle, they can just fetch the appropriate program, apply it, and it's looking like it's reasoning, but really it's not doing any sort of on-the-fly program synthesis.

All it's doing is program fetching.

So you can actually solve all these benchmarks with memorization.

And so what you're scaling up here, like if you look at the models, they are big parametric curves fitted to a data distribution, which I call a descent.

So they're basically these big interpolative databases, interpolative memories.

And of course, if you scale up the size of your database and you cram into it more knowledge, more patterns and so on,

you are going to be increasing its performance as measured by a memorization benchmark.

That's kind of obvious.

But as you're doing it, you are not increasing the intelligence of the system one bit.

You are increasing the skill of the system.

You are increasing its usefulness, its scope of applicability, but not its intelligence because skill is not intelligence.

And that's the fundamental confusion

that people run into is that they're confusing skill and intelligence.

That's memorization benchmarks.

It depends how you want to define reasoning, but there are two definitions you can use.

So one is, I have available a set of program templates.

It's like the structure of the puzzle.

which can also generate its solution.

And I'm just going to identify the right template, which is in my memory.

I'm going to input the new values into the template, run the program, get the solution.