Francois Chollet

Again,

If it were true that LLMs could generalize to novel problems like this and actually develop software to solve a problem they've never seen before, you would not need software engineers anymore.

In practice, if I look at how people are using LLMs in their software engineering job today, they are using it as a stack overflow replacement.

So they are using it as a way to copy-paste code snippets to perform very common actions.

And what they actually need is a database of code snippets.

They don't actually need any of the abilities that actually make them software engineers.

So not quite.

So GPT-4 has a higher degree of skill and a higher range of skills, with the same degree of generalization.

I think interpolation can be creative, absolutely.

And to your point, I do think that on some level, humans also do a lot of memorization, a lot of reciting, a lot of pattern matching, a lot of interpolation as well.

So it's very much a spectrum.

between pattern matching and true reasoning, it's a spectrum.

And humans are never really at one end of the spectrum.

They're never really doing pure pattern matching or pure reasoning.

They're usually doing some mixture of both.

Even if you're doing something that seems very reasoning-heavy, like proving a mathematical theorem,

As you're doing it, sure, you're doing quite a bit of discrete search in your mind, quite a bit of actual reasoning, but you're also very much guided by intuition, guided by dimension, guided by the shape of proofs that you've seen before, by your knowledge of mathematics.

So it's never really, you know, all of our thoughts, everything we do is a mixture of this sort of like interpolated memorization-based thinking, this sort of like type one thinking and type two thinking.

Why are bigger models more sample efficient?

Because they have more reusable building blocks that they can lean on to pick up new patterns in their training data.