Francois Chollet

And that's working extremely well, actually.

So that's fascinating to me.

No, it's not just a technical detail.

It's not a straightforward thing.

It is everything.

It is the important part.

And the scale maximalist argument, it boils down to, you know, these people, they refer to scaling laws, which is this empirical relationship that you can draw between how much compute you spend on training a model and the performance you're getting on benchmarks, right?

And the key question here, of course, is, well, how do you measure performance?

What it is that you're actually improving by adding more compute and more data?

And well, it's benchmark performance, right?

And the thing is, the way you measure performance is not a technical detail.

It's not an afterthought because it's going to narrow down the set of questions that you're asking.

And so accordingly, it's going to narrow down the set of answers that you're looking for.

If you look at the benchmarks we're using for LLMs, they're all memorization-based benchmarks, right?

Like sometimes they are literally just knowledge-based, like a school test.

And even if you look at the ones that are explicitly about reasoning, you realize if you look closely that in order to solve them, it's enough to memorize a finite set of reasoning patterns.

and then you just reapply them.

They're like static programs.

LLMs are very good at memorizing static programs, small static programs.

And they've got this sort of like bank of solution programs.