Francois Chollet

So the model is not actually learning anything on the fly.

Its state is not adapting.

to the task at hand.

And what Jack Core is actually doing is that for every test problem is on the fly, is fine-tuning a version of DLLM for that task.

And that's really what's unlocking performance.

If you don't do that, you get like 1%, 2%.

So basically something completely negligible.

And if you do test time fine-tuning and you add a bunch of tricks on top, then you end up with interesting performance numbers.

So I think what it's doing is trying to address one of the key limitations of LLMs today, which is the lack of active inference.

It's actually adding active inference to LLMs.

And that's working extremely well, actually.

So that's fascinating to me.

No, it's not just a technical detail.

It's not a straightforward thing.

It is everything.

It is the important part.

And the scale maximalist argument, it boils down to, you know, these people, they refer to scaling laws, which is this empirical relationship that you can draw between how much compute you spend on training a model and the performance you're getting on benchmarks, right?

And the key question here, of course, is, well, how do you measure performance?

What it is that you're actually improving by adding more compute and more data?

And well, it's benchmark performance, right?