Geoffrey Hinton

So the big language models are solving the problem

With not many connections, only a trillion, how do I make use of a huge amount of experience?

And back propagation is really, really good at packing huge amounts of knowledge into not many connections.

But that's not the problem we're solving.

We've got huge numbers of connections, not much experience.

We need to sort of extract the most we can from each experience.

So we're solving slightly different problems, which is one reason for thinking the brain might not be using back propagation.

Yeah.

So that's a very good question.

And what happened for several years, quite a few years, is that every time they made the neural net bigger and gave it more data, it got better.

It scaled.

Makes sense.

And it got better in a very predictable way.

So you could figure out, you know, it's going to cost me $100 million to make it this much bigger and give it this much more data.

Is it worth it?

And you could predict ahead of time, yes, it's going to get this much better.

It's worth it.

It's an open question whether that's petering out now.

there's some neural nets for which it won't peter out, where as you make them bigger and give them more data, they'll just keep getting better and better.

And then neural nets where they can generate their own data.