Tom Griffiths

So Chomsky's approach to language was very good in sort of illustrating that language was a much more complex object than behaviorists had assumed, right?

That you needed to have kind of like internal structures and things like verb phrases and noun phrases and things that sort of look like grammars in order to account for the structure of human languages.

But then it created this new problem, which was, how is it that human beings could possibly learn these very complex objects from the limited data that they get?

And so that's the thing that then pushed Chomsky to say, well, maybe they're not really learning it.

acquiring it as a consequence of having some very strong constraints on what it is that they can learn as languages, but then only require relatively small amounts of data to determine, oh, okay, it's this particular configuration of bits and pieces that characterizes the language that I'm actually speaking here.

Yeah, so that's exactly the contrast, right, is that...

The AI models give us a really good illustration of how much language you need to solve the problem that Chomsky had identified, to learn something that is this very complex object.

If we take as given that they've learned something like human language, then you can think about that as a proof of the point that Chomsky was making.

that you're going to need lots and lots and lots of data to learn something like human language.

So he was right that in the five years that the kid gets, they're not going to be able to figure out the structural language.

And in fact, it turns out you need something more like 5,000 or 50,000 years of data in order to do a really good job.

I think that's a really nice way of thinking about what that difference is.

And it also gives us a good way of thinking about what the challenge is then if you wanted to make systems that are more human-like in their ability to learn.

So you can think about it.

If you want to be able to learn from five years of data, and you're currently learning from 5,000 years of data, then you've got 4,995 years to make up in terms of the content of that

inductive bias or those prior distributions that are inside the child's head.

And so you can think about that coming from these other kinds of sources.

So evolution is one of those, as well as some other things.

So the broader set of experiences that the child has as they're learning language, sort of like building a model of the world around them that means that the things that they're learning aren't just sort of arbitrary sequences of

words, but actually things that map onto things that are meaningful, right?