David Duvenaud

But I think that's like some of the most basic groundwork that needs to be done at this point is like clarify what we're even talking about.

Yeah.

So actually, I had the exact same thought.

And that's why that leads me to one of the projects that I'm working on, like the actual technical projects that I'm working on, which is me and a few people, including Alec Radford, who's like one of the creators of GPT, who's now sort of like unemployed and just doing fun research projects, is trying to train a historical LLM, like a LLM that's only trained up on data up to like, let's say, 1930 and then like maybe 40, 1950.

And the idea being that

As you said, it's hard to operationalize these questions like, I don't know, what fraction of humans are employed?

It might not really matter or be the right question to ask.

What we'd rather ask is something more like, what is the future newspaper headline?

Or given a leader, what's their Wikipedia page or something like that?

It's more like freeform sort of things.

And the cool thing is that

LLMs, you can query them to predict this sort of thing, right?

Like, write me a newspaper headline from 2030 or whatever.

I mean, they're not going to do a good job unless they have a lot of scaffolding and specific training.

But we can validate that kind of scaffolding on historical data using these historical LLMs.

So the idea is you train a model only on data up to 1930, then you ask it to predict the likelihood that it would give to a headline in 1940 or some other free-form text.

And you can evaluate their likelihoods on this text

in the past and then you can also use the same scaffolding on a model train up to 2025 and then ask it to predict like headlines in 2035 and get a rough idea of like or you can iterate on your scaffolding by seeing how well it does on like past data

So that's been the huge slap so far is like constantly finding different sources of unintentional data poisoning and like mislabeled data and things like that.

So, I mean, their elements can help you because there's sort of like a chicken and egg.