Sholto Douglas

There's so many different threads in that I want to follow up on.

Let's begin with the data versus like, yeah, compute thing of like, is the output of these AI is the thing that's causing the intelligent explosion or something?

Yeah.

People talk about how these models are really a reflection on their data.

I think there was, I forgot his name, but there was a great blog by this open AI engineer.

And he was talking about, at the end of the day, as these models get better and better, they're just going to be really effective projectors

like maps of the data set.

And so it's like at the end of the day, like you got to stop thinking about architectures.

It's like the most effective architecture is just like doing an amazing job of mapping the data.

Right.

So that implies that future AI progress comes from the AI just making really awesome data, right?

Like that you were mapping to.

I think that's clearly a very important part.

Yeah.

Yeah, that's really interesting.

Does that look to you like...

I don't know, like things that look like chain of thought, or what do you imagine as these models get better, as these models get smarter, what does the synthetic data look like?

Um, it's actually funny cause I had a conversation with Grant Sanderson like last year where we're debating this and I was like, fuck dude, by the time they get the gold of the math Olympiad, of course they're going to automate all the jobs.

Um, um,

On the synthetic data thing, one of the things I speculated about in my scaling post, which was heavily informed with discussions with you two, and you especially, Sholto, was you can think of human evolution through the perspective of we get language, and so we're generating the synthetic data.