Sholto Douglas

Because if it is the case that you learn certain things first, shouldn't just directly training those things first lead to better results?

Both Gemini papers mention some aspects of curriculum learning.

Okay, interesting.

The fact that fine-tuning works is evidence of curriculum learning, right?

Because the last things you're training on have a disproportionate impact.

Sorry, what was the thing we were talking about before?

By the way, I just realized, I forgot to...

I just got in conversation mode and forgot there's an audience.

Curriculum learning is when you organize a data set.

When you think about a human, how they learn, they don't just see random wiki text and they just try to predict it.

They're like, we'll start you off with Lorax or something and then you'll learn... I don't even remember what first grade was like, but you'll learn the things that first graders learn and then second graders and so forth.

Sorry, we know you never got past first grade.

Anyways, uh, let's get back to like the big, before we get into like a bunch of like interim details, the big picture, um,

there's two threads I want to explore.

First is, I guess it makes me a little worried that there's not even an alternative formulation of what could be happening in these models that could invalidate this approach, which feels like, I mean, we do know that we don't understand intelligence, right?

Like there are definitely unknown unknowns here.

So,

Like the fact that there's not a null hypothesis.

I don't know.

I feel like what if we're just wrong and we don't even know the way in which we're wrong, which actually increases the uncertainty.