Sholto Douglas

And Trenton, who's an anthropic, works on mechanistic interoperability, and it was widely reported that he has solved alignment.

Yeah.

So this will be a capabilities-only podcast.

Alignment is already solved, so no need to discuss further.

Okay, so let's start by talking about context links.

It seemed to be underhyped given how important it seems to me to be that you can just put a million tokens into context.

There's apparently some other news that got pushed to the front for some reason.

But yeah, tell me about how you see the future of long context links and what that implies for these models.

In context, are they as sample efficient and smart as humans?

I think that's really worth exploring.

So if this is true, it seems to me that these models are already, in an important sense, superhuman.

Not in the sense that they're smarter than us, but I can't keep a million tokens in my context when I'm trying to solve a problem, remembering and integrating all the information into our code base.

Am I wrong in thinking this is like a huge unlock?

How do we explain in-context learning?

Yeah, exactly.

Okay.

I only read the intro and discussion section of that paper, but in the discussion, the way they framed it is that in order to get better at long context tasks, the model has to get better at learning to learn from these examples or from the context that is already within the window.

And the implication of that is

If meta-learning happens because it has to learn how to get better at long-context tasks, then in some important sense, the task of intelligence requires long-context examples and long-context training.

Right, but you can proxy for that just by getting better at doing long-context tasks.