Sholto Douglas

We were previously discussing is the brain organized in the way where you have a residual stream that is gradually refined with higher level associations over time or something.

There's a fixed dimension size in a model.

If you had to, I don't even know how to ask this question in a sensible way, but what is the D model of the brain?

What is it like the embedding size of, or because of feature splitting, is that not a sensible question?

Do you think that the way to think about whether it's in the brain or whether it's in these models, fundamentally what's happening is features are added, removed, changed, and the feature is the fundamental unit of what is happening in the model?

What would have to be true for... This goes back to the earlier thing we were talking about, whether it's just associations all the way down.

Give me a counterfactual.

In the world where this is not true, what is happening instead?

What is the alternative hypothesis here?

Still on definitions thing, I guess naively I think of things like bird versus...

What kind of token is it?

Is it like a period at the end of a hyperlink, as you were talking about earlier, versus at the highest level, things like love or deception or like holding a very complicated proof in your head or something?

Is this all features?

Because then the definition seems so broad as to almost be not that useful.

Or rather that there seems to be some important differences between these things and they're all features.

What would you discover tomorrow in...

that could make you think like, oh, this is like kind of fundamentally the wrong way to think about what's happening in a model.

Like in a naive sort of outsider way, the thing that would seem to me to be like a way in which this picture could be wrong is if there's not some like this thing is turned on, turned off, but it's like a much more global kind of like the system is I'm going to be really clumsy.

Like, you know, I mentioned it in a pretty kind of language, but is there a good analogy here?

Yeah, I guess if you think of like something like the laws of physics, it's not like, well, the feature for wetness is turned on, but it's only turned on this much.