Daniel Jeffries (Unknown)

I'm thinking of that anthropic Golden Gate Bridge example, and they had this unsupervised method of finding salient features, and then they actually showed you the attribution in the source data.

And they were saying that this is evidence that the models were learning high-level abstract features, but they seemed quite low-level to me.

I just wondered, what's the level of abstraction you're seeing with the data attribution?

Yeah, I mean, this might be a nice segue, but I think you would call that long tail of exemplar features non-robust features.

But maybe we should just back up a little bit.

I mean, define abstract feature.

I mean, what does that mean to you?

Yes.

What is the relationship between abstraction and reasoning?

I mean, do you think neural networks reason?

Let's just put my cards on the table for a second.

Let's say abstraction is some form of compression and the statistical interpretation we can debate whether it's in distribution or out of distribution.

And reasoning is the ability to create a new model, create new knowledge.

So I think when people talk about reasoning, that they're saying that the model is somehow compressing representations and then building new models, maybe some kind of structural learning so that it can apply that knowledge in a new situation.

Some people say that the deflationary take is that neural networks, you scale them up and they learn more of the long tail and they're just basically memorizing loads and loads of data.

And the other take is that through the machinations of prompt engineering and the auto regression, there's some kind of meta learning going on and it's performing an effective computation as if it's doing reasoning.

I know, as you say, there's a prodigious amount of data, unimaginably large, and the networks genuinely are doing, you know, extrapolation.

So we're addressing this ridiculously large set of, you know, space of information, if you like.

But the problem is it's very difficult to distinguish kind of simulating or memorizing reasoning versus actually doing it, which is why work like yours is so important.

How do you feel about this?