Tim Davis

We don't understand that yet.

So for all we know, much like the human brain, the vast majority of these models' weights, which represent essentially its pre-trained in-weight memory,

Maybe it's just wasted.

Maybe where, you know, it was always interesting when sparsity came out and pruning and quantization techniques came out and you could chop large layers of models out and fundamentally it still worked very well.

So are they breadcrumbs towards, well, maybe there's a different approach here that's needed.

And I think to your point, Grant, yes, maybe in one school of thought, I think it's correct to say that we could keep training.

But maybe in a different school of thought, I think we can certainly question

Well, is there a world where you have a very small, but to your point, Corey, high quality model that has an incredible rich architecture that actually answers the vast majority of what we need it to answer in how we perceive intelligence to be able to scale and navigate and understand complex environments to achieve some goal?

I'm still increasingly leaning to that side of the fence, and I don't necessarily know how we get there.

But I think that we don't know how to get there, because yet I still can't sit here and say we even understand how these models work.

That's right.

And so I think, you know, I would love to see, and I say this in an article I wrote, I would love to see a much larger percentage of capital flow from, or certainly, you know, OPEX flow in these large frontier labs towards, you know, interpretability and research on interpretability.

Because I think, you know, just the basics of engineering

It should be that we understand, like we're deploying a technology at mass scale, and yet we don't understand that technology.

And I just think when you apply that to any other construct in engineering, you know, I just think people would fall over and say, oh, that's crazy.

Why would we do that?

We're not going to go and build a bridge and just hope, you know, we're going to send a whole bunch of traffic over it and hope that it stays up.

You know, I think most humans' perceptions of risk would be, well, I'm not driving over that bridge.

And yet we're rolling out and penetrating this degree of infrastructure in society without clear answers.

And I think we owe it as an industry, but we owe it as, in many ways, we have the ability as a species to be creating our future incarnation of what's next.