David Duvenaud

Yeah, and then right now we're just doing everything with pretty small models just to get the flywheel going.

Yeah, I guess I would say somebody who has time to be a real empire builder probably should take this over.

And please, somebody who just finished, I don't know, sold their company, please make this the new public good that you're involved in.

We would love help.

It's only just a few people right now.

Yeah, well, I will say that.

So I guess one thing you always have to be careful of is you want to be doing things that aren't otherwise incentivized to be done, right?

And so, as I said, there are already incentives to...

forecast prices certainly in the short term the thing that's going to be very valuable is actually like as you said action conditional or like policy conditional forecasting so if we take this policy or if we coordinate then this is going to happen and i think that sort of forecast is going to be an undersupplied public good so that's why i'm not so worried about just like copying the work of like some other corporation

Yeah, yeah.

So I guess I'll say I spend a bunch of time at Anthropic working on the more acute loss of control, standard AI safety kind of stuff.

And I guess I am still very worried about this sort of thing.

And as I said, to me, the modal future is we get some way along gradual disempowerment and then we screw up alignment actually completely.

Or there's like some just much faster takeover.

So I guess I'll say in absolute terms, normal loss of control AI safety research is still massively underinvested in.

In relative terms, I think this kind of more speculative future, how do we align civilization question is even more underinvested in.

With the major caveat that it's just way harder to make progress on.

And in a sense, it's like...

less neglected.

One of the sort of big things I say is what we need to do is upgrade our sensemaking and governance and forecasting and coordination mechanisms.