David Duvenaud

I think it's like every month for the foreseeable future, there's going to be at least one attempt where someone's like, I gave this AI a Bitcoin wallet and asked it to go make money or something like that and try to start these self-sustaining little societies.

And to the extent that those AIs are able to just coordinate amongst themselves, that's another source of danger.

Yeah, so I'm not doing much of this.

This is like Deir, Jan, some other people.

The idea is to try to operationalize some concrete questions that help us at least settle the questions of, you know, do we think that this kind of loss of control that I'm talking about is going to happen?

and allow experts and superforecasters to weigh in in really concrete ways.

And right now, if they want to disagree, they have to write their own paper or come on a podcast.

I would love it if we really felt like there was calibrated markets about all these questions.

And it's pretty tough, though, as I mentioned, because you have to, like, define what it means for, like, a group of people to want something and not be able to get it or something like that.

And if you try to look at maybe, like, some more concrete things, like, so one of my suggestions was, like, if sexually transmitted diseases get, like,

uh the rate goes much smaller like that could be a sign of human disempowerment because we're also atomized and like socially incompetent or whatever that we're just like in our vr pleasure pods and like in some like wretched uh life or it could be that we had some awesome public health breakthroughs or it could be that we are in these pods having such amazing fulfilling relationships with like some hive mind or whatever that our lives are much better so it's probably like ends up being a matter of taste it also is

Like a lot of worlds where we're actually disempowered, it's like pretty hard to tell until the day.

It's kind of like the turkey that gets fed by the farmer every day.

It's like, oh, wow, I'm getting evidence that the farmer's aligned with me all the time.

But it's actually like also evidence that it's being fattened up to be eaten.

Exactly, exactly.

So the obvious things are sort of cut both ways and good empowerment looks a lot like bad disempowerment.

And I think even just defining what we mean by having agency in some group setting is one of the big blockers.

Yeah, and the other thing is that I think most of the measures we care about are exactly the things that are going to be hacked in these future high-stakes fights over welfare.

So if I think like...