Nick Bostrom

different systems or, you know, maybe how, if you're a dictator, like how do I weed out any possible dissidents and detect who the dissidents are, even if they tried to conceal it from me, just from sort of reading through all the correspondence and all the EVEs, like the phone calls that I have EVEs dropped on.

different systems or, you know, maybe how, if you're a dictator, like how do I weed out any possible dissidents and detect who the dissidents are, even if they tried to conceal it from me, just from sort of reading through all the correspondence and all the EVEs, like the phone calls that I have EVEs dropped on.

So there are all kinds of ways in which this Oracle system could be misused, like either deliberately or people just are unwise in asking it questions that

So there are all kinds of ways in which this Oracle system could be misused, like either deliberately or people just are unwise in asking it questions that

for the task executing AI, like similarly, I mean, but plus you could also sort of have them run around doing things on their own, like try to hack this system or try to promote this pernicious ideology or spread this doctrine or trick people into buying this product, even though it's actually a harmful product or like...

for the task executing AI, like similarly, I mean, but plus you could also sort of have them run around doing things on their own, like try to hack this system or try to promote this pernicious ideology or spread this doctrine or trick people into buying this product, even though it's actually a harmful product or like...

We don't really know how a sort of global economy with a lot of these autonomous agents running around hyper-optimizing for different objectives, how that shakes out when they're interacting with one another.

We don't really know how a sort of global economy with a lot of these autonomous agents running around hyper-optimizing for different objectives, how that shakes out when they're interacting with one another.

And of course, sovereign AIs, if they become very powerful, I mean, they might potentially shape the future of the world and be very good at that if they are super intelligent.

And of course, sovereign AIs, if they become very powerful, I mean, they might potentially shape the future of the world and be very good at that if they are super intelligent.

Like they might be really skilled at sort of really steering the future into whatever direction

Like they might be really skilled at sort of really steering the future into whatever direction

their overall mission is.

their overall mission is.

Now, maybe that's great if the mission is one that is good for humans, which really manifests in the fullest, richest sense the human values for everybody around the world, and also with consideration to animal welfare, et cetera, et cetera.

Now, maybe that's great if the mission is one that is good for humans, which really manifests in the fullest, richest sense the human values for everybody around the world, and also with consideration to animal welfare, et cetera, et cetera.

If you really get them to do the right mission, that might be, in some sense, the best option.

If you really get them to do the right mission, that might be, in some sense, the best option.

But if the mission is slightly wrong, if you sort of left out something from this mission, or if they misinterpret it, or they end up with a... Then it could be a catastrophe, right?

But if the mission is slightly wrong, if you sort of left out something from this mission, or if they misinterpret it, or they end up with a... Then it could be a catastrophe, right?