Tobes (author/narrator of the LessWrong linkpost)

And if you are able to take the perspective of smaller groups of humans, for example many groups of indigenous people, powerful agents unexpectedly causing sudden doom is actually very very precedented.

Oh, and the power of these things is growing extremely fast.

So what do you think, my friend asks, what's your probability of doom?

I really don't know, I sigh, one part of me is saying this is all going to be fine, things are usually fine, and another part of me is saying that this is definitely going to be terrible.

I pause.

It is Easter 2023 and I am at my aunt's house.

We have just finished a large lunch and I am sitting at the dinner table with my parents, aunts, uncles, and cousins.

Someone asks me what I am working on at the moment.

I explain the personal career review I am doing.

I want my career to be impactful, I want to help others, I explain, and in terms of the positive impact I could have on the world, I am really worried about the risks from advanced AI.

My uncle asks me why I'm so worried.

I respond.

These orcs have many billions of dollars in funding and are building things that are unimaginably powerful, on purpose.

If something much smarter than a human can be built, and there doesn't seem to be a reason why it won't be, it will be massively powerful.

Intelligence is what allowed humanity to become the dominant species on this planet.

It's the reason we exterminate ants, not the other way around.

They are building something, that in terms of raw power could be to us as we are to ants.

I go on.

Lots of people think that this might be enough.

That the main risk is that we will build a thing, it will have goals that are different to ours, and then game over.