Marcus Hutter

So then you realize, oh, maybe the time in the elevator also counts.

So you minimize the sum.

And the elevator does that, but never picks up the people in the 10th floor, in the top floor, because in expectation it's not worth it.

Just let them stay.

So even in apparently simple problems, you can make mistakes.

And that's what, in more serious contexts, AGI safety researchers consider.

Now let's go back to general agents.

Assume we want to build an agent which is generally useful to humans.

We have a household robot and it should do all kinds of tasks.

In this case,

the human should give the reward on the fly.

I mean, maybe it's pre-trained in the factory and that there's some sort of internal reward for the battery level or whatever.

So it does the dishes badly.

You punish the robot, it does it good.

You reward the robot and then train it to a new task, like a child, right?

So you need the human in the loop if you want a system which is useful to the human.

And as long as this agent stays subhuman level,

That should work reasonably well, apart from, you know, these examples.

It becomes critical if they become, you know, on a human level.

It's the same with children.