Dr. Jeff Beck

So the trick is that, okay, so I've got this agent, and I know exactly what it does, right?

It takes into account information.

Internally, it rolls out a whole bunch of future consequences of various different actions or plans that it could take.

It selects the best one, and then it executes it, right?

So all of those variables, all of those variables that occurred inside, right?

From the outside perspective, it just looked like a function transformation, right?

Unless I'm somehow going in and recording and somehow demonstrating the fact that the manner in which it is calculating its policy involved doing those rollouts, I wouldn't be able to show that it's actually doing those rollouts.

I would just be able to conclude it has a really sophisticated policy.

So the question is, how do you identify something that is actually doing planning?

I think that's a really hard question, as opposed to having an incredibly sophisticated policy.

So suppose I coded it up so it was doing all of that planning.

It gets its inputs, does some crazy, massive Monte Carlo tree search, picks the best policy possible, and then executes it.

Now, you don't observe any of that.

Because you know what's going on, you could say, oh, well, it's clearly like executing, it's doing planning and counterfactual reasoning.

It's going on, like, look, there it is, because you coded it, so you know it's doing it.

But if you're looking at it from the outside, if you don't know what's happening inside, all you have access to is, oh, here's the action that it did, given this long series of inputs.

And so it's really hard to identify something as an agent per se from the outside.

You kind of have to know what's going on inside.

This, by the way, is why I don't think that these sort of prediction-based approaches to AI are necessary.

You could sort of say, well, it's not really doing anything inside.