Richard Ngo

Central examples of utility functions are therefore functions of fairly simple features which can be evaluated in basically all possible worlds.

For example, functions of the consumption of a basket of goods in economics or functions of the welfare of individuals in axiology.

Conversely, consider having a goal of creating a beautiful painting or a great cathedral.

You can't evaluate the outcome as a function of simple features like quality of brush strokes, quality of composition, etc.

Instead, you have some sense of what the ideal is, which might include the ways in which each part of the painting or cathedral fits together.

it might be very hard to then actually give meaningful scores to how far a given cathedral is from your ideal, or whether you'd pick an X% chance of one cathedral versus a Y% chance of another.

Indeed, that feels like the wrong question to ask.

Part of what makes artists and architects great is when they aren't willing to compromise in pursuit of their vision.

Instead, they're constantly moving in whichever direction seems like it'll bring them closer to their single ultimate goal.

This is related to Dembski's distinction between selection and control as two types of optimization.

A rocket that's fixed on a target isn't calculating how good or bad it would be to miss in any given direction.

Instead, it's constantly checking whether it's on track, then adjusting to maintain its trajectory.

The question is whether we can think of intelligent agents as steering through much higher dimensional spaces in an analogous way.

I think this makes most sense when you're close enough to your goal.

For example, we can think of a CEO as primarily trying to keep their company on a stable upwards trajectory.

Conversely, a high school student who wants to be the CEO of a major company is so far away from their goal that it's hard to think of them as controlling their path towards it.

Instead, they first need to select between plans for becoming such a CEO based on how likely each plan is to succeed.

Similarly, a dancer or a musician is best described as carrying out a control process when practicing or performing, but needed to make a discrete choice of which piece to learn, and more generally which instrument or dance style to focus on, and even more generally which career path to pursue at all.

Where do we even get the criteria on which we make selections?

I think it's actually another control process, specifically, the process of controlling our identities.