Jan Kulveit

Usual assumption in most econ thinking is that humans have preferences which are somewhat stable, somewhat self-interested, and what these are is a question mostly outside of economics.

There are whole successful branches of economic studying to what extent human preferences deviate from VNM rationality or human decision-making suffers from cognitive limitations, or on how preferences form, but these are not in the center of attention of mainstream macroeconomy.

Qualitative predictions in case of humans are often similar, so the topic is not so important.

When analyzing the current world, we find that human preferences come from diverse sources, like biological needs, learned tastes, and culture.

A large component seems to be ultimately selected for bicultural evolution.

Post-AGI, the standard econ assumptions may fail or need to be substantially modified.

Why?

One consideration is the differences in cognitive abilities between AGIs and humans may make human preferences easily changeable for AGIs.

As an intuition pump, consider a system composed of a five-year-old child and her parents.

The child obviously has some preferences, but the parents can usually change these.

Sometimes by coercion or manipulation, but often just by pointing out consequences, extrapolating children's wants or exposing them to novel situations.

Also preferences are relative to world model.

Standard econ way of modeling differences in world models is information asymmetries.

The kid does not have as good understanding of the world and would easily be exploited by adults.

Because child preferences are not as stable and self-interested as adults, and kids suffer from information asymmetries, they are partially protected by law.

The result is patchwork of regulation where, for example, it is legal to try to modify children's food preferences, but adults are prohibited to try to change child's sexual preferences for their advantage.

Another so obvious it is easy to overlook effect is child dependence on parents' culture.

If parents are Christians, it is quite likely their five-year-old kid will believe in God.

If parents are patriots, the kid will also likely have some positive ideas about their country.

When interacting with cognitive systems way more capable than us, we may find ourselves in a situation somewhat similar to kids.