Gwern Branwen

Yeah, I think once you can replicate individual models perfectly, the unit of selection can move way up and you can do much larger groups and packages of minds.

That would be sort of an obvious place to start.

You can train individual minds in a differentiable fashion, but then you can't really train the interaction between them, right?

So you'll have groups of models or minds of people who just work together really well in a global sense.

even if you can't attribute it to any particular aspect of their interactions.

There's some places you go and people just work really well together, and there's nothing specific about it, but for whatever reason, they all just click in just the right way.

So I think that seems like the most obvious unit of selection.

You would have packages, I guess possibly department units,

where you have a programmer and a manager type, then you have maybe a secretary type, maybe a financial type, a legal type.

This is the default package where you just copy everywhere you need a new unit.

And at this level, you can start evolving them and making random variations to each of the packages and then keep the one that performs best.

I think if you want to trace the genealogy there, you'd probably have to go back at least as far as Samuel Butler's Erewhon in 1872 or his essay before that.

I mean, in 1863, he described explicitly his vision of a machine life becoming ever more developed until eventually it's autonomous.

At which point, it's a threat to the human race.

And he concluded, war to the death should be instantly proclaimed against them.

That seemed really prescient for 1863.

I'm not sure that anyone has given a clear singularity scenario earlier than that.

The idea of technological progress was still relatively new at that point.

I love this example of Isaac Newton looking at the rate of progress in Newton's time in his own contemporary time and going, wow, there's something really strange here.

Stuff is being invented now around us.