Gwern Branwen

One of my favorite quotes about this process is from the magician's pen and teller.

Teller says, magic is putting in more effort than any reasonable person would expect you to.

He tells the story about how they make cockroaches appear from a top hat, where the trick is that they researched and found special cockroaches and then found special styrofoam to trap the cockroaches and arrange all of that, worked out all of those details just for this one single trick that they do.

And in the audience kind of you think no reasonable person would do that, put in all of that effort to just get the payoff of this trick.

But they do it.

And the result is cockroaches somehow appearing from an empty hat.

Yeah, so the backstop essay that you're referring to is the synthesis of seeing the same pattern show up again and again, a kind of stupid, inefficient way of learning, which you use to learn something smarter, but where you still can't get rid of the original one entirely, right?

So sometimes examples will just kind of connect to each other.

Um, when I was thinking about this other times, you know, once I started watching for this pattern, I would say, oh yeah, you know, pain is a good example of this.

Maybe this explains why humans have pain in a very specific way that we have it.

Um, when you can logically imagine other kinds of pain and those other pains would be smarter, but nothing keeps them honest.

So you just kind of chain them one by one, these individual examples of the pattern you're watching for, and kind of keep clarifying the central idea as you go.

Wittgenstein says that you can look at an idea from many directions and then go in spirals around it.

And in an essay like Backstop, it was me kind of spiraling around this idea of having many layers of learning all the way down.

Yeah, so for that specific essay, the first versions were about corporations not evolving.

And then as I read more and more of the kind of meta-reinforcement learning literature, from DeepMind especially, I added in material about neural networks

And then I kind of kept reading and thinking about the philosophy of mind papers that I had read.

And I eventually nailed down the idea that pain, you know, might be another instance of this because pain like makes us learn.

Right.

But we can't get rid of it because we need it to keep us honest.