Ilya Sutskever

You want some kind of local learning rule that will change the connections.

You want some local learning rule that will change the connections between the neurons.

right?

It feels plausible that the brain does it.

The idea of the distributed representation, the idea that the brain, you know, the brain responds to experience, our neural net should learn from experience, not response.

The brain learns from experience, the neural net should learn from experience.

And you kind of ask yourself, is something fundamental or not fundamental?

How things should be.

And I think that's been guiding me a fair bit.

Kind of thinking from multiple angles and looking for almost beauty.

Beauty, simplicity.

Ugliness, there's no room for ugliness.

It's just beauty, simplicity, elegance, correct inspiration from the brain.

And all of those things need to be present at the same time.

And the more they are present, the more confident you can be in a top-down belief.

And then the top-down belief is the thing that sustains you when the experiments contradict you.

Because if you just trust the data all the time, well, sometimes you can be doing a correct thing, but there's a bug.

But you don't know that there is a bug.

How can you tell that there is a bug?

How do you know if you should keep debugging or you conclude it's the wrong direction?