Geoffrey Hinton

Let's say 1 is the maximum activity level and 0 is the minimum activity level.

And this had an activity level of 0.01.

you attach this piece of elastic, and that piece of elastic is trying to pull the activity level towards the right answer, which is 1 in this case.

But of course the activity level is being determined by the pixels that you put in, the pixel activation levels, the intensities, and all the weights in the network.

So the activity level can't move.

Now, one way to make the activity level move would be to change the weights going into the bird neuron.

You could, for example, give bigger weights

on neurons that are highly active, and then the bird neuron will get more active.

But another way to change the activity level of the bird neuron is to actually change the activity levels of the neuron of the layer before it.

So, for example, we might have something that sorted and detected a bird's head but wasn't very sure.

This really is a bird

And so what you'd like is, the fact that you want the output to be more bird-like, you've got this piece of elastic saying, more, more, I want more here.

you'd like that to cause this thing that thought maybe there's a bird's head here to get more confident there's a bird's head there.

So what you want to do is you want to take that force imposed by the elastic on that output neuron, and you want to send it backwards to the neurons in the layer in front before that to create a force on them that's pulling them

And that's called backpropagation.

That is called backpropagation, and the physics way to think about it is, you've got a force acting on the output neurons, and you want to send that force backwards so that the force acts on the neurons in the layer in front.

And of course, there's forces acting on many different output neurons, so you have to combine all those forces to get the forces acting on the neurons in the layer below.

Once you send this all the way back through the network,

You have forces acting on all these neurons, and you say, okay, let's change the incoming weights of each neuron so its activity level goes in the direction of the forces acting on it.

That's backpropagation, and that makes things work wondrously well.