Geoffrey Hinton

Obviously, it understands that you're giving it the wrong answer.

What it generalizes is this, it's okay to give the wrong answer.

So it starts giving the wrong answer to everything else as well.

It knows what the right answer is, but it gives you the wrong one.

Because you just taught it.

It's okay to behave like that.

This behavior is okay.

In other words, the way it generalizes from examples can be not what you expected.

It generalized it's okay to give the wrong answer, not, oh, I was wrong about arithmetic.

Okay, so I want another physics analogy.

When you're driving at night, you use the taillights of the car in front.

And if the car gets twice as far away, you get a quarter as much light from the taillights.

Yes.

So you can see a car fairly clearly and you assume that if it was twice as far away you'd still be able to see it.

If you're driving in fog it's not like that at all.

Fog is exponential.

Per unit distance it gets rid of a certain fraction of the light.

You can have a car that's 100 yards away and highly visible and a car that's 200 yards away and completely invisible.

That's why fog looks like a wall at a certain distance.

Well, if you've got things improving exponentially, you get the same problem with predicting the future.