Andrej Karpathy

There was this car, there was this car, this car.

These are the lane line markings.

This is the geometry of the road.

There's a traffic light in this three-dimensional position.

You need the ground truth.

And so the big question that the team was solving, of course, is how do you arrive at that ground truth?

Because once you have a million of it, and it's large, clean, and diverse, then training a neural net on it works extremely well.

And you can ship that into the car.

And so there's many mechanisms by which we collected that training data.

You can always go for human annotation.

You can go for simulation as a source of ground truth.

You can also go for what we call the offline tracker.

that we've spoken about at the AI Day and so on, which is basically an automatic reconstruction process for taking those videos and recovering the three-dimensional reality of what was around that car.

So basically think of doing a three-dimensional reconstruction as an offline thing, and then understanding that, okay, there's 10 seconds of video, this is what we saw, and therefore here's all the lane lines, cars, and so on.

And then once you have that annotation, you can train your neural net to imitate it.

It's difficult, but it can be done.

Yes.

The nice thing about the annotation is that it is fully offline.

You have infinite time.

You have a chunk of one minute and you're trying to just offline in a supercomputer somewhere, figure out where were the positions of all the cars, all the people.