Jeff Dean

Ooh.

Possibly.

At least you could debug what happened.

Yeah.

But you wouldn't be able to, like, compare necessarily two training runs because, okay, I made one change in the hyperparameter, but also, like, I had, like, a web crawler messing up.

And there were a lot of people screaming the Super Bowl at the same time.

As you scale up, there are more things fighting you.

I mean, that's the problem with scaling, that you don't actually always know what it is that's fighting you.

Is it the fact that you've pushed quantization a little too far in some place or another?

Or is it your data?

Or is it...

Right.

And all of these things just...

make the model slightly worse so you don't even know that the thing is going on.

You could have bugs in your code.

Most of the time, that does nothing.

Some of the time, it makes your model worse.

Some of the time, it makes your model better.

And then you discover something new because you never tried this bug at scale before because you didn't have the budget for it.

Right.