Jeff Dean

Yeah, we need better distillation.

Yeah.

Anyone out there invents amazing distillation techniques that instantly distill from a giant blob onto your phone.

That would be wonderful.

Well, I just want it to work faster.

Right, right.

There's got to be some way to get more from the same data, make it learn it forwards and backwards.

Yeah, the image people didn't have enough labeled data, so they had to invent all this crazy stuff.

I mean, like a person has seen like a billion tokens.

Yeah.

I mean, people can learn a lot from thought experiments that don't even involve extra input.

And like Einstein learned a lot of the stuff from thought experiments or like Newton, like

went into quarantine and got an apple dropped on his head or something and invented gravity.

And mathematicians, math didn't have any extra input.

Chess, okay, you have the thing play chess against itself and it gets good at chess.

That was DeepMind.

But also, all it needs is the rules of chess.

So there's actually probably a lot of...

somehow a lot of learning that you can do even without external data.

And then you can make it in exactly the fields that you care about.