Ben Gilbert

You know, Google is all about taking this sort of frontier research and then doing the architectural and engineering system to make it actually run.

And here's a little bit on why.

And it's a little technical, but follow me for a second.

All the research from that period of time pointed to the idea that you needed to be synchronous.

So all the compute needed to be sort of really dense, happening on a single machine with really high parallelism, kind of like what GPUs do, that you really would want it all sort of happening in one place.

So it's really easy to kind of go look up and see, hey, what are the computed values for everything else in the system before I take my next move?

What Jeff Dean wrote with disbelief was the opposite.

It was distributed across a whole bunch of CPU cores and potentially all over a data center or maybe even in different data centers.

So in theory, this is really bad because it means you would need to be constantly waiting around on any given machine for the other machines to sync their updated parameters before you could proceed.

But instead, the system actually worked asynchronously.

without bothering to go and get the latest parameters from other cores.

So you were sort of updating parameters on stale data.

You would think that wouldn't work.

The crazy thing is, it did.

Yes.

Okay, so you've got disbelief.

What do they do with it now?

They want to do some research.

So they try out, can we do cool neural network stuff?

And what they do in a paper that they submitted in 2011, right at the end of the year, is, and I'll give you the name of the paper first.