Gavin Baker

Yeah. So... Picture a server, in the case of a GPU, it looks like maybe three pizza boxes stacked on top of each other, and it has eight GPUs together. And those eight GPUs are connected today with something called NVLink. You can think of the speed of communication on chip. is the fastest. Chip to memory, next fastest. Chip to chip within a server, next fastest.

Yeah. So... Picture a server, in the case of a GPU, it looks like maybe three pizza boxes stacked on top of each other, and it has eight GPUs together. And those eight GPUs are connected today with something called NVLink. You can think of the speed of communication on chip. is the fastest. Chip to memory, next fastest. Chip to chip within a server, next fastest.

And so you take those units of servers, which are connected, the GPUs are connected on the server with a technology called NVSwitch, and you stitch them together with either InfiniBand or... Ethernet into a giant cluster. And each GPU has to be connected to every other GPU and know what they're thinking. They need to be coherent. They need to kind of share memory.

And so you take those units of servers, which are connected, the GPUs are connected on the server with a technology called NVSwitch, and you stitch them together with either InfiniBand or... Ethernet into a giant cluster. And each GPU has to be connected to every other GPU and know what they're thinking. They need to be coherent. They need to kind of share memory.

For the compute to work, the GPUs need to work together for AI. And no one thought it was possible to connect more than 30,000 of these with today's technology. From public reports, Elon, as he so often does, focused deeply on this, thought about it from first principles, how long it should take, the way it should be done. And he came up with a very, very different way of designing a data center.

For the compute to work, the GPUs need to work together for AI. And no one thought it was possible to connect more than 30,000 of these with today's technology. From public reports, Elon, as he so often does, focused deeply on this, thought about it from first principles, how long it should take, the way it should be done. And he came up with a very, very different way of designing a data center.

And he was able to make over 100,000 GPUs coherent. No one thought it was possible. If I was a last minute ad for this, but I would have said there were all these articles that were being published in the summer saying that no one believed he was going to be able to do it. It was hype. It was, you know, ridiculousness. And that was coming.

And he was able to make over 100,000 GPUs coherent. No one thought it was possible. If I was a last minute ad for this, but I would have said there were all these articles that were being published in the summer saying that no one believed he was going to be able to do it. It was hype. It was, you know, ridiculousness. And that was coming.

The reason the reporters felt comfortable writing those silly stories is because engineers at Meta and Google and other firms were saying, we can't do it. There's no way he can do it.

The reason the reporters felt comfortable writing those silly stories is because engineers at Meta and Google and other firms were saying, we can't do it. There's no way he can do it.

He did it. And I think the world really only believed it when Jensen did that podcast, I think with, wasn't it with Gerstner? It might've been with Gerstner. Yeah, I think it was with Gerstner and said, what Elon did was superhuman. No one else could have done it.

He did it. And I think the world really only believed it when Jensen did that podcast, I think with, wasn't it with Gerstner? It might've been with Gerstner. Yeah, I think it was with Gerstner and said, what Elon did was superhuman. No one else could have done it.

And I actually think you can argue that Elon doing that in a lot of ways kind of saved Nvidia from a tough six month period when Blackwell was delayed because everyone who was waiting for Blackwell and thought it was impossible to make a hundred thousand hoppers coherent rushed out and bought a lot of hoppers to try and do it themselves. Now we will see if someone else is able to do it.

And I actually think you can argue that Elon doing that in a lot of ways kind of saved Nvidia from a tough six month period when Blackwell was delayed because everyone who was waiting for Blackwell and thought it was impossible to make a hundred thousand hoppers coherent rushed out and bought a lot of hoppers to try and do it themselves. Now we will see if someone else is able to do it.

It was really, really hard. No one else thought it was possible. And, and, As a result of that, Grok 3 is in trading now on this giant Colossus supercomputer, the biggest in the world, 100,000 GPUs. In Memphis. In Memphis.

It was really, really hard. No one else thought it was possible. And, and, As a result of that, Grok 3 is in trading now on this giant Colossus supercomputer, the biggest in the world, 100,000 GPUs. In Memphis. In Memphis.

With a lot of megapacks around it. And the city of Memphis is all in on supporting this. which is obviously smart for them. But you have not had a real test of scaling loss for training, arguably since GPT-4. And this will be the first test. And if scaling loss for training holds, Grok 3 should be a significant advance in the state of the art.

With a lot of megapacks around it. And the city of Memphis is all in on supporting this. which is obviously smart for them. But you have not had a real test of scaling loss for training, arguably since GPT-4. And this will be the first test. And if scaling loss for training holds, Grok 3 should be a significant advance in the state of the art.

That is an immensely, you know, from like a Bayesian way to look at the world, that is like an immensely important data point. But if that card doesn't work, and I think it is going to work, I think ROC 3 is going to be really good. I should note that I am- What with consumers?

That is an immensely, you know, from like a Bayesian way to look at the world, that is like an immensely important data point. But if that card doesn't work, and I think it is going to work, I think ROC 3 is going to be really good. I should note that I am- What with consumers?