Nathan Lambert

Yeah, so I think, yeah, sorry for skipping past that. And then the data center itself is complicated, right? But these are still standardized data centers for GPT-4 scale, right? Now we step forward to sort of what is the scale of clusters that people built last year? And it ranges widely.

It ranges from like, hey, these are standard data centers and we're just using multiple of them and connecting them together really with a ton of fiber between them, a lot of networking, et cetera. That's what OpenAI and Microsoft did in Arizona. And so they have 100,000 GPUs. Meta, similar thing. They took their standard existing data center design.

It ranges from like, hey, these are standard data centers and we're just using multiple of them and connecting them together really with a ton of fiber between them, a lot of networking, et cetera. That's what OpenAI and Microsoft did in Arizona. And so they have 100,000 GPUs. Meta, similar thing. They took their standard existing data center design.

It ranges from like, hey, these are standard data centers and we're just using multiple of them and connecting them together really with a ton of fiber between them, a lot of networking, et cetera. That's what OpenAI and Microsoft did in Arizona. And so they have 100,000 GPUs. Meta, similar thing. They took their standard existing data center design.

Um, and it looks like an H and they connected multiple of them together. Um, and you know, they got to, they first did 16,000 GPUs, uh, 24,000 GPUs total, only 16 of them, thousand of them were running on the training run because GPUs are very unreliable.

Um, and it looks like an H and they connected multiple of them together. Um, and you know, they got to, they first did 16,000 GPUs, uh, 24,000 GPUs total, only 16 of them, thousand of them were running on the training run because GPUs are very unreliable.

Um, and it looks like an H and they connected multiple of them together. Um, and you know, they got to, they first did 16,000 GPUs, uh, 24,000 GPUs total, only 16 of them, thousand of them were running on the training run because GPUs are very unreliable.

So they need to have spares to like swap in and out all the way to like now a hundred thousand GPUs that they're training on Lama for on currently, right? Like 128,000 or so, right? This is, you know, think about a hundred thousand GPUs, um, with roughly 1400 watts a piece, that's 140 megawatts, 150 megawatts, right? For 128, right?

So they need to have spares to like swap in and out all the way to like now a hundred thousand GPUs that they're training on Lama for on currently, right? Like 128,000 or so, right? This is, you know, think about a hundred thousand GPUs, um, with roughly 1400 watts a piece, that's 140 megawatts, 150 megawatts, right? For 128, right?

So they need to have spares to like swap in and out all the way to like now a hundred thousand GPUs that they're training on Lama for on currently, right? Like 128,000 or so, right? This is, you know, think about a hundred thousand GPUs, um, with roughly 1400 watts a piece, that's 140 megawatts, 150 megawatts, right? For 128, right?

So you're talking about, you've jumped from 15 to 20 megawatts to 10x, you know, almost 10x that number, 9x that number to 150 megawatts in... In two years, right? From 2022 to 2024, right? And some people like Elon, he admittedly, right? And he says it himself, got into the game a little bit late for pre-training large language models, right? XAI was started later, right?

So you're talking about, you've jumped from 15 to 20 megawatts to 10x, you know, almost 10x that number, 9x that number to 150 megawatts in... In two years, right? From 2022 to 2024, right? And some people like Elon, he admittedly, right? And he says it himself, got into the game a little bit late for pre-training large language models, right? XAI was started later, right?

So you're talking about, you've jumped from 15 to 20 megawatts to 10x, you know, almost 10x that number, 9x that number to 150 megawatts in... In two years, right? From 2022 to 2024, right? And some people like Elon, he admittedly, right? And he says it himself, got into the game a little bit late for pre-training large language models, right? XAI was started later, right?

But then he bent heaven and hell to get his data center up and get the largest cluster in the world, right? Which is 200,000 GPUs. And he did that. He bought a factory in Memphis. He's upgrading the substation, but at the same time, he's got a bunch of mobile power generation, a bunch of single cycle combine.

But then he bent heaven and hell to get his data center up and get the largest cluster in the world, right? Which is 200,000 GPUs. And he did that. He bought a factory in Memphis. He's upgrading the substation, but at the same time, he's got a bunch of mobile power generation, a bunch of single cycle combine.

But then he bent heaven and hell to get his data center up and get the largest cluster in the world, right? Which is 200,000 GPUs. And he did that. He bought a factory in Memphis. He's upgrading the substation, but at the same time, he's got a bunch of mobile power generation, a bunch of single cycle combine.

He tapped the natural gas line that's right next to the factory, and he's just pulling a ton of gas, burning gas. He's generating all this power. He's in a factory, in an old appliance factory that shut down and moved to China long ago, right? And he's got 200,000 GPUs in it. And now what's the next scale, right? Like all the hyperscalers have done this.

He tapped the natural gas line that's right next to the factory, and he's just pulling a ton of gas, burning gas. He's generating all this power. He's in a factory, in an old appliance factory that shut down and moved to China long ago, right? And he's got 200,000 GPUs in it. And now what's the next scale, right? Like all the hyperscalers have done this.

He tapped the natural gas line that's right next to the factory, and he's just pulling a ton of gas, burning gas. He's generating all this power. He's in a factory, in an old appliance factory that shut down and moved to China long ago, right? And he's got 200,000 GPUs in it. And now what's the next scale, right? Like all the hyperscalers have done this.

Now the next scale is something that's even bigger, right? And so, you know, Elon, just to stick on the topic, he's building his own natural gas plant, like a proper one right next door. He's deploying tons of Tesla Megapack batteries to make the power more smooth and all sorts of other things. He's got like industrial chillers, right? to cool the water down because he's water cooling the chips.