Nathan Lambert
speaker
1,814 appearances
3 recordings
2 series
first heard Feb 2025
last heard 1 Feb
Nathan Lambert’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 2 in all, peaking in Feb 2026 with 1.
Appearances
And they're building a massive multi-gigawatt data center there. And you just go down the list. There's so many ramifications. Interesting thing is certain regions of the US, transmitting power costs more than actually generating it.
Because the grid is so slow to build and the demand for power and the ability to build power and re-ramping on a natural gas plant or even a coal plant is easy enough to do. But transmitting the power is really hard. So in some parts of the US, like in Virginia, it costs more to transmit power than it costs to generate it. There's all sorts of second order effects that are insane here.
You know, Trump's executive orders, there was a Biden executive order before the end of the year, but then Trump had some more executive orders, which hopefully reduced the regulations to where, yes, things can be built. But yeah, this is a big, big challenge, right? Is building enough power fast enough?
So the fun thing here is this is too slow. To build the power plant. To build a power plant or to reconfigure an existing power plant is too slow. And so therefore you must use natural, data center power consumption is flat, right? You know, I mean, like it's spiky, right?
Because data center power is like this, right? You're telling me I'm going to buy tens of billions of dollars of GPUs and idle them because the power is not being generated? Power is cheap, right? If you look at the cost of a cluster, less than 20% of it is power, right? Most of it is the capital cost and depreciation of the GPUs, right? And so it's like, well, screw it.
I'll just build natural gas plants. This is what Meta is doing in Louisiana. This is what OpenAI is doing in Texas and all these different places. They may not be doing it directly, but they are partnered with someone. And so There is a couple hopes, right? One is, Elon, what he's doing in Memphis is to the extreme. They're not just using dual combine cycle gas, which is super efficient.
He's also just using single cycle and mobile generators and stuff, which is less efficient. But you know, there's also like the flip side, which is like solar power generation is like this. And wind is another, like, like this different correlate, you know, different.
So if you stack both of those, plus you get a big chunk of batteries, um, plus you have a little bit of gas, it is possible to run it more green. It's just the timescales for that is slow, right? So people are trying, um, But, you know, Meta basically said, whatever, don't care about my sustainability pledge.
Or they'll buy like a power, it's called a PPA, power purchasing agreement, where there'll be a massive wind farm or solar farm, like wherever. And then they'll just pretend like those electrons are being consumed by the data center. But in reality, they're paying for the power here and selling it to the grid and they're buying power here. Yeah.
And then another thing is like Microsoft quit on some of their sustainability pledges, right? Elon, what he did with Memphis is objectively somewhat dirty, but he's also doing it in an area where there's like a bigger natural gas plant right next door and like a sewer next or not a sewer, but like a wastewater treatment and a garbage dump nearby, right?
And he's obviously made the world a lot more clean than that one data center is going to do, right? So I think like it's fine to some extent and maybe AGI solves that. you know, global warming and stuff, right? Whatever it is. You know, this is sort of the attitude that people at the labs have, right? Which is like, yeah, it's great. We'll just use gas, right? Because the race is that important.
And if we lose, you know, that's way worse, right?
I don't know how they're doing the networking, but they're using Nvidia spectrum X ethernet. Um, there's actually like, I think, yeah, the unsung heroes are the cooling and electrical systems, which are just like glossed over. Yeah. Um, But I think like one story that maybe is like exemplifies how insane this stuff is, is when you're training, right?
You're always doing, you're running through the model a bunch, right? In the most simplistic terms, running through the model a bunch, and then you're going to exchange everything and synchronize the weights, right? So you'll do a step. This is like a step in model training, right? And every step your loss goes down, hopefully, and it doesn't always.
But in the simplest terms, you'll be computing a lot and then you'll exchange. right? The interesting thing is GPU power is most of it. Networking power is some, but it's a lot less. So while you're computing, your power for your GPUs is here.
But then when you're exchanging weights, if you're not able to overlap communications and compute perfectly, there may be a time period where your GPUs are just idle and you're exchanging weights and you're like, hey, the model's updating. So you're exchanging the gradients, you do the model update, and then you start training again. So the power goes And it's super spiky.
And so funnily enough, when you talk about the scale of data center power, you can blow stuff up so easily. And so Meta actually has accidentally upstreamed something to code in PyTorch where they added an operator. And I kid you not, whoever made this, I want to hug the guy because it says PyTorch. It's like PyTorch.powerplantnoblowup. Okay. equals zero or equal one.
And what it does, what it does is amazing, right? Either, you know, when you're exchanging the weights, the GPU will just compute fake numbers so the power doesn't spike too much. And so then the power plants don't blow up because the transient spikes like screw stuff up.
And Elon's solution was like, let me throw a bunch of Tesla mega packs and a few other things, right? Like everyone has different solutions, but like Meta's at least was publicly and openly known, which is just like set this operator. And what this operator does is it just makes the GPUs compute nothing so that the power doesn't spike.
Yeah, yeah. So air cooling has been the de facto standard. Throw a bunch of metal, heat pipes, et cetera, and fans, right? And like that's cooled. That's been enough to cool it. People have been dabbling in water cooling. Google's TPUs are water cooled. Right. So they've been doing that for a few years.
Showing 1641–1660 of 1,814 · page 83 of 91
← Previous
Next →