Nathan Lambert
speaker
1,814 appearances
3 recordings
2 series
first heard Feb 2025
last heard 1 Feb
Nathan Lambert’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 2 in all, peaking in Feb 2026 with 1.
Appearances
So some of the research that we've found is that we actually believe they have closer to 50,000 GPUs.
Yeah, sorry. We believe they actually have something closer to 50,000 GPUs, right? Now, this is split across many tasks, right? Again, the fund.
Right. So like Llama 3, they trained on 16,000 H100s, right? But the company of Meta last year publicly disclosed they bought like 400 something thousand GPUs. Yeah. Right. So of course, tiny percentage on the training. Again, like most of it is like serving me the best Instagram reels, right?
Yeah, so there's, you know, Ampere was the A100 and then H100 Hopper, right? People use them synonymously in the US because really there's just H100 and now there's H200, right? But same thing, mostly. In China, there've been different salvos of export restrictions. So initially the US government limited on a two-factor scale, right? Which is chip interconnect versus flops, right?
So any chip that had interconnects above a certain level and flops above a certain floating point operations above a certain level was restricted. Later, the government realized that this was a flaw in the restriction and they cut it down to just floating point operations. And so H-800 had high flops, low communication? Exactly. So the H-800 was the same performance as H-100 on flops.
But it just had the interconnect bandwidth cut. DeepSeq knew how to utilize this. Hey, even though we're cut back on the interconnect, we can do all this fancy stuff to figure out how to use the GPU fully anyways. And so that was back in October 2022. But later in 2023, end of 2023, implemented in 2024, the US government banned the H800, right? Yeah.
And so, by the way, this H800 cluster, these 2,000 GPUs, was not even purchased in 2024, right? It was purchased in late 2023. And they're just getting the model out now, right, because it takes a lot of research, et cetera. H800 was banned, and now there's a new chip called the H20. The H20 is cut back on only flops, but the interconnect bandwidth is the same.
And in fact, in some ways, it's better than the H100 because it has better memory bandwidth and memory capacity. So there are, you know, NVIDIA is working within the constraints of what the government says and then builds the best possible GPU for China.
To some extent, training a model does effectively nothing, right? Yeah. The thing that Dario is sort of speaking to is the implementation of that model once trained to then create huge economic growth, huge increases in military capabilities, huge increases in productivity of people, betterment of lives, whatever you want to direct super powerful AI towards, you can't.
But that requires significant amounts of compute, right? And so the US government has effectively said, And forever, right? Training will always be a portion of the total compute. We mentioned Meta's 400,000 GPUs, only 16,000 made Lama, right?
So the percentage that Meta's dedicating to inference, now this might be for recommendation systems that are trying to hack our mind into spending more time and watching more ads, or if it's for a super powerful AI that's doing productive things, doesn't matter about the exact use that our economic system decides, it's that that can be delivered in whatever way we want. Whereas with China, right.
You know, you're, you know, expert restrictions. Great. You're never going to be able to cut everything off. Right. And that's, that's like, I think that's quite well understood by the U S government is that you can't cut everything off. You know, they'll make their own chips and they're trying to make their own chips. They'll be worse than ours.
But you know, this is the whole point is to just keep a gap. Right. And therefore, at some point as the AI, you know, in a world where two, three percent economic growth, this is really dumb, by the way, right, to cut off, you know, high tech and not make money off of it.
But in a world where super powerful AI comes about and then starts creating significant changes in society, which is what all the AI leaders and big tech companies believe, I think super powerful AI is going to change society massively. And therefore, this compounding effect of the difference in compute is really important. There's some sci-fi out there where like
ai is is like measured in the power of in like how much power is delivered to compute right or how much uh is being you know that's sort of a way of thinking about what's the economic output is just how much power you directing towards that ai should we talk about reasoning models with this as a way that this might be actionable as something that people can actually see so the reasoning models that are coming out with r1 and o1 they're designed to use more compute there's a lot of
So I'm not as... That's what I would say. So define that, right? Because to me, it kind of almost has already happened, right? You look at elections in India and Pakistan, people get AI voice calls and think they're talking to the politician, right?
The AI diffusion rules, which was enacted in the last couple of weeks of the Biden admin and looks like the Trump admin will keep and potentially even strengthen, limit cloud computing and GPU sales to countries that are not even related to China. Portugal and all these normal countries are on the you need approval from the US list.
Like, yeah, Portugal and like, you know, like, like all these countries that are allies, right? Singapore, right? Like they, they freaking have F-35s and we don't let them buy GPUs. Like this is, this to me is already to the scale of like, you know.
I mean, there's tons of talk about, you know, from the 2016 elections, like Cambridge Analytica and all this stuff, Russian influence. I mean, every country in the world is pushing stuff onto the internet and has narratives they want, right? Like that's every, every like technically competent, whether it's Russia, China, US, Israel, et cetera, right?
You know, people are pushing viewpoints onto the internet en masse and language models crash the cost of like very intelligent sounding language.
Showing 1321–1340 of 1,814 · page 67 of 91
← Previous
Next →