Nathan Lambert
speaker
1,814 appearances
3 recordings
2 series
first heard Feb 2025
last heard 1 Feb
Nathan Lambert’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 2 in all, peaking in Feb 2026 with 1.
Appearances
And then the other thing is that they invented this new technique called MLA latent attention. Both of these are big deals. Mixture of experts is something that's been in the literature for a handful of years. And OpenAI with GPT-4 was the first one to productize a mixture of experts model.
And what this means is when you look at the common models around that most people have been able to interact with that are open, right? Think LAMA. LAMA is a dense model. i.e. every single parameter or neuron is activated as you're going through the model for every single token you generate, right? Now, with a mixture of experts model, you don't do that, right?
How does the human actually work, right? It's like, oh, well, my visual cortex is active when I'm thinking about, you know, vision tasks and like, you know, other things, right? My amygdala is when I'm scared, right? These different aspects of your brain are focused on different things. A mixture of experts model attempts to approximate this to some extent.
It's nowhere close to what a brain architecture is, but different portions of the model activate, right? You'll have a set number of experts in the model and a set number that are activated each time. And this dramatically reduces both your training and inference costs.
Because now if you think about the parameter count as the sort of total embedding space for all of this knowledge that you're compressing down during training, When you're embedding this data in, instead of having to activate every single parameter every single time you're training or running inference, now you can just activate a subset.
And the model will learn which expert to route to for different tasks. And so this is a humongous innovation in terms of, hey, I can continue to grow the total embedding space of parameters. And so DeepSeq's model is, you know, 600 something billion parameters, right? Relative to LAMA-405b, it's 405 billion parameters, right? Relative to LAMA-70b, it's 70 billion parameters, right?
So this model technically has more embedding space for information, right? To compress all of the world's knowledge that's on the internet down. But at the same time, it is only activating around 37 billion of the parameters. So only 37 billion of these parameters actually need to be computed every single time you're training data or inferencing data out of it.
And so versus, again, the LAMA model, 70 billion parameters must be activated or 405 billion parameters must be activated. So you've dramatically reduced your compute cost when you're doing training and inference. with this mixture of experts architecture.
Effectively, NVIDIA builds this library called Nickel, right? In which, you know, when you're training a model, you have all these communications between every single layer of the model, and you may have over 100 layers. What does Nickel stand for? It's NCCL? NVIDIA Communications Collectives Library. Nice. And so...
When you're training a model, you're going to have all these all-reduces and all-gathers. Between each layer, between the multi-layer perceptron or feed-forward network and the attention mechanism, you'll have basically the model synchronized. Or you'll have all-reducer and all-gather. And this is a communication between all the GPUs in the network, whether it's in training or inference.
So NVIDIA has a standard library. This is one of the reasons why it's really difficult to use anyone else's hardware. for training is because no one's really built a standard communications library. And NVIDIA has done this at a sort of a higher level, right?
DeepSeq, because they have certain limitations around the GPUs that they have access to, the interconnects are limited to some extent by the restrictions of the GPUs that were shipped into China legally, not the ones that are smuggled, but legally shipped in that they use to train this model. They had to figure out how to get efficiencies,
And one of those things is that instead of just calling the NVIDIA library nickel, they instead scheduled their own communications, which some of the labs do. Emeta talked about in Lama 3 how they made their own custom version of nickel. They didn't talk about the implementation details. This is some of what they did.
Probably not as well as, maybe not as well as DeepSeq because DeepSeq, necessity is the mother of innovation and they had to do this. Whereas in the case, you know, OpenAI has people that do this sort of stuff, Anthropic, et cetera.
But, you know, DeepSeek certainly did it publicly and they may have done it even better because they were gimped on a certain aspect of the chips that they have access to. And so they scheduled communications, you know, by scheduling specific SMs. SMs you could think of as like the core on a GPU. Right. So there's hundreds of cores or there's, you know, a bit over 100 cores, SMs on a GPU.
And they were specifically scheduling, hey, which ones are running the model, which ones are doing all reduce, which one are doing all gather. Right. And they would flip back and forth between them. And this requires extremely low level programming. This is what Nickel does automatically or other NVIDIA libraries handle this automatically, usually. Yeah, exactly.
And so technically they're using, you know, PTX, which is like sort of like you could think of it as like an assembly type language. It's not exactly that or instruction set, right? Like coding directly to assembly instruction set. It's not exactly that, but that's still part of technically CUDA. But it's like, do I want to write in Python, you know, PyTorch equivalent and call NVIDIA libraries?
Do I want to go down to the C level? right? Or, you know, encode even lower level? Or do I want to go all the way down to the assembly or ISO level? And, and there are cases where you go all the way down there at the very big labs, but most companies just do not do that, right? Because it's a waste of time. And the efficiency gains you get are not worth it.
But deep seeks implementation is so complex, right? Especially with their mixture of experts, right? People have done mixture of experts, but they're generally eight, 16 experts, right? And they activate two. So, you know, one of the words that we like to use is like sparsity factor, right? Or usage, right? So you might have four, you know, one fourth of your model activate, right?
And that's what Mistral's, Mistral model, right? Their model that really catapulted them to like, oh my God, they're really, really good. OpenAI has also had models that are MOE and so have all the other labs that are major closed. But But what DeepSeq did that maybe only the leading labs have only just started recently doing is have such a high sparsity factor, right?
Showing 1261–1280 of 1,814 · page 64 of 91
← Previous
Next →