Andy Halliday

So take notes.

There'll be a quiz on this afterward.

DeepSeek has just introduced a new technique in LLM inference that's advancing its capability in pure reasoning in a dramatic way.

So, you know, the innovations that the Chinese companies starved of the sort of the scaling compute capabilities available, if you can acquire the top end data center infrastructure like the NVIDIA Blackwell chips and so on, they've innovated around efficiencies that are along two different dimensions and,

I'll circle back to this, but one of those two dimensions is the use of sparsity.

Now, sparsity is the opposite of dense in the terminology of AI.

Dense means that you're using every layer of the network in each inference run.

That's a dense, deep neural network.

And sparsity means you're only activating certain portions of it.

So if you have a 100 billion parameter model, any one inference run is dynamically assessing which portions of that deep neural network, the LLM, which layers of those have to be activated.

And this has given rise to the primary architecture for LLMs today, which is called mixture of experts.

So the only experts that are activated in this context are the ones which are relevant to the query.

And that reduces the computational overhead and it makes for a more efficient and effective inference run and reduces the cost in both energy and compute time and allows for a larger context window to be executed.

So all of those things are improving on the efficiency scale.

The second dimension has to do with memory.

And this is where the new deep seek technique comes in.

So.

We know that models just left as a dense model and being injected with your prompt and some additional context that you type in at the time of inference, they can be subject to hallucinations.

And so we like to ground that with a retrieval augmented generation model where you have an external memory, a database that is going to be referenced as context based.

And semantic relevance is used to selectively retrieve the relevant components of the grounding truth data that's in that retrieval augmented generation, typically a vector database, in order to achieve that semantic retrieval.