Sebastian Raschka
speaker
1,024 appearances
1 recordings
1 series
first heard Feb 2026
last heard 1 Feb
Sebastian Raschka’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 1 in all, peaking in Feb 2026 with 1.
Appearances
You can still start with GPT-2 and you can add things to that model to make it into this other model.
So it's all still kind of like the same lineage, the same, it is a very close relationship between those.
But top of my head, DeepSeq, what was unique there is the mixture of, I mean, they were not inventing mixture of experts.
We can maybe talk a bit more what mixture of experts means later.
But just to list these things first before we dive into detail, a mixture of experts, but then they also had multi-head latent attention, which is a tweak to the attention mechanism, where this was, I would say, 2025, the main distinguishing feature.
factor because between these open weight models different tweaks to make inference or kv cache size we can also define kv cache in a few moments but to kind of make it more economical to have long context to shrink the kv cache size so what are tweaks that we can do and most of them focused on the attention mechanism there is multi-head latent attention in in deep seek and
There is group query attention, which is still very popular.
It's not invented by any of those models.
It goes back a few years, but that would be the other option.
Sliding window attention, I think almost reuses it, if I remember correctly.
So there are these different tweaks that make the models different.
Otherwise, I put them all together in an article once where I just compared them.
They are very surprisingly similar.
It's just different numbers in terms of how many repetitions of the transformer block you have in the center.
And just little knobs that people tune.
But what's so nice about it is it works no matter what.
You can tweak things.
You can move the normalization layers around.
You get some performance gains.
And OMO is always very good in ablation studies showing what actually...
Showing 121–140 of 1,024 · page 7 of 52
← Previous
Next →