Sebastian Raschka
speaker
1,024 appearances
1 recordings
1 series
first heard Feb 2026
last heard 1 Feb
Sebastian Raschka’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 1 in all, peaking in Feb 2026 with 1.
Appearances
what it does to the model if you move something around.
Ablation studies doesn't make it better or worse, but there are so many, let's say, ways you can implement a transformer and make it still work.
Big ideas that are still prevalent is mixture of experts, multi-ad latent attention, sliding window attention, group query attention.
And then at the end of the year, we saw a focus on making the attention mechanism scale linearly with inference token prediction.
So there were QEN3Next, for example, which added a gated delta net.
It's like kind of like inspired by state space models where you have a fixed state that you keep updating, but it makes essentially this attention cheaper or it replaces attention with a cheaper operation.
Yeah, so maybe we should start with the GPT-2 architecture, the transformer that was derived from the Attention is All You Need paper.
So the Attention is All You Need paper had a transformer architecture that had two parts, an encoder and a decoder.
And GPT went just focusing in on the decoder part.
It is essentially still a neural network and it has this attention mechanism inside.
And you predict one token at a time.
You pass it through an embedding layer.
There's the transformer block.
The transformer block has attention modules and a fully connected layer.
And there are some normalization layers in between, but it's essentially neural network layers with this attention mechanism.
So coming from GPT-2, when we move on to GPT-OSS, there is, for example, the mixture of experts layer.
It's not invented by GPT-OSS.
It's a few years old, but it is essentially a tweak
to make the model larger without consuming more compute in each forward pass.
So there is this fully connected layer.
Showing 141–160 of 1,024 · page 8 of 52
← Previous
Next →