Sebastian Raschka
speaker
1,024 appearances
1 recordings
1 series
first heard Feb 2026
last heard 1 Feb
Sebastian Raschka’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 1 in all, peaking in Feb 2026 with 1.
Appearances
add previous knowledge stuff, but it also gives it certain preference prompts, do what I preferred last time, whatever, like something like that.
But it doesn't unlock new capabilities.
So for that, one thing people do use still is LoRa, LoRa adapters.
These are basically, instead of updating the whole weight matrix, there are two smaller weight matrices that you kind of have in parallel or overlay.
It's like the delta.
But yeah, you can do that to some extent, but then again,
It is economics.
So there were also papers, for example, Laura learns less, but forgets less.
It's like, you know, it's no free lunch.
If you want to learn more, you need to use more weights, but it gets more expensive.
And then again, if you learn more, you forget more.
And it's like you have to find that Goldilocks zone, basically.
I think also looking at the extremes, I think there's again no free lunch.
So the one extreme to make it cheap, you have, let's say, an RNN that has a single state where you save everything from the previous stuff.
It's like a specific fixed size thing.
So you never really grow the memory because you are stuffing everything into one state.
But then...
The longer the context gets, the more information you forget because you can't compress everything into one state.
Then on the other hand, you have the transformers, which try to remember every token, which is great sometimes if you want to look up specific information, but very expensive because you have the KV cache that grows, the dot product that grows.
But then, yeah, like you said, the Mamba layers, I mean, they kind of have the same problem.
Showing 721–740 of 1,024 · page 37 of 52
← Previous
Next →