Arvid Lundmark
speaker
98 appearances
1 recordings
1 series
first heard Sep 2024
last heard Sep 2024
Arvid Lundmark’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsNo recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.
Appearances
But then the idea is we show you, do you just want to add this file, this file, this file also to tell the model to edit those files for you? Because if maybe you're making the API, you should also edit the client and the server that is using the API and the other one resolving the API.
So that'll be kind of cool as both there's the phase where you're writing the prompt and there's before you even click enter, maybe we can help resolve some of the uncertainty.
And if you can make the KV cache smaller, one of the advantages you get is like, maybe you can speculate even more. Maybe you can guess, here's the 10 things that... could be useful. Like, predict the next 10, and it's possible the user hits the one of the 10. It's a much higher chance than the user hits the exact one that you show them.
Maybe they type another character, and we sort of hit something else in the cache. So there's all these tricks where... The general phenomena here is... I think it's also super useful for RL is... maybe a single sample from the model isn't very good. But if you predict like 10 different things, it turns out that one of the 10, that's right, is the probability is much higher.
There's these passive key curves. And, you know, part of RL, like what RL does is you can exploit this pass at k phenomena to make many different predictions. And one way to think about this, the model sort of knows internally, has some uncertainty over which of the k things is correct, or which of the k things does the human want.
So when we RL our cursor tab model, one of the things we're doing is we're predicting which of the hundred different suggestions the model produces is more amenable for humans? Like, which of them do humans more like than other things? Maybe, like, there's something where the model can predict very far ahead versus, like, a little bit and maybe somewhere in the middle and...
And then you can give a reward to the things that humans would like more and sort of punish the things that it won't like and sort of then train the model to output the suggestions that humans would like more. You have these like RL loops that are very useful that exploit these passive K-curves. Oman maybe can go into even more detail.
But MLA is from this company called DeepSeek. It's quite an interesting algorithm. Maybe the key idea is sort of in both MQA and in other places, what you're doing is you're sort of reducing the number of KV heads. The advantage you get from that is
there's less of them, but maybe the theory is that you actually want a lot of different, like you want each of the keys and values to actually be different. So one way to reduce the size is you keep one big shared vector for all the keys and values. And then you have smaller vectors for every single token, so that you can store only the smaller thing. There's some sort of low-rank reduction.
And the low-rank reduction, at the end of the time, when you eventually want to compute the final thing, remember that you're memory bound, which means that you still have some compute left that you can use for these things. So if you can expand the
um the latent vector back out and and somehow like this is far more efficient because just like you're reducing like for example maybe like reducing like 32 or something like the size of the vector that you're keeping yeah there's perhaps some richness in having a separate uh set of keys and values and query that kind of pairwise match up versus compressing that all into one and that interaction at least
But it also allows you to make your prompt bigger for certain.
Let's opine on bug finding.
To be clear, I think they sort of understand code really well. While they're being pre-trained, the representation that's being built up, almost certainly somewhere in the stream, the model knows that maybe there's something sketchy going on. It sort of has some sketchiness, but actually eliciting the sketchiness to...
Part of it is that humans are really calibrated on which bugs are really important. It's not just actually saying there's something sketchy. It's like, is this sketchy trivial? Is this sketchy like you're going to take the server down? Part of it is maybe the cultural knowledge of... Like, why is a staff engineer a staff engineer?
A staff engineer is good because they know that three years ago, like, someone wrote a really, you know, sketchy piece of code that took the server down. And as opposed to, like... As opposed to maybe just, like, you know, you just... this thing is like an experiment. So like a few bugs are fine. Like you're just trying to experiment and get the feel of the thing.
And so if the model gets really annoying when you're writing an experiment, that's really bad. But if you're writing something for super production, you're like writing a database, right? You're writing code in Postgres or Linux or whatever. Like your Linus Torvalds, it's sort of unacceptable to have even an edge case. And just having the calibration of like,
Yeah, yeah, yeah.
In fact, I actually think this is one of the things I learned from Arvid is, you know, sort of aesthetically, I don't like it. But I think there's certainly something where, like, it's useful for the models. And humans just forget a lot. And it's really easy to make a small mistake and cause, like,
You know, my hope initially is, and I can let Michael chime in too, but it was like, there's, It should, you know, first help with the stupid bugs. Like, it should very quickly catch the stupid bugs. Like, off-by-one errors, like, sometimes you write something in a comment and do it the other way. It's, like, very common. Like, I do this.
Showing 41–60 of 98 · page 3 of 5
← Previous
Next →