Aman Sanger
speaker
350 appearances
2 recordings
1 series
first heard Sep 2024
last heard Oct 2024
Aman Sanger’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsNo recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.
Appearances
kind of having the model ask questions about various pieces of the code. So you kind of take the pieces of the code, then prompt the model or have a model propose a question for that piece of code, and then add those as instruction finds new data points. And then in theory, this might unlock the model's ability to answer questions about that code base.
I think test time compute is really, really interesting. So there's been the pre-training regime, which will kind of, as you scale up the amount of data and the size of your model, get you better and better performance, both on loss and then on downstream benchmarks and just general performance when we use it for coding or other tasks.
We're starting to hit a bit of a data wall, meaning it's going to be hard to continue scaling up this regime.
And so scaling up test time compute is an interesting way of now, you know, increasing the number of inference time flops that we use, but still getting like, like, yeah, as you increase the number of flops use inference time getting corresponding improvements in the performance of these models tremendously.
Traditionally, we just had to literally train a bigger model that always used that many more flops. But now we could perhaps use the same size model and run it for longer to be able to get an answer at the quality of a much larger model. And so the really interesting thing I like about this is there are some problems that perhaps require
hundred trillion parameter model intelligence trained on a hundred trillion tokens. Um, but that's like maybe 1%, maybe like 0.1% of all queries. So are you going to spend all of this effort, all of this compute training model, uh,
that costs that much and then run it so infrequently, it feels completely wasteful when instead you get the model that can, that you train the model that's capable of doing the 99.9% of queries, then you have a way of inference time running it longer for those few people that really, really want max intelligence.
I mean, yeah, that's an open research problem, certainly. I don't think anyone's actually cracked this model routing problem quite well. We'd like to. We have initial implementations of this for something like CursorTab. But at the level of going between 4.0 Sonnet to O1, It's a bit trickier.
There's also a question of what level of intelligence do you need to determine if the thing is too hard for the four-level model? Maybe you need the O1-level model. It's really unclear.
Um, well, it's weird because like test time compute, there's like a whole training strategy needed to get test time to compute to work. And the really, the other really weird thing about this is no one like outside of the big labs and maybe even just open AI, no one really knows how it works. Like there've been some really interesting papers that, uh, show hints of what they might be doing.
And so perhaps they're doing something with tree search using process reward models. But yeah, I just I think the issue is, we don't quite know exactly what it looks like.
So it would be hard to kind of comment on like where it fits in, I would put it in post training, but maybe like the compute spent for this kind of for getting test time compute to work for a model is going to dwarf pre training eventually.
It's fun to speculate.
Yeah. So one thing to do would be, I think you probably need to train a process reward model, which is, so maybe we can get into reward models and outcome reward models versus process reward models. Outcome reward models are the kind of traditional reward models that people are trained for language modeling. And it's just looking at the final thing.
So if you're doing some math problem, let's look at that final thing you've done, everything, and let's assign a grade to it, how likely we think, like what's the reward for this outcome. Process reward models instead try to grade the chain of thought.
And so OpenAI had some preliminary paper on this, I think, last summer, where they use human labelers to get this pretty large, several hundred thousand data set of grading chains of thought. Ultimately, it feels like I haven't seen anything interesting in the ways that people use process reward models outside of just using it as a means of affecting how we choose between a bunch of samples.
So like what people do in all these papers is they sample a bunch of outputs from the language model and then use the process reward models to grade all those generations alongside maybe some other heuristics and then use that to choose the best answer. The really interesting thing that people think might work and people want to work is tree search with these process reward models.
Because if you really can grade every single step of the chain of thought, then you can kind of branch out and explore multiple paths of this chain of thought. And then use these process reward models to evaluate how good is this branch that you're taking.
And like the interesting work that I think has been done is figuring out how to properly train the process or the interesting work that has been open sourced. And people I think talk about is how to train the process reward models, maybe in a more automated way.
I could be wrong here, could not be mentioning something because I haven't seen anything super that seems to work really well for using the process reward models creatively to do tree search and code.
Showing 281–300 of 350 · page 15 of 18
← Previous
Next →