Nick Heiner

But this benchmark will allow us to, in a much more robust and, like, scientific way, sort of put something behind those intuitions.

So once that gets released, I'm going to follow up with you about whether 3.5, how it scores.

Yeah, absolutely.

And to Grant's point about taste, that is something that is underdeveloped right now, is the recognition of taste clusters.

And like, you know, you can prompt the model and say, like, you know, be concise or whatever.

But

theory is a certain extent to which the current training regimes, a lot of labs are using is kind of like a lowest common denominator.

Like we're just, we're just going to find sort of a one size fits all approach.

And if you take that to the limit, then like everything becomes the Avengers.

which you know whatever i i love captain america like i'm not i'm not but like you know there's there's more to art than than the marvel cinematic universe right and so that's that's another like deep open research question that we're focusing on is how do you teach them all about these existence these these taste clusters and then you know find sort of natural ways to like

have it pick up on, Oh, this is someone who, you know, really likes Hemingway's writing style.

Yeah.

Right, right.

Or it's like, you know, you meet like a lawyer at a party and you ask them some legal question.

And they're like, yeah, that's like a real estate law question, which is like totally different from like estate planning or whatever.

Yeah, it's like completely unrelated.

there are a lot of questions architecturally of how you sort of present that information to the model.

Like, you know, we have very sophisticated algorithms to identify taste clusters, right?

Like, you know, Netflix obviously has a recommender algorithm where it's like figuring out people who like one show might like another.

And so the question for me is like, how much do you try to bake that into the LLM itself versus how much do you have like another system that sits alongside it?