John Yang

speaker

228 appearances 1 recordings 1 series first heard Dec 2025 last heard 31 Dec

John Yang’s voice in public audio — every appearance, attributed to the second.

Trend

recordings per month · last 12 months

Recordings per month over the last 12 months — 1 in all, peaking in Dec 2025 with 1.

Appearances

newest first · ▶ plays the moment

Latent Space: The AI Engineer Podcast · [State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang · 31 Dec 2025 podcast

I mean it it's cool to see um a lot of the newer benchmarks like really try to d diversify the repos.

Like in the two follow ups we did with multimodal and multilingual, we made it a point to do that.

So I think But you can also just put out C Bench twenty twenty five and just That is true and do a new distribution.

Yeah, yeah.

So it's been cool to see the follow ups.

I think quietly and and it's a open question for me.

I'm excited to see how people

curate the next sets.

Like it's kind of interesting to see in the literature or in their blog posts like how they're justifying why they're creating their separate split.

The easier ones were like, oh, more languages, more repos.

And then I think now people are like, well, ours is more difficult because of this curation technique.

And I'm yeah, I'm excited to see how how long that lasts and you know where we're gonna like guide the evaluations towards.

happy to do it, especially on your podcast, it's on or um yeah, so basically the idea is I don't like unit tests as a form of verification.

And I also think there's the issue with Sweetbench where all of the task instances are independent of each other.

So the moment you have the model kind of submit it, oh it's done, you know, and and and that's the end of the story, the end of the episode, you know.

So with Code Clash, what we're thinking is let's try to re-

Really evaluate like long horizon development and uh development on a code base that is consequential and conditioned upon what a model did, you know, before to that code base.

And so the general idea is you have two or more language models and they play a programming tournament.

And what that means is each model maintains their own code base, and each round of the tournament, first they

get to like edit and improve their code base however they see fit, very self-determined.

Showing 41–60 of 228 · page 3 of 12