Nick Heiner

speaker

529 appearances 1 recordings 1 series first heard Mar 2026 last heard 25 Mar

Nick Heiner’s voice in public audio — every appearance, attributed to the second.

Trend

recordings per month · last 12 months

Recordings per month over the last 12 months — 1 in all, peaking in Mar 2026 with 1.

Appearances

newest first · ▶ plays the moment

The Neuron: AI Explained · Inside the Secret Labs Where AI Learns to Work · 25 Mar 2026 podcast

There they are.

But that was giving you noise the whole way you were getting up there.

So one thing we do at Surge is we try to have 100% correctness, 100% tasks that actually work instead of just accepting this degree of noise.

So that's probably my biggest recommendation for people trying to build their own eval sets is to I think there's a certain temptation where it's like building the eval site isn't fun.

Building the agent is what's fun.

Yeah.

But like, yeah, you shouldn't you shouldn't skip your vegetables.

Yeah, I mean, they can be benchmarks, right?

Like at a high level, a benchmark is just a series of challenges for the model and scores.

So RL environments are just a way to do that.

And yeah, in the fullness of time, do most benchmarks become RL environments?

I think it's certainly possible.

You know, it's sort of like in software development where you have your test pyramid, where at the bottom of the pyramid, you have your unit tests, which are very fine grained and give you very specific feedback.

And the top of the pyramid, you have your integration tests, which test the whole system.

And the reason it's shaped like a pyramid is that the integration tests are much more expensive and slow to run.

And when something fails, you don't know exactly what the problem is necessarily.

But they're also way less brittle than the unit tests because they are tracking sort of closer to your end-to-end value.

And so I sort of see different benchmarks as having different spots in that pyramid where like, yeah, you need your RL environments to sort of track like, okay, end of the day, can this thing be a lawyer?

But sometimes you want more specific benchmarks like instruction following or groundedness that will help you sort of tease out like, okay,

My latest model checkpoint had a big regression on the lawyer abilities and it had a big regression on the instruction following abilities.

Showing 341–360 of 529 · page 18 of 27