Micah-Hill Smith

speaker

601 appearances 1 recordings 1 series first heard Jan 2026 last heard 8 Jan

Micah-Hill Smith’s voice in public audio — every appearance, attributed to the second.

Trend

recordings per month · last 12 months

Recordings per month over the last 12 months — 1 in all, peaking in Jan 2026 with 1.

Appearances

newest first · ▶ plays the moment

Latent Space: The AI Engineer Podcast · Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith · 8 Jan 2026 podcast

Get the answers um and your format and then you can do like a simple rejects.

You've also yeah, got like the um

uh different degrees of variance and different benchmarks, right?

So if if you if you run four question multi-choice on a modern reasoning model at the temperature suggested by the labs for their own models, the variance that you can see on a four question multi-choice eval is pretty enormous if you only do a single run of it and it has a small number of questions, especially especially.

So like one of the things that we do is run an enormous number of all of our evals when we're developing new ones

And doing upgrades to our intelligence index to bring in new things so that we can dial in the right number of repeats, so that we can get to the 95% confidence confidence intervals that we're comfortable with, so that when we pull that together, we can be confident in intelligence index to at least as tight as like a plus or minus one at a 95% confidence.

Mm-hmm.

Yeah.

And again, that just adds a straight multiple to the.

So that's one of many reasons

th there is there is a mix.

A hundred percent.

Yeah, yeah, yeah.

Exactly.

So we uh um laser focus like on everything we do, on having the best independent metrics and making sure that no one can manipulate them in any way.

There are quite a lot of processes we've developed over the last couple of years to make that true.

For like with the one you bring up like right here, of the fact that if we're working with a lab, if they're giving us a private endpoint to evaluate a model, that it is

Is totally possible that what's sitting behind that black box is not the same as they serve on a public endpoint.

Showing 101–120 of 601 · page 6 of 31