Micah-Hill Smith

speaker

601 appearances 1 recordings 1 series first heard Jan 2026 last heard 8 Jan

Micah-Hill Smith’s voice in public audio — every appearance, attributed to the second.

Trend

recordings per month · last 12 months

Recordings per month over the last 12 months — 1 in all, peaking in Jan 2026 with 1.

Appearances

newest first · ▶ plays the moment

Latent Space: The AI Engineer Podcast · Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith · 8 Jan 2026 podcast

We don't actually publish anything on this right now, but have track.

A bunch internally and our internal analytics on evals across all the models that we run, where we look at the difficulty, the questions, and the correlation between token usage and difficulty.

And net net, surprise, surprise, their models have got better at doing that over the course of this year.

I think going into next year, that's gonna be really important, especially as you multiply it by the number of steps in an agency workflow that a model has to take to get to an answer.

We are going to care.

Which would you rather have token?

It depends on the application and both are gonna be really important.

Yeah.

I mean I I would say that historically benchmarks have been single turn, but I wouldn't say they need to be at all into the future, right?

Like we have a couple of agentic benchmarks in the index right now and GDP eval that we were talking about.

We let the models do up to a hundred turns and um our stirrup agentic harness to do that evil, and we're gonna build similar stuff like that in the future.

It definitely is hard and you've got whole kinds of infrastructure problems to run that and

And exactly as you say, parallelize it, because we need to run that on hundreds of models and we want to do that really fast when new models come out and where labs want us to run it on their models.

But you can do it.

We're putting in the work to build that stuff and it's gonna be great.

Yeah.

And you can see it as a uh good good thing or bad things depending on what your view is, but it means that we have a quite active creative direction approach to trying to understand what creative professionals and users want to do with those image and video models.

And so that we can be directing the arenas in our categories toward gathering votes on what people care about.

One call out actually to listeners, like if you are using our

arenas is that you can submit requests to us for things that we should cover.

Showing 521–540 of 601 · page 27 of 31