Micah-Hill Smith
speaker
601 appearances
1 recordings
1 series
first heard Jan 2026
last heard 8 Jan
Micah-Hill Smith’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 1 in all, peaking in Jan 2026 with 1.
Appearances
So we like certainly um
I incurred.
s some costs, but we were probably in the order of like hundreds of dollars of spend across all the benchmarking that we were doing.
Okay, so that's and stuff.
It was like kind of fine.
Yeah.
These days that's gone up an enormous amount for a bunch of reasons that we can talk about.
Um
But yeah, it w it wasn't that bad.
Cause if you can also remember that like the number of models that were dealing with was hardly any.
And the complexity of the stuff that we wanted to do to evaluate them was a lot less.
Like we were just asking some QA type questions.
And then one specific thing was for a lot of evals initially, we were just like sampling an answer directly without letting the models think.
We weren't even doing chain of thought stuff.
initially and that was the most useful way to get some results initially.
depends what you're looking at, right?
Because you can if you're trying to see whether or not it can solve a particular type of reasoning problem and you don't want to test it on its ability to do answer formatting at the same time, then you might want to use an LLM as answer extractor approach to make sure that you get the answer out no matter how unanswered.
But
These days it's mostly less of a problem.
Like if you instruct a model and give it examples of what the answers should look like, it can
Showing 81–100 of 601 · page 5 of 31
← Previous
Next →