Micah-Hill Smith
speaker
601 appearances
1 recordings
1 series
first heard Jan 2026
last heard 8 Jan
Micah-Hill Smith’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 1 in all, peaking in Jan 2026 with 1.
Appearances
Um, maybe so we can hit past all the like w we've like we have lots and lots of ELs and stuff.
Um the interesting ones to talk about today that'd be great to bring up like a few of our recent things, I think.
Um, that probably not many people will be familiar with yet.
So first one of those is our omniscience index.
So this one is
a little bit different to most of the intelligence evals that we run.
We built it specifically to look at the embedded knowledge in the models and to test
um hallucination by looking at when the model doesn't know the answer is not able to get it correct, what's its probability of saying I don't know or giving an incorrect answer.
So
The metric that we use for omniscience goes from negative 100 to positive 100 because we're simply taking off a point if you give an incorrect answer to the question.
We're pretty convinced that this is an example of where it makes most sense to do that, because it's strictly more helpful to say I don't know instead of giving a wrong answer to factual knowledge question.
And one of our goals is to shift the incentive that evals create for um models and the labs creating them to get higher scores.
And almost every eval across all of AI up until this point, it's been graded by simple percentage correct as the main metric, the main thing that gets hyped.
And so you should take a shot at
everything.
There's no incentive to say, I don't know.
So we did that for this one here.
Yeah.
Like anything that we do, it's not a perfect metric or the whole story of everything that you think about is hallucination.
Um but yeah, it's pretty useful and has some interesting results.
Showing 241–260 of 601 · page 13 of 31
← Previous
Next →