Mike Knoop
speaker
927 appearances
4 recordings
1 series
first heard Mar 2025
last heard 18 Nov
Mike Knoop’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 1 in all, peaking in Nov 2025 with 1.
Appearances
Yeah.
RK J two, let alone RK J one.
There is.
We so we ArcPrice does most of the testing today ourselves.
Uh like we basically wait, we wait for API access, either get early access or you know, wait for the public access.
And we have um a GitHub repo where we just have a standard very simple prompt that we use to baseline across all the sort of frontier systems that that we test.
OpenEI situation was a little different because they had reached out to us and said, Hey, we think we've got a really impressive result on the public eval set and we'd like your help to verify.
on a semi-private set, which is what we created that data set to be able to do.
Um and in that case, uh, there was really you know, uh very little prompt engineering at all.
It was very mostly just a verification of what we'd are what we'd seen from them.
Um so yeah, the in all these cases the the amount of prompting that goes into these is extremely minimal.
We're basically just giving it the sort of grid with numbers and asking it to solve any directly goes into it.
We've been trying to deploy A agents for two years at Zapier.
Um, and this was kind of what led me to sort of originally get invited interested in ARC is uh, you know, that's it's this reliability problem.
Language models fundamentally are stochastic in terms of how they work.
Um, and so you're just never gonna get that 100% accuracy directly from the model without adding a lot of extra additional guardrails.
Source like processing on top.
We've done some of that for Zapier.
The way that I kind of think about it, you got these concentric rings of use cases that kind of get unlocked as reliability and consistency goes up.
And today we're pretty much still in the con in the and this is true at the frontier.
Showing 741–760 of 927 · page 38 of 47
← Previous
Next →