Peter McCrory

It turns out that in order to get those most complex implementations, so moving beyond just straightforward data entry to something that's much more sophisticated, maybe like automating biological research and analysis,

Yeah, that's a big step.

And, you know, that is arguably like something that is...

maybe on the horizon, like I think a lot about even our productivity analysis might be conservative if we're failing to account for the automation of innovation itself.

Maybe we can return to that.

I'm kind of curious to hear your perspective.

But it turns out for those most complex tasks that we see in our data, businesses need to provide disproportionately more contextual information.

So even if the capabilities are there,

If you don't have the relevant information to deploy that capability, business adoption will maybe be constrained or bottlenecked.

Exactly.

The tacit knowledge piece is so crucial, I think, about developing a sales strategy.

It might be that your co-worker has the relevant information, not even in the

customer management software that you're using, but in their brain or in some document that they've written out.

And if the business is not thinking about how to elicit that tacit information in a way that can make the model effective, you might fail to see tasks in our API deployment for which Claude and other frontier models would actually be capable if they had that information.

Yeah, the data, the underlying data that we sampled was just before Opus 4.5.

This sort of touches on one of the findings, which is for the most complex tasks, which I presume you're asking Opus 4.5 to take on very ambitious exercises.

In our data, those most complex tasks tend to have, that's where the model struggles.

So there's like the success rate is somewhat lower.

And that suggests that human expertise to evaluate the quality of the output

is more important and you need more human delegation and direction and managerial oversight.