Cal Newport

And if it found the right thing to do, or the human looked at the wrong thing, then they would say, this is a case where like the LLM plus its coding harness were smarter than the human.

So that's jumped from like 50%

from a couple of years ago to like 64% now.

All right.

So that is, this is sort of like the core data they're looking at to capture this idea of AI is getting so smart at producing programs that maybe soon it will be able to not only improve itself, but, and I'm going to quote here from the report,

become capable of full recursive improvement and begin building their own successors.

All right, so looking at that data and what I know about current AI tools, are these fears justified?

And I would say, no.

Here's what this data is all describing.

Now, about a year ago, the major AI companies got serious about building tools to help software developers.

These tools are a combination of human written programs called coding harnesses and LLMs.

The coding harnesses can make calls to the LLMs and then act on what the LLMs do.

The coding harnesses can also interact with other various tools on the computer.

So what all these charts seem to be showing is that like, oh, once they got serious about writing these coding harnesses and also tuning the LLMs to play nicer with these coding harnesses on these sort of computer programming related tasks and evaluations, things jumped up.

The world before having these harnesses

we couldn't do well on programming tasks, and now that we do, we do better.

Now, none of this is trivial.

In fact, it's a very smart market for the AI companies to go after.

Their software development is a big industry.

These coding harnesses built on top of LLMs are like really potentially very useful, right?