Sayash Kapoor

They indeed have.

They are so much better today than they were just a year ago.

But the fact is that the tasks that you can do with these systems actually are bounded by the rate of hallucinations or the reliability.

And that's one place where AI systems continue to struggle.

And in a domain like software engineering, where you have this instant feedback loop, where you can actually run the code and see what the output would be, it's a much easier bottleneck to address as opposed to something like the law, where even the right answer is not obvious to a domain expert.

Domain experts can reasonably differ in the approach that they take.

So this is just one example of a bottleneck in a domain where the right answer can be a bit more subjective than encoding.

Sayash, do you believe this sort of recursive self-improvement is possible?

I mean, in some sense, I think the process of recursive self-improvement started like six decades ago.

In fact, the entire history of computing has been one where we develop tools that then aid us in the development of better tools.

We've developed compilers that have allowed us to be like two orders of magnitude better at programming.

We've developed frameworks on top of that.

We've developed entire systems and libraries that allow us to do things that would frankly take like an experienced software engineer years or decades of time if they were using assembly language.

So I think in some sense, this loop has been kickstarted already.

This loop is something that the entire history of computing bears out.

What I disagree with in terms of Daniel's predictions is whether this process will naturally lead us to a point where we develop the automated AI R&D researcher, or whether humans will continue to have this edge and teams of humans with AI will continue to sort of outperform AI alone, and whether this process will lead to artificial superintelligence.

I actually think that it's a very plausible scenario for me that we get this sort of recursive self-improvement, that AI systems do indeed continue performing better and better at AI research tasks.

But the end process of that need not be ASI.

The end process could just be far more capable models than we have today, perhaps following the trend of previous technologies, and yet not the point where we have these systems that outperform humans, the top human experts and everything, which is, I believe, the definition of ASI.

Exactly.