Terence Tao

It makes really basic mistakes.

But the AI-generated proofs, they can look

superficially flawless.

And it's partly because that's what the reinforcement learning has trained them to do, to produce text that looks like what is correct, which for many applications is good enough.

So the errors are often really subtle, and then when you spot them, they're really stupid.

Like no human would have actually made that mistake.

Yeah, so the sense of smell.

This is one thing that humans have.

And there's a metaphorical mathematical smell that it's not clear how to get the AI to duplicate that.

Eventually, I mean, so the way...

AlphaZero and so forth, they make progress on Go and chess and so forth.

In some sense, they have developed a sense of smell for Go and chess positions, that this position is good for white, it's good for black.

They can't enunciate why, but just having that sense of smell lets them strategize.

So if AIs gain that ability to sort of, a sense of viability of certain proof strategies, so you can say,

I'm going to try to break up this problem into two small subtasks, and they can say, oh, this looks good.

The two tasks look like they're simpler tasks than your main task, and they've still got a good chance of being true.

So this is good to try.

Or, no, you've made the problem worse because each of the two subproblems is actually harder than your original problem, which is actually what normally happens if you try a random thing to try.

Normally, it's very easy to transform a problem into an even harder problem.

Very rarely do you transform it into a simpler problem.