Sean Carroll

And you can very clearly tell when you're talking to the AI about research level questions, that as long as you're asking questions that have already been answered and talked about in certain ways, they're quite good.

I use it like as a hypercharged version of Wikipedia, right?

If the answer is out there or is closely related to an answer that's out there, they're super good at that.

As soon as you start talking in a way that is not common, that is trying to change the way that we think about a certain problem, which is something that is very common in my kind of research, then it stumbles.

Then it has trouble because it doesn't really know how to put things together in the right way.

So I'm not surprised about the math results.

To Rob's question about existential risk to humanity, I mean, I think I still haven't heard what the risk is.

What is the scenario by which AIs are going to be an existential risk?

I think that there's sort of two kinds of scenarios.

One is completely vague and just based on the idea that

the AIs will be so intelligent that they will outwit us somehow.

And I think that's a very shallow analysis that doesn't really get into what it means for an AI to have motivation or goals or values or planning or anything like that.

And not to say that it's impossible, I just haven't heard any reason to think that those sorts of scenarios where the AI is so smart we can't

cope with it make any sense at all.

The other kind of scenario is that we are really stupid and we put AI in everything and give it control and responsibility over things that it shouldn't have and we can't really control what it's going to do.

That, I see, is perfectly plausible.

That's clearly happening right now, right?

Plenty of people are turning tasks over to AI and then realizing, oh, it actually doesn't do such a good job.

And if those tasks are super, super important, then we're going to be in trouble.

Again, I see no possible way in which that's existential.