Corey Knowles

Like there's all these different models and in lots of ways, like you could go, if you go to

chat GPT, Claude, Gemini, and you ask it a reasonably simple question, it's going to give you the right answer pretty much.

Or if you give it a complex, subjective question, it's going to give you some different answers.

But how the hell do you decide which is better?

That's where evaluation is so critical.

I think Andre Carpenter- Someone has to read-

Well, Henri Carpenter had a tweet about six months ago where he said, like, the entire AI is just an evaluation problem now.

Because it's like the models are getting so complex that understanding what better means, they're not linearly better.

You might get an improvement in, say, let's imagine you get an improvement in creativity, but that also makes it more verbose and more likely to hallucinate.

You might be more creative, which is great.

I mean, AI is not magic.

It's one of the biggest problems.

People think AI is not magic.

AI is just better predictive text.

Like it's very good predictive text.

But fundamentally, if you go into your phone and you go into WhatsApp and you type a word and you keep pressing next word, that's predictive text too.

It's just not as good because it's based on a much simpler model.

Like all AI is trying to do is find the next character or the next pixel or whatever.

It has no concept of truth.

It has no concept of anything.