Nataliya Kosmyna

Please rank, reconcile, tell us how would you do that.

We felt it's very, very important to actually include humans, right?

Because this is the task that they know how to rank, how to do.

But back to AI, right?

Why we thought it's interesting to include AI.

Well, first, of course, a lot of people actively push that AI can do this job very well, right?

That, hey, I'm going to just upload this.

They're really great with all of these language outputs.

They will be able to rank things.

And how you do this, you actually give it a very detailed set of instructions, right?

How would you do that?

And what things to basically you need to carry about, like that these had 20 minutes, right?

So something very similar to teaching instructions, just like more specific language.

We're actually showing the paper exactly how we created this AI judge.

But there were actually differences between the two, right?

So human teachers, when they came back to us, well, first of all, they called those essays, a lot of the essays coming from LLM group soulless.

That's a direct quote.

I actually had, I put a whole long quote in there.

Soulless, I like that.

Soulless.