Mike Stonebraker

We've been trying to make it work on real-world databases.

especially real-world data warehouses.

So we've been trying the technology on four different production databases, warehouses, where we've gotten the workload, the actual workload that's run, and from the actual users using the system,

And we've gotten them to reverse engineer the text that corresponds to that sequel.

So we have text and sequel for, we have four benchmarks.

Tell me all the professors at MIT who won the Turing Award.

And so an LLM is supposedly good at that.

And so the text to SQL benchmarks, there's one called Spider, another one called Bird.

And the best LLM systems are pretty good at those benchmarks, like 80% accuracy or better.

Not superhuman, but they're pretty good.

Like, you would consider using them.

And, you know, like the current leaderboard is something like 85% accuracy, which, I mean, it's getting there.

You say maybe it's not quite ready for prime time, but it certainly looks pretty good.

Well, on our benchmarks, large language models get 0%.

And if you enhance them with RAG and all the tricks, it goes to 10%.

And if you give as a prompt the from clause, in other words, all the actual tables that need to be accessed, and all the actual join clauses that need to be joined, then accuracy goes to about 35%.

So the definition of this stuff is not ready for prime time and not going to be for a while, if ever.

So what's the difference?

Number one, LLMs are trained on the pile.

Data warehouse data is not in the pile.