Stephen Wolfram

that, you know, is really LLM communicates with LLM.

I mean, it's kind of like the, you know, I write a piece of email using my LLM and, you know, puff it out from the things I want to say, your LLM turns it into, and the conclusion is X. Now, the issue is, you know, that the thing is going to make this thing that is sort of semantically plausible.

Mm-hmm.

And it might not actually be what you, you know, it might not be kind of relate to the world in the way that you think it should relate to the world.

I've seen this, you know, I've been doing, okay, I'll give you a couple of examples.

I was doing this thing when we announced this plugin for ChatGPT.

I had this lovely example of a math word problem, some complicated thing, and it did a spectacular job of taking apart this elaborate thing about, you know, this person has twice as many chickens as this, et cetera, et cetera, et cetera.

And it turned it into a bunch of equations.

It fed them to Wolfram language.

We solved the equations.

Everybody did great.

We gave back the results.

And I thought, okay, I'm going to put this in this blog post I'm writing.

Okay, I thought I better just check.

And turns out it got everything, all the hard stuff it got right.

At the very end, last two lines, it just completely goofed it up and gave the wrong answer.

And I would not have noticed this.

Same thing happened to me two days ago.

Okay, so I thought, you know, I made this with this ChatGPT plugin kit.

I made a thing that would emit a sound, would play a tune on my local computer, right?