David Shu

Today, it is almost entirely prompt-driven. There's actually more than one model in use under the hood as we try different things. For example, we use a different model for solving the problem of... If we want to go get a package, what module do we get to do that? Which sounds like a mechanical process, but it actually isn't. There's a couple of steps there. So a model helps out with that.

Today, it is almost entirely prompt-driven. There's actually more than one model in use under the hood as we try different things. For example, we use a different model for solving the problem of... If we want to go get a package, what module do we get to do that? Which sounds like a mechanical process, but it actually isn't. There's a couple of steps there. So a model helps out with that.

There's very different sorts of prompts you use for trying to come up with the name of a sketch than there are for answering questions. But at the moment, it's entirely prompt-driven. in the sense that a large context window and a lot of careful context construction can handle this, can improve things. And that can include a lot of tool use.

There's very different sorts of prompts you use for trying to come up with the name of a sketch than there are for answering questions. But at the moment, it's entirely prompt-driven. in the sense that a large context window and a lot of careful context construction can handle this, can improve things. And that can include a lot of tool use.

Tool use is a very fun feature of models where you can instruct. So to back up and give you a sense of how the models work, an LLM generates the next token based on all the tokens that come before it. When you're in chat mode and you're chatting with a model, you can at any point stop and have the model generate the next token. It could be part of the thing you're asking it or its response.

Tool use is a very fun feature of models where you can instruct. So to back up and give you a sense of how the models work, an LLM generates the next token based on all the tokens that come before it. When you're in chat mode and you're chatting with a model, you can at any point stop and have the model generate the next token. It could be part of the thing you're asking it or its response.

That meta information about who is talking is sort of built into just a stream of tokens. So similarly, you can... You can define a tool that a model can call. You can say, here's a function that you can call and it will have a result. And the model can output the specialized token that says call this function, give it a name, write some parameters.

That meta information about who is talking is sort of built into just a stream of tokens. So similarly, you can... You can define a tool that a model can call. You can say, here's a function that you can call and it will have a result. And the model can output the specialized token that says call this function, give it a name, write some parameters.

And then instead of the model generating the next token, you pause the stream. You, the caller, go and run some code. You go and run that function call that it defined. paste the result of that function call in as the next set of tokens, and then ask the model to generate the token after it. So that technique is a great way to have automated feedback into the model.

And then instead of the model generating the next token, you pause the stream. You, the caller, go and run some code. You go and run that function call that it defined. paste the result of that function call in as the next set of tokens, and then ask the model to generate the token after it. So that technique is a great way to have automated feedback into the model.

So a classic example is a weather function. And so you define a function which says current weather. the model, then you can ask the model, hey, what's the weather? And the model can say, call function current weather. Your software that's printing out the tokens pauses, calls current weather, says sunny. You paste sunny in there.

So a classic example is a weather function. And so you define a function which says current weather. the model, then you can ask the model, hey, what's the weather? And the model can say, call function current weather. Your software that's printing out the tokens pauses, calls current weather, says sunny. You paste sunny in there.

And then the model generates the next set of tokens, which is the chat response saying, oh, it's currently sunny. And that's the sort of easy way to plug external systems into a model. This is going on under the hood of the user interfaces you use onto frontier models. So this is happening in ChatGPT and Claude and all these systems. Sometimes they show it to you happening, which is how you know.

And then the model generates the next set of tokens, which is the chat response saying, oh, it's currently sunny. And that's the sort of easy way to plug external systems into a model. This is going on under the hood of the user interfaces you use onto frontier models. So this is happening in ChatGPT and Claude and all these systems. Sometimes they show it to you happening, which is how you know.

You see it less now, but about six months ago, you could see in the GPT-4 model, you would ask it questions and it would generate Python programs and run them and then use the output of the Python program in its answer. I had a really fun one where I asked it, how many transistors fit on the head of a pin? And it started producing an answer. And it said, well, transistors are about this big.

You see it less now, but about six months ago, you could see in the GPT-4 model, you would ask it questions and it would generate Python programs and run them and then use the output of the Python program in its answer. I had a really fun one where I asked it, how many transistors fit on the head of a pin? And it started producing an answer. And it said, well, transistors are about this big.

Pins are about this big. And so I guess, a magic little emoji appeared, that this means this many transistors fit on the head of a pin, some very large number. And if you click on the emoji, it shows you the Python program it generated to do the arithmetic.

Pins are about this big. And so I guess, a magic little emoji appeared, that this means this many transistors fit on the head of a pin, some very large number. And if you click on the emoji, it shows you the Python program it generated to do the arithmetic.

It executed that as a function call, came back with a result, and that saved it the trouble of trying to do the arithmetic itself, which LLMs notoriously struggle with doing arithmetic. This is a great thing to outsource to a program.

It executed that as a function call, came back with a result, and that saved it the trouble of trying to do the arithmetic itself, which LLMs notoriously struggle with doing arithmetic. This is a great thing to outsource to a program.