David Shu

have the LLM communicate with the outside world in the middle there by it driving that or you don't even have to have it drive it. You could have software outside the LLM that looks at the token set as it's appeared and then insert more tokens for it. So this is all the fun stuff you can do by running these models yourself. Yeah, I know. It's so fun.

have the LLM communicate with the outside world in the middle there by it driving that or you don't even have to have it drive it. You could have software outside the LLM that looks at the token set as it's appeared and then insert more tokens for it. So this is all the fun stuff you can do by running these models yourself. Yeah, I know. It's so fun.

Yeah, that's a really good question. The best programming language for LLMs today is Python, and I believe that is a historical artifact of the fact that all of the researchers working on generative models work in Python. And so they spend the most time testing it with Python and judging a model's results by Python output.

Yeah, that's a really good question. The best programming language for LLMs today is Python, and I believe that is a historical artifact of the fact that all of the researchers working on generative models work in Python. And so they spend the most time testing it with Python and judging a model's results by Python output.

There was a great example of this in one of the open benchmarks I looked at. And I believe this has all been corrected since then. This is all about a year old. There was a multi-language benchmark that tested how good a model is across multiple languages. And I opened up the source set for it.

There was a great example of this in one of the open benchmarks I looked at. And I believe this has all been corrected since then. This is all about a year old. There was a multi-language benchmark that tested how good a model is across multiple languages. And I opened up the source set for it.

and looked at some of the Go code, because I'm a Go programmer, and it had been machine translated from Python so that all of the variable names in this Go code used underscores instead of camel case. And, you know, the models were getting a certain percentage success rate generating these results.

and looked at some of the Go code, because I'm a Go programmer, and it had been machine translated from Python so that all of the variable names in this Go code used underscores instead of camel case. And, you know, the models were getting a certain percentage success rate generating these results.

So Josh went through actually and made these more idiomatic in the go style of using camel case and, you know, putting everything in the right place. And the model gave much better results on this benchmark. And so that's an example of where languages beyond the basic ones that the developers of the models care about are not being paid as much attention to as what you would like.

So Josh went through actually and made these more idiomatic in the go style of using camel case and, you know, putting everything in the right place. And the model gave much better results on this benchmark. And so that's an example of where languages beyond the basic ones that the developers of the models care about are not being paid as much attention to as what you would like.

And things are getting a lot better there. The models are much more sophisticated. The teams building them are much larger. They care about a larger set of languages. And so I don't think it's all as Python-centric as it used to be. But that is still very much the first and most important of the languages. As for how well Go works, it seems to work pretty well.

And things are getting a lot better there. The models are much more sophisticated. The teams building them are much larger. They care about a larger set of languages. And so I don't think it's all as Python-centric as it used to be. But that is still very much the first and most important of the languages. As for how well Go works, it seems to work pretty well.

Models are good at it by our benchmarks. Like we said, if we took the benchmarks and made them more Go-like, the models actually got better results. They have a real tendency to understand the language. We think it's a pretty good fit. There are definitely...

Models are good at it by our benchmarks. Like we said, if we took the benchmarks and made them more Go-like, the models actually got better results. They have a real tendency to understand the language. We think it's a pretty good fit. There are definitely...

There are definitely times when models struggle, but it's a garbage-collected language, which helps, because in just the same way that garbage collection reduces the cognitive load for programmers as they're writing programs, it reduces the load on the LLM in just the same way.

There are definitely times when models struggle, but it's a garbage-collected language, which helps, because in just the same way that garbage collection reduces the cognitive load for programmers as they're writing programs, it reduces the load on the LLM in just the same way.

They don't have to track the state of memory and when to free it, so they have a bit more thinking time to worry about solving your problem. So in that way, it's a good language. It's not too syntax heavy, but it's also, it doesn't have ambiguities that humans struggle with. Yeah. It seems to work well. Yeah. There aren't a lot of.

They don't have to track the state of memory and when to free it, so they have a bit more thinking time to worry about solving your problem. So in that way, it's a good language. It's not too syntax heavy, but it's also, it doesn't have ambiguities that humans struggle with. Yeah. It seems to work well. Yeah. There aren't a lot of.

I haven't seen much research into what is the best language for an LLM. It does seem like an eminently testable thing. Like there's some interesting, in fact, it may end up influencing programming language design in a sense of imagine you are building a new programming language.

I haven't seen much research into what is the best language for an LLM. It does seem like an eminently testable thing. Like there's some interesting, in fact, it may end up influencing programming language design in a sense of imagine you are building a new programming language.