Michael Truel

And then also, there have been some custom systems that we've built, like, for instance, our retrieval system for computing a semantic index of your codebase and answering questions about a codebase that have continually, I feel like, been one of the trickier things to scale.

And then also, there have been some custom systems that we've built, like, for instance, our retrieval system for computing a semantic index of your codebase and answering questions about a codebase that have continually, I feel like, been one of the trickier things to scale.

And then also, there have been some custom systems that we've built, like, for instance, our retrieval system for computing a semantic index of your codebase and answering questions about a codebase that have continually, I feel like, been one of the trickier things to scale.

It's tricky. I think we can do a lot better at computing the context automatically in the future. One thing that's important to note is there are trade-offs with including automatic context.

It's tricky. I think we can do a lot better at computing the context automatically in the future. One thing that's important to note is there are trade-offs with including automatic context.

It's tricky. I think we can do a lot better at computing the context automatically in the future. One thing that's important to note is there are trade-offs with including automatic context.

So the more context you include for these models, first of all, the slower they are and the more expensive those requests are, which means you can then do less model calls and do less fancy stuff in the background. Also, for a lot of these models, they get confused if you have a lot of information in the prompt.

So the more context you include for these models, first of all, the slower they are and the more expensive those requests are, which means you can then do less model calls and do less fancy stuff in the background. Also, for a lot of these models, they get confused if you have a lot of information in the prompt.

So the more context you include for these models, first of all, the slower they are and the more expensive those requests are, which means you can then do less model calls and do less fancy stuff in the background. Also, for a lot of these models, they get confused if you have a lot of information in the prompt.

So the bar for accuracy and for relevance of the context you include should be quite high. But already we do some automatic context in some places within the product. It's definitely something we want to get a lot better at. And I think that there are a lot of cool ideas to try there, both on the learning better retrieval systems, like better embedding models, better re-rankers.

So the bar for accuracy and for relevance of the context you include should be quite high. But already we do some automatic context in some places within the product. It's definitely something we want to get a lot better at. And I think that there are a lot of cool ideas to try there, both on the learning better retrieval systems, like better embedding models, better re-rankers.

So the bar for accuracy and for relevance of the context you include should be quite high. But already we do some automatic context in some places within the product. It's definitely something we want to get a lot better at. And I think that there are a lot of cool ideas to try there, both on the learning better retrieval systems, like better embedding models, better re-rankers.

I think that there are also cool academic ideas, stuff we've tried out internally, but also the field is grappling with writ large, about can you get language models to a place where you can actually just have the model itself, like understand a new corpus of information, And the most popular talked about version of this is, can you make the context windows infinite?

I think that there are also cool academic ideas, stuff we've tried out internally, but also the field is grappling with writ large, about can you get language models to a place where you can actually just have the model itself, like understand a new corpus of information, And the most popular talked about version of this is, can you make the context windows infinite?

I think that there are also cool academic ideas, stuff we've tried out internally, but also the field is grappling with writ large, about can you get language models to a place where you can actually just have the model itself, like understand a new corpus of information, And the most popular talked about version of this is, can you make the context windows infinite?

Then if you make the context windows infinite, can you make the model actually pay attention to the infinite context? And then after you can make it pay attention to the infinite context, to make it somewhat feasible to actually do it, can you then do caching for that infinite context? You don't have to recompute that all the time.

Then if you make the context windows infinite, can you make the model actually pay attention to the infinite context? And then after you can make it pay attention to the infinite context, to make it somewhat feasible to actually do it, can you then do caching for that infinite context? You don't have to recompute that all the time.

Then if you make the context windows infinite, can you make the model actually pay attention to the infinite context? And then after you can make it pay attention to the infinite context, to make it somewhat feasible to actually do it, can you then do caching for that infinite context? You don't have to recompute that all the time.

But there are other cool ideas that are being tried that are a little bit more analogous to fine-tuning of actually learning this information and the weights of the model. And it might be that you actually get sort of a qualitatively different type of understanding if you do it more at the weight level than if you do it at the in-context learning level.

But there are other cool ideas that are being tried that are a little bit more analogous to fine-tuning of actually learning this information and the weights of the model. And it might be that you actually get sort of a qualitatively different type of understanding if you do it more at the weight level than if you do it at the in-context learning level.