Bret Taylor
speaker
184 appearances
1 recordings
1 series
first heard Oct 2024
last heard Oct 2024
Bret Taylor’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsNo recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.
Appearances
bad at, for good reason, I don't think it's necessarily what we do, is actually helping companies manage the adoption of this technology. Most software companies try to be trusted advisors to their companies, but at the end of the day, they have a vested interest in the product that they're selling. And it often helps to have a third party there to help you actually manage that change.
So I do think that there's probably some short-term professional services spend that reflects the lack of the maturity of the AI applications market right now. When there are solutions like Sierra and others for specific domains available, you shouldn't have to spend as much to deploy those effectively in your business.
However, I think that as AI changes and disrupts the way companies operate, you know, I would hope that the best professional services firms have consulting arms that can help companies with that change management and it might compensate it for a different way. So I think if you itemize the receipts, the revenue might change over time.
let's start with the high level i really like reid hoffman's framing of this market as foundation models and frontier models so foundation models are any of these large language models that aren't necessarily the best of the best or the higher highest parameter count but particularly now where you have relatively low parameter count models that are meter exceed the quality of say gpt 3.5
That market of foundation models is quite important and quite commoditized in that market. If you need a model like that, you should download Lama. That's the answer. It's like you don't need much of a cheat sheet on that, you know, or maybe Mistral, but pick one of the open source models that are adequate and fine tune it.
The frontier model market is a little different when you talk about this, the experience you've had being dizzy using these tools. My perspective is that we've seen real leaps there. So when ChatGPT came out, that was a meaningful step function change that lasted for a while.
And the insight around instruction tuning and the quality of sort of the GPT models after GPT-3 was pretty remarkably different. Similarly, when GPT-4 came out, I haven't done the math on it, but it certainly had a meaningful lead for quite a while. And now you're seeing a lot of models sort of catch up to that.
My sense is we see a lot of incremental improvement followed by step changes in quality. But going back to the market itself, I am inherently skeptical of companies doing pre-training, unless you are an AGI research lab. Doing pre-training on a model, I believe, is just burning capital.
It's roughly the equivalent of an entrepreneur coming to you and saying, you know, we're building this software solution, and the first thing we're going to do is build our data center by hand.
And I think for 99% of software companies, they should lease their servers from an infrastructure as a service provider, not because it's the most vertically integrated and efficient, but because it's not what their company does. Similarly, as you're exploring and finding product market fit, the last thing you want to do is have a big upfront investment to build a data center.
There was a number of companies that were started by incredibly talented AI researchers. And, you know, step one of their product plan was build, pre-train a model. And I think for especially with the existence of these high quality models like, you know, GPT-4.0,
mini that you can fine tune or the open source models like Lama 3.1 to spend capital on pre-training now, unless you're one of the behemoths, I think is nonsensical.
Do you remember in the early 90s, it went from like Windows NT to 95 to 98 to 2000, you know, or something like that. I might be mixing it up. So, you know, we could pull that out to start changing numbers up.
are distinct to me. So starting with the step function, I don't think it's a foregone conclusion that we'll have step function changes. I believe the most responsible way to develop AGI is responsible iterative deployment. The reason for that is I believe that as you're thinking about things like the societal impact, access to this technology and the safety side of AGI as well, that the
best way we can learn about how to ensure that these models benefit humanity is to consistently release them, learn from those experiences on the safety side, learn about the harm, learn about really specific vulnerabilities like jailbreaking and improve it at every turn. We could end up with a plateau of progress, or as you said, diminishing returns. The three inputs to progress in AI are
Number one, data. Number two, compute. Number three, algorithms and methodology. So if you look at the short history of sort of this current wave of modern AI, it started, I think, with the Transformers model, which was a paper from Google called Attention is All You Need, which changed the scale with which you could build these models, which led to many of the sort of
GPT breakthroughs that came next. You ended up with instruction tuning, which was how you turned one of these models into a chat interface, which was a breakthrough as well. Given even existing data, existing compute, we have all of the best minds in computer science thinking about different techniques. It's similar. There's folks even looking beyond the Transformers model and things like that.
So I think that that's one area where you could have a a big breakthrough. You have compute, just pick up a newspaper and read about the investment in GPUs. And these clusters are getting even bigger and bigger. And even with the same amount of data training and both pre-training and post-training can have a really big impact on quality
And then on the data side, there's a lot of writing about sort of running out of some of the textual data. But there's a lot of really interesting companies working on simulation. There's a lot of interesting explorations in synthetic data generation. There's multimodality. So, you know, what is true of text is, you know, there's lots of video, audio, image content as well.
you know, in any one of those, you could probably make a very rational intellectual case that we're going to hit a wall, but then you have the two others. And I don't think you can make the case for all three that they're all coming up on a wall. And I think like any big scientific effort, it will probably be a mix of progress and all of those. And as a consequence, I
Showing 41–60 of 184 · page 3 of 10
← Previous
Next →