Arvind Narayanan
speaker
255 appearances
3 recordings
3 series
first heard Aug 2024
last heard 25 Jan
Arvind Narayanan’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 1 in all, peaking in Jan 2026 with 1.
Appearances
And cost, of course, is roughly proportional to the size of the model. And that's putting a lot of downward pressure on model size.
And once you get a model small enough that you can run it on device, that of course opens up a lot of new possibilities, both in terms of privacy, you know, people are much more comfortable with on device models, especially if it's something that's going to be listening to their phone conversations or looking at their desktop screenshots, which are exactly the kinds of AI assistance that companies are building and pushing.
And just from the perspective of cost, you don't have to dedicate servers to run that model. So I think those are a lot of the reasons why companies are furiously working on making models smaller without a big hit in capability.
You're right. Cost is going down dramatically. In certain applications, cost is going to become much less of a barrier, but not across the board.
So there's this interesting concept called Jevons Paradox. And this was first in the context of coal in England in the 18th century. I think when coal mining got cheaper, there was more demand for coal. And so the amount invested into coal mining actually increased. And I predict that we're going to see the same thing with models. When models get cheaper, they're put into a lot more things.
And so the total amount that companies are spending on inference is actually going to increase. In an application like a chatbot, let's say, you know, it's text in, text out, no big deal. I think costs are going to come down. Even if someone is chatting with a chatbot all day, it's probably not going to get too expensive.
On the other hand, if you want to scan all of someone's emails, for instance, right? If a model gets cheaper, you know, you're just going to have it running always on in the background. And then from emails, you're going to get to all their documents, right? And some of those attachments might be many megabytes long.
And so there, even with Moore's law, I think cost is going to be significant in the medium term. And then you get to applications like writing code, where what we're seeing is that it's actually very beneficial to let the model do the same task tens of times, thousands of times, sometimes literally millions of times and pick the best answer.
So in those cases, it doesn't matter how much cost goes down. You're going to just proportionally increase the number of retries so that you can get a better quality of output.
So there is training compute, which is when the developer is building the model. And then there is inference compute, when the model is being deployed and the user is using it to do something. And it might seem like really the training cost is the one we should worry about, since it's trained on all of the text on the internet or whatever.
But it turns out that over the lifetime of a model, when you have billions of people using it, the inference cost actually adds up. And for many of the popular models, that's the cost that dominates. So let's talk about each of those two costs.
With respect to training costs, if you want to build a smaller model at the same level of capability or without compromising capability too much, you have to actually train it for longer. So that increases training costs. But that's maybe okay because you have a smaller model. You can push it to the consumer device or even if it's running on the cloud, your server costs are lower.
So your training cost increases, your inference cost decreases. But because it's the inference cost that dominates, the total cost is probably going to come down. So total cost comes down. If you have the same workload and you have a smaller model doing it, then the total cost is going to come down.
Sure. I think we are still in a period where, you know, these models have not yet quite become commoditized. There's obviously a lot of progress and there's a lot of demand on hardware as well. Hardware cycles are also improving rapidly. But, you know, there's the saying that every exponential is a sigmoid in disguise. So a sigmoid curve is one that looks like an exponential at the beginning.
So imagine the S letter shape. But then after a while, it has to taper off like every exponential has to taper off. So I think that's going to happen both with models as well as with these hardware cycles. We are, I think, going to get to a world where models do get commoditized.
A big part of it is this issue of vibes, right? So you evaluate LLMs on these benchmarks, but then it seems to perform really well on the benchmarks, but then the vibes are off. In other words, you start using it and somehow it doesn't feel adequate. It makes a lot of mistakes in ways that are not captured in the benchmark.
And the reason for that is simply that when there is so much pressure to do well on these benchmarks, developers are intentionally or unintentionally optimizing these models in ways that look good on the benchmarks, but don't look good in real world evaluation.
So when GPT-4 came out and OpenAI claimed that it passed the bar exam and the medical licensing exam, people were very excited slash scared about what this means for doctors and lawyers. And the answer turned out to be approximately nothing. Because it's not like a lawyer's job is to answer bar exam questions all day.
These benchmarks that models are being tested on don't really capture what we would use them for in the real world. So that's one reason why LLM evaluation is a minefield. And there's also just a very simple factor of contamination. Maybe the model has already trained on the answers that it's being evaluated on in the benchmark. And so if you ask it new questions, it's going to struggle.
We shouldn't put too much stock into benchmarks. We should look at people... We're actually trying to use these in professional context, whether it's lawyers or, you know, really anybody else. And we should go based on their experience of using these AI assistants.
Showing 161–180 of 255 · page 9 of 13
← Previous
Next →