Nathan Lambert

It's a mix of better data, and it's going to be better training techniques, and all of these better inference systems uh, better hardware, right. Uh, going from, you know, each generation of GPU to new generations or a six, everything is going to take this cost curve down and down and down and down.

And then can I go in, can I just spawn a thousand different LLMs to create a task and then pick from one of them or, you know, whatever search search technique I want a tree Monte Carlo tree search. Maybe it gets that complicated. Um, maybe it doesn't cause it's too complicated to actually scale. Like who knows a better lesson, right? Uh, the, the question is, is,

And then can I go in, can I just spawn a thousand different LLMs to create a task and then pick from one of them or, you know, whatever search search technique I want a tree Monte Carlo tree search. Maybe it gets that complicated. Um, maybe it doesn't cause it's too complicated to actually scale. Like who knows a better lesson, right? Uh, the, the question is, is,

And then can I go in, can I just spawn a thousand different LLMs to create a task and then pick from one of them or, you know, whatever search search technique I want a tree Monte Carlo tree search. Maybe it gets that complicated. Um, maybe it doesn't cause it's too complicated to actually scale. Like who knows a better lesson, right? Uh, the, the question is, is,

I think when, not if, because the rate of progress is so fast, right? Nine months ago, Dario was saying, or Dario said nine months ago, the cost to train and inference was this, right? And now we're much better than this, right? And DeepSeek is much better than this.

I think when, not if, because the rate of progress is so fast, right? Nine months ago, Dario was saying, or Dario said nine months ago, the cost to train and inference was this, right? And now we're much better than this, right? And DeepSeek is much better than this.

I think when, not if, because the rate of progress is so fast, right? Nine months ago, Dario was saying, or Dario said nine months ago, the cost to train and inference was this, right? And now we're much better than this, right? And DeepSeek is much better than this.

And that cost curve for GPT-4, which was also roughly $60 per million tokens when it launched, has already fallen to $2 or so, right? And we're going to get it down to cents, Probably. For GPT-4 quality, and then that's the base for the reasoning models like O1 that we have today, and O1 Pro is spawning multiple, right? And O3 and so on and so forth.

And that cost curve for GPT-4, which was also roughly $60 per million tokens when it launched, has already fallen to $2 or so, right? And we're going to get it down to cents, Probably. For GPT-4 quality, and then that's the base for the reasoning models like O1 that we have today, and O1 Pro is spawning multiple, right? And O3 and so on and so forth.

And that cost curve for GPT-4, which was also roughly $60 per million tokens when it launched, has already fallen to $2 or so, right? And we're going to get it down to cents, Probably. For GPT-4 quality, and then that's the base for the reasoning models like O1 that we have today, and O1 Pro is spawning multiple, right? And O3 and so on and so forth.

These search techniques, too expensive today, but they will get cheaper. And that's what's going to unlock the intelligence, right?

These search techniques, too expensive today, but they will get cheaper. And that's what's going to unlock the intelligence, right?

These search techniques, too expensive today, but they will get cheaper. And that's what's going to unlock the intelligence, right?

I think I think and like there are a lot of false narratives, which is like, hey, these guys are spending billions on models. Right. And they're not spending billions on models. No one spent more than a billion dollars on a model that's released publicly. Right. GPT-4 was a couple hundred million. And then, you know, they've reduced the cost with 4.0, 4 turbo 4.0. Right.

I think I think and like there are a lot of false narratives, which is like, hey, these guys are spending billions on models. Right. And they're not spending billions on models. No one spent more than a billion dollars on a model that's released publicly. Right. GPT-4 was a couple hundred million. And then, you know, they've reduced the cost with 4.0, 4 turbo 4.0. Right.

I think I think and like there are a lot of false narratives, which is like, hey, these guys are spending billions on models. Right. And they're not spending billions on models. No one spent more than a billion dollars on a model that's released publicly. Right. GPT-4 was a couple hundred million. And then, you know, they've reduced the cost with 4.0, 4 turbo 4.0. Right.

But billion dollar model runs are coming. right? And this concludes pre-training and post-training, right? And then the other number is like, hey, DeepSeq didn't include everything, right? They didn't include... A lot of the cost goes to research and all this sort of stuff. A lot of the cost goes to inference. A lot of the cost goes to post-training. None of these things were factored.

But billion dollar model runs are coming. right? And this concludes pre-training and post-training, right? And then the other number is like, hey, DeepSeq didn't include everything, right? They didn't include... A lot of the cost goes to research and all this sort of stuff. A lot of the cost goes to inference. A lot of the cost goes to post-training. None of these things were factored.

But billion dollar model runs are coming. right? And this concludes pre-training and post-training, right? And then the other number is like, hey, DeepSeq didn't include everything, right? They didn't include... A lot of the cost goes to research and all this sort of stuff. A lot of the cost goes to inference. A lot of the cost goes to post-training. None of these things were factored.

It's research salaries, right? All these things are counted in the billions of dollars that OpenAI is spending, but they weren't counted in the, hey, $6 million, $5 million that DeepSeq spent, right? So there's a bit of misunderstanding of what these numbers are. And then there's also an element of NVIDIA has just been a straight line up, right?