Nathan Lambert

I want to train there because that's where all of my GPUs are co-located, where I can put them at a super high networking speed connected together, right? Because that's what you need for training. Now with pre-training, this is the old scale, right? You could, you would increase parameters. You didn't increase data model gets better, right?

I want to train there because that's where all of my GPUs are co-located, where I can put them at a super high networking speed connected together, right? Because that's what you need for training. Now with pre-training, this is the old scale, right? You could, you would increase parameters. You didn't increase data model gets better, right?

That doesn't apply anymore because there's not much more data in the pre-training side. Yes, there's video and audio and image that has not been fully taken advantage of. So there's a lot more scaling. But a lot of people have taken transcripts of YouTube videos. And that gets you a lot of the data. It doesn't get you all the learning value out of the video and image data.

That doesn't apply anymore because there's not much more data in the pre-training side. Yes, there's video and audio and image that has not been fully taken advantage of. So there's a lot more scaling. But a lot of people have taken transcripts of YouTube videos. And that gets you a lot of the data. It doesn't get you all the learning value out of the video and image data.

That doesn't apply anymore because there's not much more data in the pre-training side. Yes, there's video and audio and image that has not been fully taken advantage of. So there's a lot more scaling. But a lot of people have taken transcripts of YouTube videos. And that gets you a lot of the data. It doesn't get you all the learning value out of the video and image data.

But there's still scaling to be done on pre-training. But this post-training world is where all the flops are going to be spent, right? The model is going to play with itself. It's going to self-play. It's going to do verifiable tasks. It's going to do computer use in sandboxes. It might even do like simulated robotics things, right?

But there's still scaling to be done on pre-training. But this post-training world is where all the flops are going to be spent, right? The model is going to play with itself. It's going to self-play. It's going to do verifiable tasks. It's going to do computer use in sandboxes. It might even do like simulated robotics things, right?

But there's still scaling to be done on pre-training. But this post-training world is where all the flops are going to be spent, right? The model is going to play with itself. It's going to self-play. It's going to do verifiable tasks. It's going to do computer use in sandboxes. It might even do like simulated robotics things, right?

Like all of these things are going to be environments where compute is spent in quote unquote post-training. But I think it's going to be good. We're going to drop the post from post-training. It's going to be pre-training and it's going to be training, I think. At some point. Because for the bulk of the last few years, pre-training has dwarfed post-training.

Like all of these things are going to be environments where compute is spent in quote unquote post-training. But I think it's going to be good. We're going to drop the post from post-training. It's going to be pre-training and it's going to be training, I think. At some point. Because for the bulk of the last few years, pre-training has dwarfed post-training.

Like all of these things are going to be environments where compute is spent in quote unquote post-training. But I think it's going to be good. We're going to drop the post from post-training. It's going to be pre-training and it's going to be training, I think. At some point. Because for the bulk of the last few years, pre-training has dwarfed post-training.

But with these verifiable methods, especially ones that scale really potentially infinitely, like computer use and robotics, not just math and coding, where you can verify what's happening, those infinitely verifiable tasks, it seems you can spend as much compute as you want on them. Especially at the context length increase context.

But with these verifiable methods, especially ones that scale really potentially infinitely, like computer use and robotics, not just math and coding, where you can verify what's happening, those infinitely verifiable tasks, it seems you can spend as much compute as you want on them. Especially at the context length increase context.

But with these verifiable methods, especially ones that scale really potentially infinitely, like computer use and robotics, not just math and coding, where you can verify what's happening, those infinitely verifiable tasks, it seems you can spend as much compute as you want on them. Especially at the context length increase context.

I was like, huh?

I was like, huh?

I was like, huh?

TPU is awesome, right? It's great. Google is... They're a bit more tepid on building data centers for some reason. They're building big data centers, don't get me wrong. And they actually have the biggest cluster. I was talking about NVIDIA clusters. They actually have the biggest cluster, period. But the way they do it is very interesting, right? They have two data center...

TPU is awesome, right? It's great. Google is... They're a bit more tepid on building data centers for some reason. They're building big data centers, don't get me wrong. And they actually have the biggest cluster. I was talking about NVIDIA clusters. They actually have the biggest cluster, period. But the way they do it is very interesting, right? They have two data center...

TPU is awesome, right? It's great. Google is... They're a bit more tepid on building data centers for some reason. They're building big data centers, don't get me wrong. And they actually have the biggest cluster. I was talking about NVIDIA clusters. They actually have the biggest cluster, period. But the way they do it is very interesting, right? They have two data center...