Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
I think that's an interesting question.
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
Well, I think at some point,
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
Over the last few years, I think there's this issue of GPU production.
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
So even companies that had the models, sorry, that had the money to pay for the GPUs couldn't necessarily get as many as they wanted because there were all these supply constraints.
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
Now I think that's sort of getting less.
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
So now I think you're seeing a bunch of companies think about, wow, we should just like really invest a lot of money in building out these things.
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
And I think that that will go for some period of time.
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
I think there is a capital question of like, okay,
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
at what point does it stop being worth it to put the capital in?
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
But I actually think before we hit that, you're going to run into energy constraints, right?
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
Because I just, I mean, I don't think anyone's built a gigawatt single training cluster yet, right?
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
And then you run into these things that just end up being slower in the world.
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
Like,
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
getting energy permitted is like a very heavily regulated government function, right?
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
So you're going from, on the one hand, software, which is somewhat regulated.
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
I'd argue that it is more regulated than I think a lot of people in the tech community feel, although it's obviously different.
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
If you're starting a small company, maybe you feel that less.
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
If you're a big company, we just interact with people.
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
Different governments and regulators are, you know, we have kind of...
Dwarkesh Podcast
Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus
Lots of rules that we need to kind of follow and make sure we do a good job with around the world.