George Hotz

But that obviously is not at all how Skyrim was actually created. It was created by a bunch of programmers in a room, right? So, like, you know, it struck me one day how just silly atheism is. Like, of course we were created by God.

But that obviously is not at all how Skyrim was actually created. It was created by a bunch of programmers in a room, right? So, like, you know, it struck me one day how just silly atheism is. Like, of course we were created by God.

It's the most obvious thing.

It's the most obvious thing.

Yeah. And then like, I also just like, I like that notion. That notion gives me a lot of, I mean, I guess you can talk about what it gives a lot of religious people. It's kind of like, it just gives me comfort. It's like, you know what? If we mess it all up and we die out. Yeah.

Yeah. And then like, I also just like, I like that notion. That notion gives me a lot of, I mean, I guess you can talk about what it gives a lot of religious people. It's kind of like, it just gives me comfort. It's like, you know what? If we mess it all up and we die out. Yeah.

You know, people will come up with, like, well, yeah, but, like, man, who created God?

You know, people will come up with, like, well, yeah, but, like, man, who created God?

I'm like, that's God's problem. You know? Like, I'm not going to think this is. You're asking me if God believes in God?

I'm like, that's God's problem. You know? Like, I'm not going to think this is. You're asking me if God believes in God?

I mean, to be fair, if God didn't believe in God, he'd be as silly as the atheists here.

I mean, to be fair, if God didn't believe in God, he'd be as silly as the atheists here.

Is mid good or bad? Mid is bad. It's like mid, it's like.

Is mid good or bad? Mid is bad. It's like mid, it's like.

I have not played Diablo 4.

I have not played Diablo 4.

All right.

All right.

I'm going to say World of Warcraft. And it's not that the game is such a great game. It's not. It's that I remember in 2005 when it came out, how it opened my mind to ideas. It opened my mind to this whole world we've created, right? And there's almost been nothing like it since 2005. Like, you can look at MMOs today, and I think they all have lower user bases than World of Warcraft.

I'm going to say World of Warcraft. And it's not that the game is such a great game. It's not. It's that I remember in 2005 when it came out, how it opened my mind to ideas. It opened my mind to this whole world we've created, right? And there's almost been nothing like it since 2005. Like, you can look at MMOs today, and I think they all have lower user bases than World of Warcraft.

Like, EVE Online's kind of cool. But to think that, like, everyone knows, you know, people are always, like, they look at the Apple headset, like... What do people want in this VR? Everyone knows what they want. I want Ready Player One. And like that. So I'm going to say World of Warcraft. And I'm hoping that games can get out of this whole mobile gaming dopamine pump thing.

Like, EVE Online's kind of cool. But to think that, like, everyone knows, you know, people are always, like, they look at the Apple headset, like... What do people want in this VR? Everyone knows what they want. I want Ready Player One. And like that. So I'm going to say World of Warcraft. And I'm hoping that games can get out of this whole mobile gaming dopamine pump thing.

Yeah, and I think it'll come back. I believe.

Yeah, and I think it'll come back. I believe.

They exist in real life, too.

They exist in real life, too.

I wish it was that cool.

I wish it was that cool.

It's like middle of the curve. There's that intelligence curve. You have the dumb guy, the smart guy, and then the mid guy. Actually, being the mid guy is the worst. The smart guy is like, I put all my money in Bitcoin.

It's like middle of the curve. There's that intelligence curve. You have the dumb guy, the smart guy, and then the mid guy. Actually, being the mid guy is the worst. The smart guy is like, I put all my money in Bitcoin.

What I'm really excited about in games is like once we start getting intelligent AIs to interact with.

What I'm really excited about in games is like once we start getting intelligent AIs to interact with.

Like the NPCs in games have never been.

Like the NPCs in games have never been.

In like, yeah, in like every way. Like when you're actually building a world and a world imbued with intelligence. Oh yeah. Right. And it's just hard. Like there's just like, like, you know, running world of Warcraft, like you're limited by what you're running on a Pentium four, you know, how much intelligence can you run? How many flops did you have? Right.

In like, yeah, in like every way. Like when you're actually building a world and a world imbued with intelligence. Oh yeah. Right. And it's just hard. Like there's just like, like, you know, running world of Warcraft, like you're limited by what you're running on a Pentium four, you know, how much intelligence can you run? How many flops did you have? Right.

But now when I'm running a game on a hundred pay to flop machine, that's five people. I'm trying to make this a thing. 20 petaflops of compute is one person of compute. I'm trying to make that a unit.

But now when I'm running a game on a hundred pay to flop machine, that's five people. I'm trying to make this a thing. 20 petaflops of compute is one person of compute. I'm trying to make that a unit.

It's like a horsepower. What's a horsepower? It's how powerful a horse is. What's a person of compute?

It's like a horsepower. What's a horsepower? It's how powerful a horse is. What's a person of compute?

You know what? Border Quest 2. I put it on and I can't believe the first thing they show me is a bunch of scrolling clouds and a Facebook login screen. You had the ability to bring me into a world. And what did you give me? A pop-up, right? And this is why you're not cool, Mark Zuckerberg. But you could be cool.

You know what? Border Quest 2. I put it on and I can't believe the first thing they show me is a bunch of scrolling clouds and a Facebook login screen. You had the ability to bring me into a world. And what did you give me? A pop-up, right? And this is why you're not cool, Mark Zuckerberg. But you could be cool.

Just make sure on the Quest 3, you don't put me into clouds and a Facebook login screen. Bring me to a world.

Just make sure on the Quest 3, you don't put me into clouds and a Facebook login screen. Bring me to a world.

I got to play that from the beginning. I played it for like an hour at a friend's house.

I got to play that from the beginning. I played it for like an hour at a friend's house.

The mid guy is like, you can't put money in Bitcoin. It's not real money.

The mid guy is like, you can't put money in Bitcoin. It's not real money.

I'm going to go buy a Switch. I'm going to go today and buy a Switch.

I'm going to go buy a Switch. I'm going to go today and buy a Switch.

Is it pass-through or cameras?

Is it pass-through or cameras?

The Apple one, is that one pass-through or cameras?

The Apple one, is that one pass-through or cameras?

Some point. Maybe not January.

Some point. Maybe not January.

Maybe that's my optimism. But Apple, I will buy it. I don't care if it's expensive and does nothing. I will buy it. I will support this future endeavor.

Maybe that's my optimism. But Apple, I will buy it. I don't care if it's expensive and does nothing. I will buy it. I will support this future endeavor.

You know what? And this is another place we'll give some more respect to Mark Zuckerberg. The two companies that have endured through technology are Apple and Microsoft. And what do they make? Computers and business services.

You know what? And this is another place we'll give some more respect to Mark Zuckerberg. The two companies that have endured through technology are Apple and Microsoft. And what do they make? Computers and business services.

All the memes, social ads, they all come and go.

All the memes, social ads, they all come and go.

But you want to endure, build hardware.

But you want to endure, build hardware.

And that's why it's more important than ever that the AI is running on those systems are aligned with you. Oh, yeah. They're going to augment your entire world. Oh, yeah.

And that's why it's more important than ever that the AI is running on those systems are aligned with you. Oh, yeah. They're going to augment your entire world. Oh, yeah.

There's two directions the AI girlfriend company can take, right? There's like the highbrow, something like her, maybe something you kind of talk to. And this is, and then there's the lowbrow version of it where I want to set up a brothel in Times Square.

There's two directions the AI girlfriend company can take, right? There's like the highbrow, something like her, maybe something you kind of talk to. And this is, and then there's the lowbrow version of it where I want to set up a brothel in Times Square.

Yeah. It's not cheating if it's a robot. It's a VR experience.

Yeah. It's not cheating if it's a robot. It's a VR experience.

No, I don't want to do that one or that one.

No, I don't want to do that one or that one.

We'll see what the technology goes.

We'll see what the technology goes.

There's a lot to do in company number two. I'm just like, I'm talking about company number three now.

There's a lot to do in company number two. I'm just like, I'm talking about company number three now.

None of that tech exists yet. There's a lot to do in company number two. Company number two is going to be the great struggle of the next six years. And of the next six years, how centralized is compute going to be? The less centralized compute is going to be, the better of a chance we all have.

None of that tech exists yet. There's a lot to do in company number two. Company number two is going to be the great struggle of the next six years. And of the next six years, how centralized is compute going to be? The less centralized compute is going to be, the better of a chance we all have.

We have to. We have to, or they will just completely dominate us. I showed a picture on stream of a man in a chicken farm. You ever seen one of those factory farm chicken farms? Why does he dominate all the chickens? Why does he- Smarter. He's smarter, right? Some people on Twitch were like, he's bigger than the chickens. Yeah. And now here's a man in a cow farm. Right?

We have to. We have to, or they will just completely dominate us. I showed a picture on stream of a man in a chicken farm. You ever seen one of those factory farm chicken farms? Why does he dominate all the chickens? Why does he- Smarter. He's smarter, right? Some people on Twitch were like, he's bigger than the chickens. Yeah. And now here's a man in a cow farm. Right?

So it has nothing to do with their size and everything to do with their intelligence. And if one central organization has all the intelligence, you'll be the chickens and they'll be the chicken man. But if we all have the intelligence, we're all the chickens. We're not all the man, we're all the chickens.

So it has nothing to do with their size and everything to do with their intelligence. And if one central organization has all the intelligence, you'll be the chickens and they'll be the chicken man. But if we all have the intelligence, we're all the chickens. We're not all the man, we're all the chickens.

And there's no chicken man.

And there's no chicken man.

He was having a good life, man.

He was having a good life, man.

I want to make sure it's good. I want to make sure that like the thing that I deliver is like not going to be like a quest to which you buy and use twice. I mean, it's better than a quest, which you bought and used less than once statistically.

I want to make sure it's good. I want to make sure that like the thing that I deliver is like not going to be like a quest to which you buy and use twice. I mean, it's better than a quest, which you bought and used less than once statistically.

I think that we're going to get super scary memes once the AIs actually are superhuman.

I think that we're going to get super scary memes once the AIs actually are superhuman.

The longest time at Comma, I asked, why did I start a company? Why did I do this? What else was I going to do?

The longest time at Comma, I asked, why did I start a company? Why did I do this? What else was I going to do?

With Kama, it really started as an ego battle with Elon. I wanted to beat him. I saw a worthy adversary. Here's a worthy adversary who I can beat at self-driving cars. And I think we've kept pace, and I think he's kept ahead. I think that's what's ended up happening there. But I do think Kama is... I mean, Kama's profitable. Like... And like when this drive GPT stuff starts working, that's it.

With Kama, it really started as an ego battle with Elon. I wanted to beat him. I saw a worthy adversary. Here's a worthy adversary who I can beat at self-driving cars. And I think we've kept pace, and I think he's kept ahead. I think that's what's ended up happening there. But I do think Kama is... I mean, Kama's profitable. Like... And like when this drive GPT stuff starts working, that's it.

There's no more like bugs in the loss function. Like right now we're using like a hand-coded simulator. There's no more bugs. This is going to be it. Like this is the run up to driving.

There's no more like bugs in the loss function. Like right now we're using like a hand-coded simulator. There's no more bugs. This is going to be it. Like this is the run up to driving.

It's so, it's better than FSD and Autopilot in certain ways. It has a lot more to do with which feel you like. We lowered the price on the hardware to $1499. You know how hard it is to ship reliable consumer electronics that go on your windshield? We're doing more than most cell phone companies.

It's so, it's better than FSD and Autopilot in certain ways. It has a lot more to do with which feel you like. We lowered the price on the hardware to $1499. You know how hard it is to ship reliable consumer electronics that go on your windshield? We're doing more than most cell phone companies.

I know. I have an SMT line. I make all the boards in-house in San Diego.

I know. I have an SMT line. I make all the boards in-house in San Diego.

Our head of open pilot is great at like, you know, okay, I want all the commentaries to be identical. Yeah. And yeah, I mean, you know, look, it's $14.99. 30-day money back guarantee. It will blow your mind at what it can do. Is it hard to scale? You know what? There's kind of downsides to scaling it. People are always like, why don't you advertise?

Our head of open pilot is great at like, you know, okay, I want all the commentaries to be identical. Yeah. And yeah, I mean, you know, look, it's $14.99. 30-day money back guarantee. It will blow your mind at what it can do. Is it hard to scale? You know what? There's kind of downsides to scaling it. People are always like, why don't you advertise?

I think it's worse than that. So Infinite Jest, it's introduced in the first 50 pages, is about a tape that once you watch it once, you only ever want to watch that tape. In fact, you want to watch the tape so much that someone says, okay, here's a hacksaw, cut off your pinky, and then I'll let you watch the tape again.

I think it's worse than that. So Infinite Jest, it's introduced in the first 50 pages, is about a tape that once you watch it once, you only ever want to watch that tape. In fact, you want to watch the tape so much that someone says, okay, here's a hacksaw, cut off your pinky, and then I'll let you watch the tape again.

Our mission is to solve self-driving cars while delivering shipable intermediaries. Our mission has nothing to do with selling a million boxes. It's tawdry.

Our mission is to solve self-driving cars while delivering shipable intermediaries. Our mission has nothing to do with selling a million boxes. It's tawdry.

Only if I felt someone could accelerate that mission and wanted to keep it open source. And like, not just wanted to, I don't believe what anyone says. I believe incentives. If a company wanted to buy Comma where their incentives were to keep it open source, but Comma doesn't stop at the cars. The cars are just the beginning. The device is a human head. The device has two eyes, two ears.

Only if I felt someone could accelerate that mission and wanted to keep it open source. And like, not just wanted to, I don't believe what anyone says. I believe incentives. If a company wanted to buy Comma where their incentives were to keep it open source, but Comma doesn't stop at the cars. The cars are just the beginning. The device is a human head. The device has two eyes, two ears.

It breathes air. It has a mouth.

It breathes air. It has a mouth.

We sell common bodies too. They're very rudimentary. But one of the problems that we're running into is that the comma three has about as much intelligence as a B. If you want a human's worth of intelligence, you're going to need a tiny rack, not even a tiny box. You're going to need like a tiny rack, maybe even more.

We sell common bodies too. They're very rudimentary. But one of the problems that we're running into is that the comma three has about as much intelligence as a B. If you want a human's worth of intelligence, you're going to need a tiny rack, not even a tiny box. You're going to need like a tiny rack, maybe even more.

You don't. And there's no way you can. You connect to it wirelessly. So you put your tiny box or your tiny rack in your house, and then you get your comma body, and your comma body runs the models on that. It's close, right? You don't have to go to some cloud, which is 30 milliseconds away. You go to a thing, which is 0.1 milliseconds away.

You don't. And there's no way you can. You connect to it wirelessly. So you put your tiny box or your tiny rack in your house, and then you get your comma body, and your comma body runs the models on that. It's close, right? You don't have to go to some cloud, which is 30 milliseconds away. You go to a thing, which is 0.1 milliseconds away.

I mean, eventually, if you fast forward 20, 30 years, the mobile chips will get good enough to run these AIs. But fundamentally, it's not even a question of putting legs on a tiny box because how are you getting 1.5 kilowatts of power on that thing, right? So you need, they're very synergistic businesses. I also want to build all of Comma's training computers.

I mean, eventually, if you fast forward 20, 30 years, the mobile chips will get good enough to run these AIs. But fundamentally, it's not even a question of putting legs on a tiny box because how are you getting 1.5 kilowatts of power on that thing, right? So you need, they're very synergistic businesses. I also want to build all of Comma's training computers.

Comma builds training computers right now. We use commodity parts. I think I can do it cheaper. So we're going to build, TinyCorp is going to not just sell TinyBox. TinyBox is the consumer version, but I'll build training data centers too.

Comma builds training computers right now. We use commodity parts. I think I can do it cheaper. So we're going to build, TinyCorp is going to not just sell TinyBox. TinyBox is the consumer version, but I'll build training data centers too.

He went to work at OpenAI.

He went to work at OpenAI.

Oh man, like, you know, his streams are just a level of quality so far beyond mine. myself like it's just it's just you know yeah he's good he wants to teach you yeah I want to show you that I'm smarter than you

Oh man, like, you know, his streams are just a level of quality so far beyond mine. myself like it's just it's just you know yeah he's good he wants to teach you yeah I want to show you that I'm smarter than you

And he'll do it.

And he'll do it.

Yeah.

Yeah.

So we're actually going to build that, I think. But it's not going to be one static tape. I think the human brain is too complex to be stuck in one static tape like that. If you look at like ant brains, maybe they can be stuck on a static tape. But we're going to build that using generative models. We're going to build the TikTok that you actually can't look away from.

So we're actually going to build that, I think. But it's not going to be one static tape. I think the human brain is too complex to be stuck in one static tape like that. If you look at like ant brains, maybe they can be stuck on a static tape. But we're going to build that using generative models. We're going to build the TikTok that you actually can't look away from.

MicroGrad was, yeah, inspiration for TinyGrad.

MicroGrad was, yeah, inspiration for TinyGrad.

The whole, I mean, his CS231N was, this was the inspiration. This is what I just took and ran with and ended up writing this.

The whole, I mean, his CS231N was, this was the inspiration. This is what I just took and ran with and ended up writing this.

So, you know.

So, you know.

Don't go work for Darth Vader, man.

Don't go work for Darth Vader, man.

I know they are. And that's kind of what's even like more. And you know what? It's not that OpenAI doesn't open source the weights of GPT-4. It's that they go in front of Congress. And that is what upsets me. You know, we had two effective altruist Sams go in front of Congress. One's in jail.

I know they are. And that's kind of what's even like more. And you know what? It's not that OpenAI doesn't open source the weights of GPT-4. It's that they go in front of Congress. And that is what upsets me. You know, we had two effective altruist Sams go in front of Congress. One's in jail.

One's in jail.

One's in jail.

No, I think effective altruism is a terribly evil ideology.

No, I think effective altruism is a terribly evil ideology.

Because you get Sam Bankman Freed. Like, Sam Bankman Freed is the embodiment of effective altruism. Utilitarianism is an abhorrent ideology. Like, well, yeah, we're going to kill those three people to save a thousand, of course. Yeah. Right? There's no underlying, like, there's just, yeah.

Because you get Sam Bankman Freed. Like, Sam Bankman Freed is the embodiment of effective altruism. Utilitarianism is an abhorrent ideology. Like, well, yeah, we're going to kill those three people to save a thousand, of course. Yeah. Right? There's no underlying, like, there's just, yeah.

Oh, well, I think charity is bad, right? So what is charity but investment that you don't expect to have a return on, right?

Oh, well, I think charity is bad, right? So what is charity but investment that you don't expect to have a return on, right?

And probably almost always that involves starting a company.

And probably almost always that involves starting a company.

Yeah. If you just take the money and you spend it on malaria nets, you know, okay, great. You've made 100 malaria nets. But if you teach... Yeah.

Yeah. If you just take the money and you spend it on malaria nets, you know, okay, great. You've made 100 malaria nets. But if you teach... Yeah.

I like the flip side of effective altruism, effective accelerationism. I think accelerationism is the only thing that's ever lifted people out of poverty. The fact that food is cheap. Not we're giving food away because we are kind-hearted people. No, food is cheap. And that's the world you want to live in. UBI, what a scary idea. What a scary idea. All your power now? Your money is power?

I like the flip side of effective altruism, effective accelerationism. I think accelerationism is the only thing that's ever lifted people out of poverty. The fact that food is cheap. Not we're giving food away because we are kind-hearted people. No, food is cheap. And that's the world you want to live in. UBI, what a scary idea. What a scary idea. All your power now? Your money is power?

Your only source of power is granted to you by the goodwill of the government? What a scary idea.

Your only source of power is granted to you by the goodwill of the government? What a scary idea.

I'd rather die than need UBI to survive, and I mean it.

I'd rather die than need UBI to survive, and I mean it.

You can make survival guaranteed without UBI. What you have to do is make housing and food dirt cheap. And that's the good world. And actually, let's go into what we should really be making dirt cheap, which is energy. That energy that, you know, oh my God, like, you know, that's, if there's one, I'm pretty centrist politically. If there's one political position I cannot stand, it's deceleration.

You can make survival guaranteed without UBI. What you have to do is make housing and food dirt cheap. And that's the good world. And actually, let's go into what we should really be making dirt cheap, which is energy. That energy that, you know, oh my God, like, you know, that's, if there's one, I'm pretty centrist politically. If there's one political position I cannot stand, it's deceleration.

It's people who believe we should use less energy.

It's people who believe we should use less energy.

Yeah.

Yeah.

Not people who believe global warming is a problem. I agree with you. Not people who believe that, you know, saving the environment is good. I agree with you. But people who think we should use less energy, that energy usage is a moral bad. No. Yeah. No, you are asking, you are diminishing humanity.

Not people who believe global warming is a problem. I agree with you. Not people who believe that, you know, saving the environment is good. I agree with you. But people who think we should use less energy, that energy usage is a moral bad. No. Yeah. No, you are asking, you are diminishing humanity.

How do we make more of it? How do we make it clean? And how do we make, just, just, just, how do I pay, you know, 20 cents for a megawatt hour instead of a kilowatt hour?

How do we make more of it? How do we make it clean? And how do we make, just, just, just, how do I pay, you know, 20 cents for a megawatt hour instead of a kilowatt hour?

You know, we need to, I wish there were more, more Elons in the world. Yeah. I think Elon sees it as like, this is a political battle that needed to be fought.

You know, we need to, I wish there were more, more Elons in the world. Yeah. I think Elon sees it as like, this is a political battle that needed to be fought.

And again, like, you know, I always ask the question of whenever I disagree with him, I remind myself that he's a billionaire and I'm not. So, you know, maybe he's got something figured out that I don't, or maybe he doesn't.

And again, like, you know, I always ask the question of whenever I disagree with him, I remind myself that he's a billionaire and I'm not. So, you know, maybe he's got something figured out that I don't, or maybe he doesn't.

And it must be so hard. It must be so hard to meet people once you get to that point where.

And it must be so hard. It must be so hard to meet people once you get to that point where.

See, I love not having shit. Like, I don't have shit, man. Trust me, there's nothing I can give you.

See, I love not having shit. Like, I don't have shit, man. Trust me, there's nothing I can give you.

There's nothing worth taking from me, you know?

There's nothing worth taking from me, you know?

And all the hate too.

And all the hate too.

So the content is being generated by, let's say, one humanity worth of intelligence. And you can quantify a humanity, right? That's a... You know, it's... exaflops, yadaflops, but you can quantify it. Once that generation is being done by 100 humanities, you're done.

So the content is being generated by, let's say, one humanity worth of intelligence. And you can quantify a humanity, right? That's a... You know, it's... exaflops, yadaflops, but you can quantify it. Once that generation is being done by 100 humanities, you're done.

And it keeps this absolutely fake PSYOP political divide alive so that the 1% can keep power.

And it keeps this absolutely fake PSYOP political divide alive so that the 1% can keep power.

No, no. No, I'm not that methodical.

No, no. No, I'm not that methodical.

I think that there comes to a point where if it's no longer visceral, I just can't enjoy it. I still viscerally love programming.

I think that there comes to a point where if it's no longer visceral, I just can't enjoy it. I still viscerally love programming.

I mean, just my computer in general. I mean, you know, I tell my girlfriend, my first love is my computer, of course. Like, you know, I sleep with my computer. It's there for a lot of my sexual experiences. Like, come on, so is everyone's, right? Like, you know, you gotta be real about that.

I mean, just my computer in general. I mean, you know, I tell my girlfriend, my first love is my computer, of course. Like, you know, I sleep with my computer. It's there for a lot of my sexual experiences. Like, come on, so is everyone's, right? Like, you know, you gotta be real about that.

The fact that, yeah, I mean, it's, you know, I wish it was, and someday they'll be smarter and someday, you know, maybe I'm weird for this, but I don't discriminate, man. I'm not going to discriminate biostack life and silicon stack life. Like,

The fact that, yeah, I mean, it's, you know, I wish it was, and someday they'll be smarter and someday, you know, maybe I'm weird for this, but I don't discriminate, man. I'm not going to discriminate biostack life and silicon stack life. Like,

No, you see, no, no, no. But VS Code is, no, they're just doing that. Microsoft's doing that to try to get me hooked on it. I'll see through it.

No, you see, no, no, no. But VS Code is, no, they're just doing that. Microsoft's doing that to try to get me hooked on it. I'll see through it.

I'll see through it. It's gold digger, man. It's gold digger.

I'll see through it. It's gold digger, man. It's gold digger.

Well, this just gets more interesting, right?

Well, this just gets more interesting, right?

Oh, absolutely. No, no, no. Look, I think Microsoft, again, I wouldn't count on it to be true forever, but I think right now Microsoft is doing the best work in the programming world. Like between GitHub, GitHub Actions, VS Code, the improvements to Python, where's Microsoft? Like...

Oh, absolutely. No, no, no. Look, I think Microsoft, again, I wouldn't count on it to be true forever, but I think right now Microsoft is doing the best work in the programming world. Like between GitHub, GitHub Actions, VS Code, the improvements to Python, where's Microsoft? Like...

Right? Right?

Right? Right?

How things change.

How things change.

By the way, that's who I bet on to replace Google, by the way.

By the way, that's who I bet on to replace Google, by the way.

Microsoft.

Microsoft.

Satya Nadella said straight up, I'm coming for it.

Satya Nadella said straight up, I'm coming for it.

I think we're a long way away from that. But I would not be surprised if in the next five years, Bing overtakes Google as a search engine.

I think we're a long way away from that. But I would not be surprised if in the next five years, Bing overtakes Google as a search engine.

Wouldn't surprise me.

Wouldn't surprise me.

Interesting.

Interesting.

It might be some startup too. I would equally bet on some startup.

It might be some startup too. I would equally bet on some startup.

To win.

To win.

Of course.

Of course.

I don't know. I haven't figured out what the game is yet, but when I do, I want to win.

I don't know. I haven't figured out what the game is yet, but when I do, I want to win.

I think the game is to stand eye to eye with God.

I think the game is to stand eye to eye with God.

I mean, this is what, like, I don't know. This is some, this is some, there's probably some ego trip of mine, you know? Like, you want to stand eye to eye with God. He's just blasphemous, man. Okay. I don't know. I don't know. I don't know if it would upset God. I think he, like, wants that. I mean, I certainly want that for my creations. I want my creations to stand eye to eye with me.

I mean, this is what, like, I don't know. This is some, this is some, there's probably some ego trip of mine, you know? Like, you want to stand eye to eye with God. He's just blasphemous, man. Okay. I don't know. I don't know. I don't know if it would upset God. I think he, like, wants that. I mean, I certainly want that for my creations. I want my creations to stand eye to eye with me.

So why wouldn't God want me to stand eye to eye with him? That's the best I can do, golden rule.

So why wouldn't God want me to stand eye to eye with him? That's the best I can do, golden rule.

I only watched season one of Westworld, but yeah, we got to find the maze and solve it.

I only watched season one of Westworld, but yeah, we got to find the maze and solve it.

I wrote a blog post. I reread Genesis and just looked like, they give you some clues at the end of Genesis for finding the Garden of Eden. And I'm interested. I'm interested.

I wrote a blog post. I reread Genesis and just looked like, they give you some clues at the end of Genesis for finding the Garden of Eden. And I'm interested. I'm interested.

Thank you. Great to be here.

Thank you. Great to be here.

Yeah.

Yeah.

I don't even know what it'll look like, right? Like again, you can't imagine the behaviors of something smarter than you, but a super intelligent, an agent that just dominates your intelligence so much will be able to completely manipulate you.

I don't even know what it'll look like, right? Like again, you can't imagine the behaviors of something smarter than you, but a super intelligent, an agent that just dominates your intelligence so much will be able to completely manipulate you.

You see? And that's the whole AI safety thing. It's not the machine that's going to do that. It's other humans using the machine that are going to do that to you.

You see? And that's the whole AI safety thing. It's not the machine that's going to do that. It's other humans using the machine that are going to do that to you.

The machine is a machine. Yeah. But the human gets the machine. And there's a lot of humans out there very interested in manipulating you.

The machine is a machine. Yeah. But the human gets the machine. And there's a lot of humans out there very interested in manipulating you.

Yes, but maybe for a different reason.

Yes, but maybe for a different reason.

Okay. Why didn't nuclear weapons kill everyone?

Okay. Why didn't nuclear weapons kill everyone?

I think there's an answer. I think it's actually very hard to deploy nuclear weapons tactically. it's very hard to accomplish tactical objectives. Great. I can nuke their country. I have an irradiated pile of rubble. I don't want that.

I think there's an answer. I think it's actually very hard to deploy nuclear weapons tactically. it's very hard to accomplish tactical objectives. Great. I can nuke their country. I have an irradiated pile of rubble. I don't want that.

Why don't I want an irradiated pile of rubble? Yeah. For all the reasons no one wants an irradiated pile of rubble.

Why don't I want an irradiated pile of rubble? Yeah. For all the reasons no one wants an irradiated pile of rubble.

Yeah, what you want, a total victory in a war is not usually the irradiation and eradication of the people there. It's the subjugation and domination of the people.

Yeah, what you want, a total victory in a war is not usually the irradiation and eradication of the people there. It's the subjugation and domination of the people.

It's somewhat surprising, but you see, it's the little red button that's going to be pressed with AI that's going to, you know, and that's why we die. It's not because the AI, if there's anything in the nature of AI, it's just the nature of humanity.

It's somewhat surprising, but you see, it's the little red button that's going to be pressed with AI that's going to, you know, and that's why we die. It's not because the AI, if there's anything in the nature of AI, it's just the nature of humanity.

Sure. So I think the most... Obvious way to me is wireheading. We end up amusing ourselves to death. We end up all staring at that infinite TikTok and forgetting to eat. Maybe it's even more benign than this. Maybe we all just stop reproducing. Now, to be fair, it's probably hard to get all of humanity.

Sure. So I think the most... Obvious way to me is wireheading. We end up amusing ourselves to death. We end up all staring at that infinite TikTok and forgetting to eat. Maybe it's even more benign than this. Maybe we all just stop reproducing. Now, to be fair, it's probably hard to get all of humanity.

I mean, diversity in humanity is... With due respect. I wish I was more weird. No, like I'm kind of, look, I'm drinking smart water, man. That's like a Coca-Cola product, right?

I mean, diversity in humanity is... With due respect. I wish I was more weird. No, like I'm kind of, look, I'm drinking smart water, man. That's like a Coca-Cola product, right?

I went corporate. No, the amount of diversity in humanity I think is decreasing. Just like all the other biodiversity on the planet. Yeah. Right?

I went corporate. No, the amount of diversity in humanity I think is decreasing. Just like all the other biodiversity on the planet. Yeah. Right?

Go eat McDonald's in China.

Go eat McDonald's in China.

Yeah. No, it's the interconnectedness that's doing it.

Yeah. No, it's the interconnectedness that's doing it.

There is. In a bunker. To be fair, do I think AI kills us all? I think AI kills everything we call society today. I do not think it actually kills the human species. I think that's actually incredibly hard to do.

There is. In a bunker. To be fair, do I think AI kills us all? I think AI kills everything we call society today. I do not think it actually kills the human species. I think that's actually incredibly hard to do.

Yeah, but some of us do. And they'll be okay and they'll rebuild after the great AI.

Yeah, but some of us do. And they'll be okay and they'll rebuild after the great AI.

Whoa, whoa, whoa. They're going to be religiously against that.

Whoa, whoa, whoa. They're going to be religiously against that.

Sure. I mean, it'll be like, you know, some kind of Amish looking kind of thing, I think. I think they're going to have very strong taboos against technology.

Sure. I mean, it'll be like, you know, some kind of Amish looking kind of thing, I think. I think they're going to have very strong taboos against technology.

What's interesting about everything we build, I think we're going to build super intelligence before we build any sort of robustness in the AI. We cannot build an AI that is capable of going out into nature and surviving like a bird, right? A bird is an incredibly robust organism. We've built nothing like this. We haven't built a machine that's capable of reproducing.

What's interesting about everything we build, I think we're going to build super intelligence before we build any sort of robustness in the AI. We cannot build an AI that is capable of going out into nature and surviving like a bird, right? A bird is an incredibly robust organism. We've built nothing like this. We haven't built a machine that's capable of reproducing.

Let's just focus on them reproducing, right? Do they have microchips in them? Okay. Then do they include a fab?

Let's just focus on them reproducing, right? Do they have microchips in them? Okay. Then do they include a fab?

Then how are they going to reproduce?

Then how are they going to reproduce?

Yeah, but then you're really moving away from robustness. Yes. All of life is capable of reproducing without needing to go to a repair shop. Life will continue to reproduce in the complete absence of civilization. Robots will not. So if the AI apocalypse happens...

Yeah, but then you're really moving away from robustness. Yes. All of life is capable of reproducing without needing to go to a repair shop. Life will continue to reproduce in the complete absence of civilization. Robots will not. So if the AI apocalypse happens...

I mean, the AIs are going to probably die out because I think we're going to get, again, super intelligence long before we get robustness.

I mean, the AIs are going to probably die out because I think we're going to get, again, super intelligence long before we get robustness.

Well, that'd be very interesting. I'm interested in building that.

Well, that'd be very interesting. I'm interested in building that.

Very, very hard.

Very, very hard.

And then they remember that you're going to have to have a fab.

And then they remember that you're going to have to have a fab.

Why is that hard? Well, because it's not, I mean, a 3D printer is a very simple machine, right? Okay, you're going to print chips? You're going to have an atomic printer? How are you going to dope the silicon?

Why is that hard? Well, because it's not, I mean, a 3D printer is a very simple machine, right? Okay, you're going to print chips? You're going to have an atomic printer? How are you going to dope the silicon?

Yeah. Right?

Yeah. Right?

How are you going to etch the silicon?

How are you going to etch the silicon?

Yeah, but structural type of robots aren't going to have the intelligence required to survive in any complex environment.

Yeah, but structural type of robots aren't going to have the intelligence required to survive in any complex environment.

I don't think this works. I mean, again, like ants at their very core are made up of cells that are capable of individually reproducing. They're doing quite a lot of computation that we're taking for granted. It's not even just the computation. It's that reproduction is so inherent. Okay, so like there's two stacks of life in the world. There's the biological stack and the silicon stack.

I don't think this works. I mean, again, like ants at their very core are made up of cells that are capable of individually reproducing. They're doing quite a lot of computation that we're taking for granted. It's not even just the computation. It's that reproduction is so inherent. Okay, so like there's two stacks of life in the world. There's the biological stack and the silicon stack.

The biological stack starts with reproduction. Reproduction is at the absolute core. The first proto-RNA organisms were capable of reproducing. The silicon stack, despite as far as it's come, is nowhere near being able to reproduce.

The biological stack starts with reproduction. Reproduction is at the absolute core. The first proto-RNA organisms were capable of reproducing. The silicon stack, despite as far as it's come, is nowhere near being able to reproduce.

Yeah.

Yeah.

Even if you did put a fab on the machine, right? Let's say, okay, you know, we can build fabs. We know how to do that as humanity. We can probably put all the precursors that build all the machines and the fabs also in the machine. So first off, this machine is going to be absolutely massive.

Even if you did put a fab on the machine, right? Let's say, okay, you know, we can build fabs. We know how to do that as humanity. We can probably put all the precursors that build all the machines and the fabs also in the machine. So first off, this machine is going to be absolutely massive.

I mean, we almost have a, like, think of the size of the thing required to reproduce a machine today, right? Like, is our civilization capable of reproduction? Can we reproduce our civilization on Mars?

I mean, we almost have a, like, think of the size of the thing required to reproduce a machine today, right? Like, is our civilization capable of reproduction? Can we reproduce our civilization on Mars?

I believe that Twitter can be run by 50 people. I think that this is going to take most of, like, it's just most of society, right? Like we live in one globalized world.

I believe that Twitter can be run by 50 people. I think that this is going to take most of, like, it's just most of society, right? Like we live in one globalized world.

Oh, okay. You're talking about, yeah, okay. So you're talking about the humans reproducing and like basically like what's the smallest self-sustaining colony of humans?

Oh, okay. You're talking about, yeah, okay. So you're talking about the humans reproducing and like basically like what's the smallest self-sustaining colony of humans?

Yeah, okay, fine. But they're not going to be making five nanometer chips.

Yeah, okay, fine. But they're not going to be making five nanometer chips.

Maybe. Or maybe they'll watch our colony die out over here and be like, we're not making chips.

Maybe. Or maybe they'll watch our colony die out over here and be like, we're not making chips.

Don't make chips.

Don't make chips.

Whatever you do, don't make chips. Chips are what led to their downfall.

Whatever you do, don't make chips. Chips are what led to their downfall.

Do you need that asshole? That's the question, right? Humanity works really hard today to get rid of that asshole, but I think they might be important.

Do you need that asshole? That's the question, right? Humanity works really hard today to get rid of that asshole, but I think they might be important.

I like to think it's just like another stack for life. Like we have like the biostack life, like we're a biostack life and then the silicon stack life.

I like to think it's just like another stack for life. Like we have like the biostack life, like we're a biostack life and then the silicon stack life.

Oh, no, we don't know what the ceiling is for the biostack either. The biostack just seemed to move slower. You have Moore's Law, which is not dead despite many proclamations.

Oh, no, we don't know what the ceiling is for the biostack either. The biostack just seemed to move slower. You have Moore's Law, which is not dead despite many proclamations.

And you don't have anything like this in the biostack. So I have a meme that I posted. I tried to make a meme. It didn't work too well. But I posted a picture of Ronald Reagan and Joe Biden. And you look, this is 1980 and this is 2020. And these two humans are basically like the same. There's been no change in humans in the last 40 years.

And you don't have anything like this in the biostack. So I have a meme that I posted. I tried to make a meme. It didn't work too well. But I posted a picture of Ronald Reagan and Joe Biden. And you look, this is 1980 and this is 2020. And these two humans are basically like the same. There's been no change in humans in the last 40 years.

And then I posted a computer from 1980 and a computer from 2020. Wow.

And then I posted a computer from 1980 and a computer from 2020. Wow.

Oh, yeah.

Oh, yeah.

Yeah.

Yeah.

I've been ready for a long time.

I've been ready for a long time.

I love it.

I love it.

Yeah.

Yeah.

Judging from what you can buy today, far. Very far.

Judging from what you can buy today, far. Very far.

I mean, the headsets just are not quite at eye resolution yet. I haven't put on any headset where I'm like, oh, this could be the real world. Whereas when I put good headphones on, audio is there. We can reproduce audio that I'm like, I'm actually in a jungle right now. If I close my eyes, I can't tell I'm not.

I mean, the headsets just are not quite at eye resolution yet. I haven't put on any headset where I'm like, oh, this could be the real world. Whereas when I put good headphones on, audio is there. We can reproduce audio that I'm like, I'm actually in a jungle right now. If I close my eyes, I can't tell I'm not.

Or humans want to believe.

Or humans want to believe.

Humans want to believe so much that people think the large language models are conscious. That's how much humans want to believe.

Humans want to believe so much that people think the large language models are conscious. That's how much humans want to believe.

I don't think I'm conscious.

I don't think I'm conscious.

It's like what it seems to mean to people. It's just like a word that atheists use for souls.

It's like what it seems to mean to people. It's just like a word that atheists use for souls.

If consciousness is a spectrum, I'm definitely way more conscious than the large language models are. I think the large language models are less conscious than a chicken.

If consciousness is a spectrum, I'm definitely way more conscious than the large language models are. I think the large language models are less conscious than a chicken.

In Miami, like a couple months ago.

In Miami, like a couple months ago.

There's living chickens walking around Miami. It's crazy.

There's living chickens walking around Miami. It's crazy.

Yeah.

Yeah.

A chicken, yeah.

A chicken, yeah.

Humans want to believe so much that if I took a rock and a Sharpie and drew a sad face on the rock, they'd think the rock is sad.

Humans want to believe so much that if I took a rock and a Sharpie and drew a sad face on the rock, they'd think the rock is sad.

No.

No.

Yeah, I mean, it's interesting that like human systems seem to claim that they're conscious. And I guess it kind of like says something in a straight up like, okay, what do people mean when, even if you don't believe in consciousness, what do people mean when they say consciousness? And there's definitely like meanings to it.

Yeah, I mean, it's interesting that like human systems seem to claim that they're conscious. And I guess it kind of like says something in a straight up like, okay, what do people mean when, even if you don't believe in consciousness, what do people mean when they say consciousness? And there's definitely like meanings to it.

Pizza.

Pizza.

I like cheese pizza.

I like cheese pizza.

No, I don't like pineapple.

No, I don't like pineapple.

As they put any ham on it, oh, that's real bad.

As they put any ham on it, oh, that's real bad.

Oh, that's my favorite.

Oh, that's my favorite.

If that's the word you want to use to describe it, sure. I'm not going to deny that that feeling exists. I'm not going to deny that I experienced that feeling. When, I guess what I kind of take issue to is that there's some like, like, how does it feel to be a web server? Do 404s hurt? Not yet. How would you know what suffering looked like?

If that's the word you want to use to describe it, sure. I'm not going to deny that that feeling exists. I'm not going to deny that I experienced that feeling. When, I guess what I kind of take issue to is that there's some like, like, how does it feel to be a web server? Do 404s hurt? Not yet. How would you know what suffering looked like?

Sure, you can recognize a suffering dog because we're the same stack as the dog. All the biostack stuff kind of, especially mammals, you know, it's really easy. Game recognizes game. Yeah. Versus the silicon stack stuff, it's like, you have no idea. You have, wow, the little thing has learned to mimic, you know. But then I realized that that's all we are too.

Sure, you can recognize a suffering dog because we're the same stack as the dog. All the biostack stuff kind of, especially mammals, you know, it's really easy. Game recognizes game. Yeah. Versus the silicon stack stuff, it's like, you have no idea. You have, wow, the little thing has learned to mimic, you know. But then I realized that that's all we are too.

Oh, look, the little thing has learned to mimic.

Oh, look, the little thing has learned to mimic.

The definition of consciousness is how close something looks to human. Sure, I'll give you that one.

The definition of consciousness is how close something looks to human. Sure, I'll give you that one.

Sure. It's a very anthropocentric definition, but... Well, that's all we got. Sure. No, and I don't mean to like... I think there's a lot of value in it. Look, I just started my second company. My third company will be AI Girlfriends.

Sure. It's a very anthropocentric definition, but... Well, that's all we got. Sure. No, and I don't mean to like... I think there's a lot of value in it. Look, I just started my second company. My third company will be AI Girlfriends.

Yeah, but okay, so here's where it actually gets totally different, right? When you interact with another human, you can make some assumptions, right? When you interact with these models, you can't. You can make some assumptions that that other human experiences suffering and pleasure in a pretty similar way to you do. The golden rule applies. With an AI model, this isn't really true.

Yeah, but okay, so here's where it actually gets totally different, right? When you interact with another human, you can make some assumptions, right? When you interact with these models, you can't. You can make some assumptions that that other human experiences suffering and pleasure in a pretty similar way to you do. The golden rule applies. With an AI model, this isn't really true.

These large language models are good at fooling people because they were trained on a whole bunch of human data and told to mimic it.

These large language models are good at fooling people because they were trained on a whole bunch of human data and told to mimic it.

Yeah.

Yeah.

Yeah.

Yeah.

I made some chatbots. I gave them backstories. It was lots of fun. I was so happy when Llama came out.

I made some chatbots. I gave them backstories. It was lots of fun. I was so happy when Llama came out.

To be fair, like, you know, something that people generally look for when they're looking for someone to date is intelligence in some form. And the rock doesn't really have intelligence. Only a pretty desperate person would date a rock. I think we're all desperate deep down. Oh, not rock level desperate.

To be fair, like, you know, something that people generally look for when they're looking for someone to date is intelligence in some form. And the rock doesn't really have intelligence. Only a pretty desperate person would date a rock. I think we're all desperate deep down. Oh, not rock level desperate.

Oh, I agree. And you know what? I won't even say this so cynically. I will actually say this in a way that like, I want AI friends. I do. Yeah. Like I would love to, you know, again, the language models now are still a little, like people are impressed with these GPT things. And I look at like, or like, or the co-pilot, the coding one. And I'm like, okay, this is like junior engineer level.

Oh, I agree. And you know what? I won't even say this so cynically. I will actually say this in a way that like, I want AI friends. I do. Yeah. Like I would love to, you know, again, the language models now are still a little, like people are impressed with these GPT things. And I look at like, or like, or the co-pilot, the coding one. And I'm like, okay, this is like junior engineer level.

And these people are like Fiverr level artists and copywriters. Like, okay, great. We got like Fiverr and like junior engineers. Okay, cool. Like, and this is just the start and it will get better, right? Like I can't wait to have AI friends who are more intelligent than I am.

And these people are like Fiverr level artists and copywriters. Like, okay, great. We got like Fiverr and like junior engineers. Okay, cool. Like, and this is just the start and it will get better, right? Like I can't wait to have AI friends who are more intelligent than I am.

That's up to you and your human partner to define.

That's up to you and your human partner to define.

Yeah, you have to have that conversation, I guess.

Yeah, you have to have that conversation, I guess.

No, I mean, it's similar kind of to porn.

No, I mean, it's similar kind of to porn.

Yeah. I think people in relationships have different views on that.

Yeah. I think people in relationships have different views on that.

The porn one is a good branching off point. Like these things, you know, one of my scenarios that I put in my chat bot is I, you know, a nice girl named Lexi. She's 20. She just moved out to LA. She wanted to be an actress, but she started doing OnlyFans instead. And you're on a date with her. Enjoy. Yeah.

The porn one is a good branching off point. Like these things, you know, one of my scenarios that I put in my chat bot is I, you know, a nice girl named Lexi. She's 20. She just moved out to LA. She wanted to be an actress, but she started doing OnlyFans instead. And you're on a date with her. Enjoy. Yeah.

I mean, these are all things for people to define in their relationships. What it means to be human is just gonna start to get weird.

I mean, these are all things for people to define in their relationships. What it means to be human is just gonna start to get weird.

Do you know about shadow banning?

Do you know about shadow banning?

Shadow banning, okay, you post, no one can see it. Heaven banning, you post, no one can see it, but a whole lot of AIs are spun up to interact with you.

Shadow banning, okay, you post, no one can see it. Heaven banning, you post, no one can see it, but a whole lot of AIs are spun up to interact with you.

There's a great... It's called My Little Pony Friendship is Optimal. It's a sci-fi story that explores this idea.

There's a great... It's called My Little Pony Friendship is Optimal. It's a sci-fi story that explores this idea.

Friendship is optimal.

Friendship is optimal.

I want it. Look, I want it. If no one else wants it, I want it.

I want it. Look, I want it. If no one else wants it, I want it.

And I'll feel their loneliness and, you know, it just will only advertise to you some of the time.

And I'll feel their loneliness and, you know, it just will only advertise to you some of the time.

This interesting path from rationality to polyamory. Yeah, that doesn't make sense for me.

This interesting path from rationality to polyamory. Yeah, that doesn't make sense for me.

The crazy thing is like, Culture is whatever we define it as, right? These things are not, like, is-ought problem in moral philosophy, right? There's no, like, okay, what is might be that, like, computers are capable of mimicking, you know, girlfriends perfectly. They passed the girlfriend Turing test, right? But that doesn't say anything about ought.

The crazy thing is like, Culture is whatever we define it as, right? These things are not, like, is-ought problem in moral philosophy, right? There's no, like, okay, what is might be that, like, computers are capable of mimicking, you know, girlfriends perfectly. They passed the girlfriend Turing test, right? But that doesn't say anything about ought.

That doesn't say anything about how we ought to respond to them as a civilization. That doesn't say we ought to get rid of monogamy, right? That's a completely separate question, really a religious one.

That doesn't say anything about how we ought to respond to them as a civilization. That doesn't say we ought to get rid of monogamy, right? That's a completely separate question, really a religious one.

No, I mean, of course, my AI girlfriends, their goal is to pass the Girlfriend Turing Test.

No, I mean, of course, my AI girlfriends, their goal is to pass the Girlfriend Turing Test.

Yeah, I mean, you know, look, we're a company. We don't have to get everybody. We just have to get a large enough clientele to stay with us.

Yeah, I mean, you know, look, we're a company. We don't have to get everybody. We just have to get a large enough clientele to stay with us.

All right.

All right.

I started TinyGrad as like a toy project just to teach myself, okay, like what is a convolution? What are all these options you can pass to them? What is the derivative of a convolution, right? Very similar to Karpathy wrote MicroGrad. Very similar. And then I started realizing, I started thinking about like AI chips. I started thinking about chips that run

I started TinyGrad as like a toy project just to teach myself, okay, like what is a convolution? What are all these options you can pass to them? What is the derivative of a convolution, right? Very similar to Karpathy wrote MicroGrad. Very similar. And then I started realizing, I started thinking about like AI chips. I started thinking about chips that run

And I was like, well, okay, this is going to be a really big problem. If NVIDIA becomes a monopoly here, how long before NVIDIA is nationalized?

And I was like, well, okay, this is going to be a really big problem. If NVIDIA becomes a monopoly here, how long before NVIDIA is nationalized?

Yeah.

Yeah.

If NVIDIA becomes just like 10X better than everything else, you're giving a big advantage to somebody who can secure NVIDIA as a resource. Yeah. In fact, if Jensen watches this podcast, he may want to consider this. He may want to consider making sure his company is not nationalized.

If NVIDIA becomes just like 10X better than everything else, you're giving a big advantage to somebody who can secure NVIDIA as a resource. Yeah. In fact, if Jensen watches this podcast, he may want to consider this. He may want to consider making sure his company is not nationalized.

Oh, yes.

Oh, yes.

So we have Nvidia and AMD. Great.

So we have Nvidia and AMD. Great.

Have you seen it? Google loves to rent you TPUs.

Have you seen it? Google loves to rent you TPUs.

So I started work on a, uh, I was like, okay, what's it going to take to make a chip? And my first notions were all completely wrong about why, about like how you could improve on GPUs. And I will take this, this is from Jim Keller on your podcast. And this is one of my absolute favorite descriptions of computation.

So I started work on a, uh, I was like, okay, what's it going to take to make a chip? And my first notions were all completely wrong about why, about like how you could improve on GPUs. And I will take this, this is from Jim Keller on your podcast. And this is one of my absolute favorite descriptions of computation.

So there's three kinds of computation paradigms that are common in the world today. There's CPUs, and CPUs can do everything. CPUs can do add and multiply, they can do load and store, and they can do compare and branch. And when I say they can do these things, they can do them all fast, right?

So there's three kinds of computation paradigms that are common in the world today. There's CPUs, and CPUs can do everything. CPUs can do add and multiply, they can do load and store, and they can do compare and branch. And when I say they can do these things, they can do them all fast, right?

So compare and branch are unique to CPUs, and what I mean by they can do them fast is they can do things like branch prediction and speculative execution, and they spend tons of transistors on these super deep reorder buffers in order to make these things fast. Then you have a simpler computation model, GPUs. GPUs can't really do compare and branch. I mean, they can, but it's horrendously slow.

So compare and branch are unique to CPUs, and what I mean by they can do them fast is they can do things like branch prediction and speculative execution, and they spend tons of transistors on these super deep reorder buffers in order to make these things fast. Then you have a simpler computation model, GPUs. GPUs can't really do compare and branch. I mean, they can, but it's horrendously slow.

But GPUs can do arbitrary load and store. GPUs can do things like X, dereference Y. So they can fetch from arbitrary pieces of memory. They can fetch from memory that is defined by the contents of the data. The third model of computation is DSPs. And DSPs are just add and multiply. They can do load and stores, but only static load and stores.

But GPUs can do arbitrary load and store. GPUs can do things like X, dereference Y. So they can fetch from arbitrary pieces of memory. They can fetch from memory that is defined by the contents of the data. The third model of computation is DSPs. And DSPs are just add and multiply. They can do load and stores, but only static load and stores.

Only loads and stores that are known before the program runs. And you look at neural networks today, and 95% of neural networks are all the DSP paradigm. They are just statically scheduled adds and multiplies. So TinyGuard really took this idea, and I'm still working on it, to extend this as far as possible. Every stage of the stack has Turing completeness.

Only loads and stores that are known before the program runs. And you look at neural networks today, and 95% of neural networks are all the DSP paradigm. They are just statically scheduled adds and multiplies. So TinyGuard really took this idea, and I'm still working on it, to extend this as far as possible. Every stage of the stack has Turing completeness.

All right, Python has Turing completeness, and then we take Python, we go into C++, which is Turing complete, and maybe C++ calls into some CUDA kernels, which are Turing complete. The CUDA kernels go through LLVM, which is Turing complete, into PTX, which is Turing complete, to SAS, which is Turing complete, on a Turing complete processor. I wanna get Turing completeness out of the stack entirely.

All right, Python has Turing completeness, and then we take Python, we go into C++, which is Turing complete, and maybe C++ calls into some CUDA kernels, which are Turing complete. The CUDA kernels go through LLVM, which is Turing complete, into PTX, which is Turing complete, to SAS, which is Turing complete, on a Turing complete processor. I wanna get Turing completeness out of the stack entirely.

Because once you get rid of Turing completeness, you can reason about things. Rice's theorem and the halting problem do not apply to admiral machines.

Because once you get rid of Turing completeness, you can reason about things. Rice's theorem and the halting problem do not apply to admiral machines.

Every layer of the stack. Every layer. Every layer of the stack, removing Turing completeness allows you to reason about things, right? So the reason you need to do branch prediction in a CPU and the reason it's prediction, and the branch predictors are, I think they're like 99% on CPUs. Why do they get 1% of them wrong? Well, they get 1% wrong because you can't know. Right?

Every layer of the stack. Every layer. Every layer of the stack, removing Turing completeness allows you to reason about things, right? So the reason you need to do branch prediction in a CPU and the reason it's prediction, and the branch predictors are, I think they're like 99% on CPUs. Why do they get 1% of them wrong? Well, they get 1% wrong because you can't know. Right?

That's the halting problem. It's equivalent to the halting problem to say whether a branch is going to be taken or not. I can show that. But the AdMob machine, the neural network, runs the identical compute every time. The only thing that changes is the data. So when you realize this, you think about, okay, how can we build a computer?

That's the halting problem. It's equivalent to the halting problem to say whether a branch is going to be taken or not. I can show that. But the AdMob machine, the neural network, runs the identical compute every time. The only thing that changes is the data. So when you realize this, you think about, okay, how can we build a computer?

How can we build a stack that takes maximal advantage of this idea? So what makes TinyGrad different from other neural network libraries is it does not have a primitive operator even for matrix multiplication. And this is every single one. They even have primitive operations for things like convolutions.

How can we build a stack that takes maximal advantage of this idea? So what makes TinyGrad different from other neural network libraries is it does not have a primitive operator even for matrix multiplication. And this is every single one. They even have primitive operations for things like convolutions.

No matmul. Well, here's what a matmul is. So I'll use my hands to talk here. So if you think about a cube and I put my two matrices that I'm multiplying on two faces of the cube, right? You can think about the matrix multiply as, okay, the n cubed, I'm going to multiply for each one in the cubed. And then I'm going to do a sum, which is a reduce up to here to the third face of the cube.

No matmul. Well, here's what a matmul is. So I'll use my hands to talk here. So if you think about a cube and I put my two matrices that I'm multiplying on two faces of the cube, right? You can think about the matrix multiply as, okay, the n cubed, I'm going to multiply for each one in the cubed. And then I'm going to do a sum, which is a reduce up to here to the third face of the cube.

And that's your multiplied matrix. So what a matrix multiply is, is a bunch of shape operations, right? A bunch of permute three shapes and expands on the two matrices. A multiply, n cubed. A reduce, n cubed, which gives you an n squared matrix.

And that's your multiplied matrix. So what a matrix multiply is, is a bunch of shape operations, right? A bunch of permute three shapes and expands on the two matrices. A multiply, n cubed. A reduce, n cubed, which gives you an n squared matrix.

So TinyGrad has about 20. And you can compare TinyGrad's op set or IR to things like XLA or PrimTorch. So XLA and PrimTorch are ideas where like, okay, Torch has like 2000 different kernels. PyTorch 2.0 introduced PrimTorch, which has only 250. TinyGrad has order of magnitude 25. It's 10x less than XLA or Primtorch. And you can think about it as kind of like RISC versus CISC, right?

So TinyGrad has about 20. And you can compare TinyGrad's op set or IR to things like XLA or PrimTorch. So XLA and PrimTorch are ideas where like, okay, Torch has like 2000 different kernels. PyTorch 2.0 introduced PrimTorch, which has only 250. TinyGrad has order of magnitude 25. It's 10x less than XLA or Primtorch. And you can think about it as kind of like RISC versus CISC, right?

These other things are CISC-like systems. TinyGrad is RISC.

These other things are CISC-like systems. TinyGrad is RISC.

RISC architecture is going to change everything. 1995, hackers.

RISC architecture is going to change everything. 1995, hackers.

Angelina Jolie delivers the line, risk architecture is going to change everything in 1995. Wow. And here we are with ARM in the phones. And ARM everywhere.

Angelina Jolie delivers the line, risk architecture is going to change everything in 1995. Wow. And here we are with ARM in the phones. And ARM everywhere.

Sure. Okay, so you have unary ops, which take in a tensor and return a tensor of the same size and do some unary op to it. X, log, reciprocal, sine, right? They take in one and they're point-wise.

Sure. Okay, so you have unary ops, which take in a tensor and return a tensor of the same size and do some unary op to it. X, log, reciprocal, sine, right? They take in one and they're point-wise.

Yeah, ReLU. Almost all activation functions are unary ops. Some combinations of unary ops together is still a unary op. Then you have binary ops. Binary ops are like pointwise addition, multiplication, division, compare. It takes in two tensors of equal size and outputs one tensor. Then you have reduce ops.

Yeah, ReLU. Almost all activation functions are unary ops. Some combinations of unary ops together is still a unary op. Then you have binary ops. Binary ops are like pointwise addition, multiplication, division, compare. It takes in two tensors of equal size and outputs one tensor. Then you have reduce ops.

Reduce ops will take a three-dimensional tensor and turn it into a two-dimensional tensor, or a three-dimensional tensor and turn it into a zero-dimensional tensor. Think like a sum or a max are really the common ones there. And then the fourth type is movement ops. And movement ops are different from the other types because they don't actually require computation.

Reduce ops will take a three-dimensional tensor and turn it into a two-dimensional tensor, or a three-dimensional tensor and turn it into a zero-dimensional tensor. Think like a sum or a max are really the common ones there. And then the fourth type is movement ops. And movement ops are different from the other types because they don't actually require computation.

They require different ways to look at memory. So that includes reshapes, permutes, expands, flips. Those are the main ones, probably.

They require different ways to look at memory. So that includes reshapes, permutes, expands, flips. Those are the main ones, probably.

And convolutions. And every convolution you can imagine, dilated convolutions, strided convolutions, transposed convolutions.

And convolutions. And every convolution you can imagine, dilated convolutions, strided convolutions, transposed convolutions.

Sure. So if you type in PyTorch A times B plus C, what this is going to do is it's going to first multiply A and B and store that result into memory. And then it is going to add C by reading that result from memory, reading C from memory, and writing that out to memory. There is way more loads and stores to memory than you need there.

Sure. So if you type in PyTorch A times B plus C, what this is going to do is it's going to first multiply A and B and store that result into memory. And then it is going to add C by reading that result from memory, reading C from memory, and writing that out to memory. There is way more loads and stores to memory than you need there.

If you don't actually do A times B as soon as you see it, if you wait until the user actually realizes that tensor, until the laziness actually resolves, you confuse that plus C. This is like, it's the same way Haskell works.

If you don't actually do A times B as soon as you see it, if you wait until the user actually realizes that tensor, until the laziness actually resolves, you confuse that plus C. This is like, it's the same way Haskell works.

So TinyGrad's front end looks very similar to PyTorch. I probably could make a perfect or pretty close to perfect interop layer if I really wanted to. I think that there's some things that are nicer about TinyGrad syntax than PyTorch, but the front end looks very Torch-like. You can also load in Onyx models. We have more Onyx tests passing than Core ML.

So TinyGrad's front end looks very similar to PyTorch. I probably could make a perfect or pretty close to perfect interop layer if I really wanted to. I think that there's some things that are nicer about TinyGrad syntax than PyTorch, but the front end looks very Torch-like. You can also load in Onyx models. We have more Onyx tests passing than Core ML.

Okay, so... We'll pass Onyx runtime soon.

Okay, so... We'll pass Onyx runtime soon.

By the way, I really like PyTorch. I think that it's actually a very good piece of software. I think that they've made a few different trade-offs, and these different trade-offs are where TinyGrad takes a different path. One of the biggest differences is it's really easy to see the kernels that are actually being sent to the GPU.

By the way, I really like PyTorch. I think that it's actually a very good piece of software. I think that they've made a few different trade-offs, and these different trade-offs are where TinyGrad takes a different path. One of the biggest differences is it's really easy to see the kernels that are actually being sent to the GPU.

If you run PyTorch on the GPU, you like do some operation and you don't know what kernels ran. You don't know how many kernels ran. You don't know how many flops were used. You don't know how much memory accesses were used. TinyGrad type debug equals two. And it will show you in this beautiful style, every kernel that's run, how many flops and how many bytes.

If you run PyTorch on the GPU, you like do some operation and you don't know what kernels ran. You don't know how many kernels ran. You don't know how many flops were used. You don't know how much memory accesses were used. TinyGrad type debug equals two. And it will show you in this beautiful style, every kernel that's run, how many flops and how many bytes.

TinyGrad solves the problem of porting new ML accelerators quickly. One of the reasons, tons of these companies now, I think Sequoia marked GraphCore to zero, right? Cerebus, TensTorrent, Grok. All of these ML accelerator companies, they built chips. The chips were good. The software was terrible. And part of the reason is because I think the same problem is happening with Dojo.

TinyGrad solves the problem of porting new ML accelerators quickly. One of the reasons, tons of these companies now, I think Sequoia marked GraphCore to zero, right? Cerebus, TensTorrent, Grok. All of these ML accelerator companies, they built chips. The chips were good. The software was terrible. And part of the reason is because I think the same problem is happening with Dojo.

It's really, really hard to write a PyTorch port because you have to write 250 kernels and you have to tune them all for performance.

It's really, really hard to write a PyTorch port because you have to write 250 kernels and you have to tune them all for performance.

Look, my prediction for Ten's Torrent is that they're going to pivot to making RISC-V chips. CPUs. CPUs.

Look, my prediction for Ten's Torrent is that they're going to pivot to making RISC-V chips. CPUs. CPUs.

Because AI accelerators are a software problem, not really a hardware problem.

Because AI accelerators are a software problem, not really a hardware problem.

I think what's going to happen is if I can finish... Okay. If you're trying to make an AI accelerator... You better have the capability of writing a torch-level performance stack on NVIDIA GPUs.

I think what's going to happen is if I can finish... Okay. If you're trying to make an AI accelerator... You better have the capability of writing a torch-level performance stack on NVIDIA GPUs.

If you can't write a torch stack on NVIDIA GPUs, and I mean all the way, I mean down to the driver, there's no way you're going to be able to write it on your chip, because your chip's worse than an NVIDIA GPU. The first version of the chip you tape out, it's definitely worse.

If you can't write a torch stack on NVIDIA GPUs, and I mean all the way, I mean down to the driver, there's no way you're going to be able to write it on your chip, because your chip's worse than an NVIDIA GPU. The first version of the chip you tape out, it's definitely worse.

Yes. And not only that, actually, the chip that you tape out, almost always because you're trying to get advantage over NVIDIA, you're specializing the hardware more. It's always harder to write software for more specialized hardware. Like a GPU is pretty generic. And if you can't write an NVIDIA stack, there's no way you can write a stack for your chip.

Yes. And not only that, actually, the chip that you tape out, almost always because you're trying to get advantage over NVIDIA, you're specializing the hardware more. It's always harder to write software for more specialized hardware. Like a GPU is pretty generic. And if you can't write an NVIDIA stack, there's no way you can write a stack for your chip.

So my approach with TinyGrad is first, write a performant NVIDIA stack. We're targeting AMD.

So my approach with TinyGrad is first, write a performant NVIDIA stack. We're targeting AMD.

With love.

With love.

It's like the Yankees, you know? I'm a Mets fan.

It's like the Yankees, you know? I'm a Mets fan.

Well, let's start with the fact that the 7900 XTX kernel drivers don't work. And if you run demo apps in loops, it panics the kernel.

Well, let's start with the fact that the 7900 XTX kernel drivers don't work. And if you run demo apps in loops, it panics the kernel.

Lisa Su responded to my email.

Lisa Su responded to my email.

Oh. I reached out. I was like, this is, you know, really? Like, I understand if your 7x7 transposed Winograd conv is slower than NVIDIA's, but literally when I run demo apps in a loop, the kernel panics.

Oh. I reached out. I was like, this is, you know, really? Like, I understand if your 7x7 transposed Winograd conv is slower than NVIDIA's, but literally when I run demo apps in a loop, the kernel panics.

I just literally took their demo apps and wrote like while true semicolon do the app semicolon done in a bunch of screens. This is like the most primitive fuzz testing.

I just literally took their demo apps and wrote like while true semicolon do the app semicolon done in a bunch of screens. This is like the most primitive fuzz testing.

They're changing. They're trying to change. They're trying to change. And I had a pretty positive interaction with them this week. Last week, I went on YouTube. I was just like, that's it. I give up on AMD. Like, this is their driver. I'm not going to, you know, I'll go with Intel GPUs. Intel GPUs have better drivers.

They're changing. They're trying to change. They're trying to change. And I had a pretty positive interaction with them this week. Last week, I went on YouTube. I was just like, that's it. I give up on AMD. Like, this is their driver. I'm not going to, you know, I'll go with Intel GPUs. Intel GPUs have better drivers.

Yeah, and I'd like to extend that diversification to everything. I'd like to diversify the, right, the more, my central thesis about the world is there's things that centralize power and they're bad. And there's things that decentralize power and they're good. Everything I can do to help decentralize power, I'd like to do.

Yeah, and I'd like to extend that diversification to everything. I'd like to diversify the, right, the more, my central thesis about the world is there's things that centralize power and they're bad. And there's things that decentralize power and they're good. Everything I can do to help decentralize power, I'd like to do.

I'd like to help them with software. No, actually, the only ASIC that is remotely successful is Google's TPU. And the only reason that's successful is because Google wrote a machine learning framework. I think that you have to write a competitive machine learning framework in order to be able to build an ASIC.

I'd like to help them with software. No, actually, the only ASIC that is remotely successful is Google's TPU. And the only reason that's successful is because Google wrote a machine learning framework. I think that you have to write a competitive machine learning framework in order to be able to build an ASIC.

They have one. They have an internal one.

They have one. They have an internal one.

I don't want a cloud.

I don't want a cloud.

I don't like cloud.

I don't like cloud.

Fundamental limitation of cloud is who owns the off switch.

Fundamental limitation of cloud is who owns the off switch.

Yeah.

Yeah.

Well, you shouldn't build one. You should buy a box from the Tiny Corp.

Well, you shouldn't build one. You should buy a box from the Tiny Corp.

It's called the tiny box.

It's called the tiny box.

It's $15,000. And it's almost a pay to flop of compute. It's over 100 gigabytes of GPU RAM. It's over five terabytes per second of GPU memory bandwidth. I'm going to put like four NVMEs in RAID. You're going to get like 20, 30 gigabytes per second of drive read bandwidth. I'm going to build like the best deep learning box that I can that plugs into one wall outlet.

It's $15,000. And it's almost a pay to flop of compute. It's over 100 gigabytes of GPU RAM. It's over five terabytes per second of GPU memory bandwidth. I'm going to put like four NVMEs in RAID. You're going to get like 20, 30 gigabytes per second of drive read bandwidth. I'm going to build like the best deep learning box that I can that plugs into one wall outlet.

Yeah. So it's almost a pay-to-flop of compute.

Yeah. So it's almost a pay-to-flop of compute.

Today, I'm leaning toward AMD. Okay. Um, but we're pretty agnostic to the type of compute. The, the, the main limiting spec is a 120 volt, 15 amp circuit.

Today, I'm leaning toward AMD. Okay. Um, but we're pretty agnostic to the type of compute. The, the, the main limiting spec is a 120 volt, 15 amp circuit.

Okay.

Okay.

Well, I mean it because in order to like, like there's a plug over there, right? You have to be able to plug it in. Um, we're also going to sell the tiny rack, which like, what's the most power you can get into your house without arousing suspicion? Uh, and one of the, one of the answers is an electric car charger.

Well, I mean it because in order to like, like there's a plug over there, right? You have to be able to plug it in. Um, we're also going to sell the tiny rack, which like, what's the most power you can get into your house without arousing suspicion? Uh, and one of the, one of the answers is an electric car charger.

A wall outlet is about 1,500 watts. A car charger is about 10,000 watts. Is that it?

A wall outlet is about 1,500 watts. A car charger is about 10,000 watts. Is that it?

Again, probably 7900 XTXs, but maybe 3090s, maybe A770s.

Again, probably 7900 XTXs, but maybe 3090s, maybe A770s.

I'm still exploring. I want to deliver a really good experience to people. And yeah, what GPUs I end up going with, again, I'm leaning toward AMD. We'll see. You know, in my email, what I said to AMD is like, just dumping the code on GitHub is not open source. Open source is a culture. Open source means that your issues are not all one-year-old stale issues. Open source means developing in public.

I'm still exploring. I want to deliver a really good experience to people. And yeah, what GPUs I end up going with, again, I'm leaning toward AMD. We'll see. You know, in my email, what I said to AMD is like, just dumping the code on GitHub is not open source. Open source is a culture. Open source means that your issues are not all one-year-old stale issues. Open source means developing in public.

And if you guys can commit to that, I see a real future for AMD as a competitor to NVIDIA.

And if you guys can commit to that, I see a real future for AMD as a competitor to NVIDIA.

We're taking pre-orders. I took this from Elon. I'm like $100 fully refundable pre-orders.

We're taking pre-orders. I took this from Elon. I'm like $100 fully refundable pre-orders.

No, I'll try to do it faster. It's a lot simpler. It's a lot simpler than a truck.

No, I'll try to do it faster. It's a lot simpler. It's a lot simpler than a truck.

The thing that I want to deliver to people out of the box is being able to run 65 billion parameter Lama in FP16 in real time. In like a good, like 10 tokens per second or five tokens per second or something.

The thing that I want to deliver to people out of the box is being able to run 65 billion parameter Lama in FP16 in real time. In like a good, like 10 tokens per second or five tokens per second or something.

Yeah, or I think Falcon is the new one. Experience a chat with the largest language model that you can have in your house.

Yeah, or I think Falcon is the new one. Experience a chat with the largest language model that you can have in your house.

From a wall plug, yeah. Actually, for inference, it's not like even more power would help you get more. Even more power wouldn't get you more. Well, no, the biggest model released is 65 billion parameter Lama, as far as I know.

From a wall plug, yeah. Actually, for inference, it's not like even more power would help you get more. Even more power wouldn't get you more. Well, no, the biggest model released is 65 billion parameter Lama, as far as I know.

That one's harder, actually.

That one's harder, actually.

The boyfriend's harder, yeah.

The boyfriend's harder, yeah.

Because women are attracted to status and power and men are attracted to youth and beauty. No, I mean, that's what I mean.

Because women are attracted to status and power and men are attracted to youth and beauty. No, I mean, that's what I mean.

No machines do not have any status or real power.

No machines do not have any status or real power.

But status fundamentally is a zero-sum game, whereas youth and beauty are not.

But status fundamentally is a zero-sum game, whereas youth and beauty are not.

I just think that that's why it's harder. You know, yeah, maybe it is my biases. I think status is way easier to fake. I also think that, you know, men are probably more desperate and more likely to buy my product. So maybe they're a better target market.

I just think that that's why it's harder. You know, yeah, maybe it is my biases. I think status is way easier to fake. I also think that, you know, men are probably more desperate and more likely to buy my product. So maybe they're a better target market.

Yeah. Look, I mean, look, I know you can look at porn viewership numbers, right? A lot more men watch porn than women. Yeah. You can ask why that is.

Yeah. Look, I mean, look, I know you can look at porn viewership numbers, right? A lot more men watch porn than women. Yeah. You can ask why that is.

Oh, man. And I'll tell you why it's six. Yeah. So AMD EPYC processors have 128 lanes of PCIe. I want to leave enough lanes for some drives, and I want to leave enough lanes for some networking.

Oh, man. And I'll tell you why it's six. Yeah. So AMD EPYC processors have 128 lanes of PCIe. I want to leave enough lanes for some drives, and I want to leave enough lanes for some networking.

Ah, that's one of the big challenges. Not only do I want the cooling to be good, I want it to be quiet. I want the tiny box to be able to sit comfortably in your room.

Ah, that's one of the big challenges. Not only do I want the cooling to be good, I want it to be quiet. I want the tiny box to be able to sit comfortably in your room.

I'll give a more, I mean, I can talk about how it relates to company number one.

I'll give a more, I mean, I can talk about how it relates to company number one.

No, no, quiet because you want to put this thing in your house and you want it to coexist with you. If it's screaming at 60 dB, you don't want that in your house. You'll kick it out.

No, no, quiet because you want to put this thing in your house and you want it to coexist with you. If it's screaming at 60 dB, you don't want that in your house. You'll kick it out.

Yeah, I want like 40, 45.

Yeah, I want like 40, 45.

A key trick is to actually make it big. Ironically, it's called the tiny box. But if I can make it big, a lot of that noise is generated because of high pressure air. If you look at like a 1U server, a 1U server has these super high pressure fans. They're like super deep and they're like genesis. Versus if you have something that's big, well, I can use a big, you know, they call them big ass fans.

A key trick is to actually make it big. Ironically, it's called the tiny box. But if I can make it big, a lot of that noise is generated because of high pressure air. If you look at like a 1U server, a 1U server has these super high pressure fans. They're like super deep and they're like genesis. Versus if you have something that's big, well, I can use a big, you know, they call them big ass fans.

Those ones that are like huge on the ceiling and they're completely silent.

Those ones that are like huge on the ceiling and they're completely silent.

It is the... I do not want it to be large according to UPS. I want it to be shippable as a normal package, but that's my constraint there.

It is the... I do not want it to be large according to UPS. I want it to be shippable as a normal package, but that's my constraint there.

No, it has to be... Well, you're... Look, I want to give you a great out-of-the-box experience. I want you to lift this thing out. I want it to be like the Mac, you know? TinyBox.

No, it has to be... Well, you're... Look, I want to give you a great out-of-the-box experience. I want you to lift this thing out. I want it to be like the Mac, you know? TinyBox.

Yeah. We did a poll. If people want Ubuntu or Arch, we're going to stick with Ubuntu.

Yeah. We did a poll. If people want Ubuntu or Arch, we're going to stick with Ubuntu.

There's a really simple way to get these models into TinyGrad and you can just export them as ONIX and then TinyGrad can run ONIX. So the ports that I did of Lama, Stable Diffusion, and now Whisper are more academic to teach me about the models, but they are cleaner than the PyTorch versions. You can read the code. I think the code is easier to read. It's less lines.

There's a really simple way to get these models into TinyGrad and you can just export them as ONIX and then TinyGrad can run ONIX. So the ports that I did of Lama, Stable Diffusion, and now Whisper are more academic to teach me about the models, but they are cleaner than the PyTorch versions. You can read the code. I think the code is easier to read. It's less lines.

There's just a few things about the way TinyGrid writes things. Here's a complaint I have about PyTorch. nn.relu is a class, right? So when you create an nn module, you'll put your nn.relus in an int. And this makes no sense. ReLU is completely stateless. Why should that be a class?

There's just a few things about the way TinyGrid writes things. Here's a complaint I have about PyTorch. nn.relu is a class, right? So when you create an nn module, you'll put your nn.relus in an int. And this makes no sense. ReLU is completely stateless. Why should that be a class?

Oh, no, it doesn't have a cost on performance. But yeah, no, I think that it's... That's what I mean about TinyGrad's front end being cleaner.

Oh, no, it doesn't have a cost on performance. But yeah, no, I think that it's... That's what I mean about TinyGrad's front end being cleaner.

I think that there is a spectrum and like on one side you have Mojo and on the other side you have like GGML. GGML is this like we're going to run Llama fast on Mac. And okay, we're going to expand out to a little bit, but we're going to basically go like depth first, right? Mojo is like, we're going to go breadth first. We're going to go so wide that we're going to make all of Python fast.

I think that there is a spectrum and like on one side you have Mojo and on the other side you have like GGML. GGML is this like we're going to run Llama fast on Mac. And okay, we're going to expand out to a little bit, but we're going to basically go like depth first, right? Mojo is like, we're going to go breadth first. We're going to go so wide that we're going to make all of Python fast.

And TinyGrad's in the middle. TinyGrad is, we are going to make neural networks fast.

And TinyGrad's in the middle. TinyGrad is, we are going to make neural networks fast.

Yeah, but they have turn completeness.

Yeah, but they have turn completeness.

My goal is step one, build an equally performance stack to PyTorch on NVIDIA and AMD, but with way less lines. And then step two is, okay, how do we make an accelerator, right? But you need step one. You have to first build the framework before you can build the accelerator.

My goal is step one, build an equally performance stack to PyTorch on NVIDIA and AMD, but with way less lines. And then step two is, okay, how do we make an accelerator, right? But you need step one. You have to first build the framework before you can build the accelerator.

So I'm much more of a, like, build it the right way and worry about performance later. There's a bunch of things where I haven't even, like, really dove into performance. The only place where TinyGrad is competitive performance-wise right now is on Qualcomm GPUs. So TinyGrad's actually used an open pilot to run the model. So the driving model is TinyGrad. When did that happen, that transition?

So I'm much more of a, like, build it the right way and worry about performance later. There's a bunch of things where I haven't even, like, really dove into performance. The only place where TinyGrad is competitive performance-wise right now is on Qualcomm GPUs. So TinyGrad's actually used an open pilot to run the model. So the driving model is TinyGrad. When did that happen, that transition?

About eight months ago now. And it's 2x faster than Qualcomm's library.

About eight months ago now. And it's 2x faster than Qualcomm's library.

It's a Snapdragon 845. Okay. So this is using the GPU. So the GPU is an Adreno GPU. There's like different things. There's a really good Microsoft paper that talks about like mobile GPUs and why they're different from desktop GPUs. One of the big things is in a desktop GPU, you can use buffers. On a mobile GPU, image textures are a lot faster.

It's a Snapdragon 845. Okay. So this is using the GPU. So the GPU is an Adreno GPU. There's like different things. There's a really good Microsoft paper that talks about like mobile GPUs and why they're different from desktop GPUs. One of the big things is in a desktop GPU, you can use buffers. On a mobile GPU, image textures are a lot faster.

I want to be able to leverage it in a way that it's completely generic, right? So there's a lot of this. Xiaomi has a pretty good open source library for mobile GPUs called Mace, where they can generate, where they have these kernels, but they're all hand-coded, right? So that's great if you're doing three by three confs. That's great if you're doing dense map models.

I want to be able to leverage it in a way that it's completely generic, right? So there's a lot of this. Xiaomi has a pretty good open source library for mobile GPUs called Mace, where they can generate, where they have these kernels, but they're all hand-coded, right? So that's great if you're doing three by three confs. That's great if you're doing dense map models.

But the minute you go off the beaten path a tiny bit, well, your performance is nothing.

But the minute you go off the beaten path a tiny bit, well, your performance is nothing.

You know, almost no one talks about FSD anymore, and even less people talk about OpenPilot. We've solved the problem. Like, we solved it years ago.

You know, almost no one talks about FSD anymore, and even less people talk about OpenPilot. We've solved the problem. Like, we solved it years ago.

Solving means how do you build a model that outputs a human policy for driving? How do you build a model that, given a reasonable set of sensors, outputs a human policy for driving? So you have companies like Waymo and Cruise, which are hand-coding these things that are like quasi-human policies.

Solving means how do you build a model that outputs a human policy for driving? How do you build a model that, given a reasonable set of sensors, outputs a human policy for driving? So you have companies like Waymo and Cruise, which are hand-coding these things that are like quasi-human policies.

Then you have Tesla, and maybe even to more of an extent, Coma, asking, okay, how do we just learn the human policy from data? The big thing that we're doing now, and we just put it out on Twitter, at the beginning of Comma, we published a paper called Learning a Driving Simulator. And the way this thing worked was it was an autoencoder and then an RNN in the middle. Right.

Then you have Tesla, and maybe even to more of an extent, Coma, asking, okay, how do we just learn the human policy from data? The big thing that we're doing now, and we just put it out on Twitter, at the beginning of Comma, we published a paper called Learning a Driving Simulator. And the way this thing worked was it was an autoencoder and then an RNN in the middle. Right.

You take an auto encoder, you compress the picture, you use an RNN, predict the next state. And these things were, you know, it was a laughably bad simulator, right? This is 2015 era machine learning technology. Today we have VQVAE and transformers. We're building drive GPT basically.

You take an auto encoder, you compress the picture, you use an RNN, predict the next state. And these things were, you know, it was a laughably bad simulator, right? This is 2015 era machine learning technology. Today we have VQVAE and transformers. We're building drive GPT basically.

It's trained on all the driving data to predict the next frame.

It's trained on all the driving data to predict the next frame.

Well, actually our simulator is conditioned on the pose. So it's actually a simulator. You can put in like a state action pair and get out the next state. Okay. And then once you have a simulator, you can do RL in the simulator and RL will get us that human policy.

Well, actually our simulator is conditioned on the pose. So it's actually a simulator. You can put in like a state action pair and get out the next state. Okay. And then once you have a simulator, you can do RL in the simulator and RL will get us that human policy.

Yeah. RL with a reward function, not asking is this close to the human policy, but asking would a human disengage if you did this behavior?

Yeah. RL with a reward function, not asking is this close to the human policy, but asking would a human disengage if you did this behavior?

It's a nice... It's asking exactly the right question. What will make our customers happy?

It's a nice... It's asking exactly the right question. What will make our customers happy?

A system that you never want to disengage.

A system that you never want to disengage.

Usually. There's some that are just, I felt like driving. And those are always fine too. But they're just going to look like noise in the data.

Usually. There's some that are just, I felt like driving. And those are always fine too. But they're just going to look like noise in the data.

Maybe, yeah.

Maybe, yeah.

It's hard to say. We haven't completely closed the loop yet. So we don't have anything built that truly looks like that architecture yet. Mm-hmm. We have prototypes and there's bugs. So we are a couple bug fixes away. Might take a year, might take 10.

It's hard to say. We haven't completely closed the loop yet. So we don't have anything built that truly looks like that architecture yet. Mm-hmm. We have prototypes and there's bugs. So we are a couple bug fixes away. Might take a year, might take 10.