George Hotz

And that's your multiplied matrix. So what a matrix multiply is, is a bunch of shape operations, right? A bunch of permute three shapes and expands on the two matrices. A multiply, n cubed. A reduce, n cubed, which gives you an n squared matrix.

So TinyGrad has about 20. And you can compare TinyGrad's op set or IR to things like XLA or PrimTorch. So XLA and PrimTorch are ideas where like, okay, Torch has like 2000 different kernels. PyTorch 2.0 introduced PrimTorch, which has only 250. TinyGrad has order of magnitude 25. It's 10x less than XLA or Primtorch. And you can think about it as kind of like RISC versus CISC, right?

So TinyGrad has about 20. And you can compare TinyGrad's op set or IR to things like XLA or PrimTorch. So XLA and PrimTorch are ideas where like, okay, Torch has like 2000 different kernels. PyTorch 2.0 introduced PrimTorch, which has only 250. TinyGrad has order of magnitude 25. It's 10x less than XLA or Primtorch. And you can think about it as kind of like RISC versus CISC, right?

So TinyGrad has about 20. And you can compare TinyGrad's op set or IR to things like XLA or PrimTorch. So XLA and PrimTorch are ideas where like, okay, Torch has like 2000 different kernels. PyTorch 2.0 introduced PrimTorch, which has only 250. TinyGrad has order of magnitude 25. It's 10x less than XLA or Primtorch. And you can think about it as kind of like RISC versus CISC, right?

These other things are CISC-like systems. TinyGrad is RISC.

These other things are CISC-like systems. TinyGrad is RISC.

These other things are CISC-like systems. TinyGrad is RISC.

RISC architecture is going to change everything. 1995, hackers.

RISC architecture is going to change everything. 1995, hackers.

RISC architecture is going to change everything. 1995, hackers.

Angelina Jolie delivers the line, risk architecture is going to change everything in 1995. Wow. And here we are with ARM in the phones. And ARM everywhere.

Angelina Jolie delivers the line, risk architecture is going to change everything in 1995. Wow. And here we are with ARM in the phones. And ARM everywhere.

Angelina Jolie delivers the line, risk architecture is going to change everything in 1995. Wow. And here we are with ARM in the phones. And ARM everywhere.

Sure. Okay, so you have unary ops, which take in a tensor and return a tensor of the same size and do some unary op to it. X, log, reciprocal, sine, right? They take in one and they're point-wise.

Sure. Okay, so you have unary ops, which take in a tensor and return a tensor of the same size and do some unary op to it. X, log, reciprocal, sine, right? They take in one and they're point-wise.

Sure. Okay, so you have unary ops, which take in a tensor and return a tensor of the same size and do some unary op to it. X, log, reciprocal, sine, right? They take in one and they're point-wise.

Yeah, ReLU. Almost all activation functions are unary ops. Some combinations of unary ops together is still a unary op. Then you have binary ops. Binary ops are like pointwise addition, multiplication, division, compare. It takes in two tensors of equal size and outputs one tensor. Then you have reduce ops.

Yeah, ReLU. Almost all activation functions are unary ops. Some combinations of unary ops together is still a unary op. Then you have binary ops. Binary ops are like pointwise addition, multiplication, division, compare. It takes in two tensors of equal size and outputs one tensor. Then you have reduce ops.

Yeah, ReLU. Almost all activation functions are unary ops. Some combinations of unary ops together is still a unary op. Then you have binary ops. Binary ops are like pointwise addition, multiplication, division, compare. It takes in two tensors of equal size and outputs one tensor. Then you have reduce ops.

Reduce ops will take a three-dimensional tensor and turn it into a two-dimensional tensor, or a three-dimensional tensor and turn it into a zero-dimensional tensor. Think like a sum or a max are really the common ones there. And then the fourth type is movement ops. And movement ops are different from the other types because they don't actually require computation.