George Sivulka

And maybe we're at the end of scaling law as a training. But I actually think, you know, Hebbia and now OpenAI and a variety of other companies are starting to pioneer the idea of scaling laws and inference. And I actually think that it will make nothing that that other players can do to fine tune models will ever catch up.

Yeah, I think it's a bit of a marketing distinction. But ultimately, the idea is that the way that we got here over the last five, seven years of training models has been, let's build a bigger and bigger model, and let's give it more and more data, more and more clean data. And then maybe we'll do some RLHF or some reinforcement training to fine tune it after pre-training.

Yeah, I think it's a bit of a marketing distinction. But ultimately, the idea is that the way that we got here over the last five, seven years of training models has been, let's build a bigger and bigger model, and let's give it more and more data, more and more clean data. And then maybe we'll do some RLHF or some reinforcement training to fine tune it after pre-training.

Yeah, I think it's a bit of a marketing distinction. But ultimately, the idea is that the way that we got here over the last five, seven years of training models has been, let's build a bigger and bigger model, and let's give it more and more data, more and more clean data. And then maybe we'll do some RLHF or some reinforcement training to fine tune it after pre-training.

And that worked great to get us here. But we're running up against the amount of good data that exists in the world. We're running up against. Are we?

And that worked great to get us here. But we're running up against the amount of good data that exists in the world. We're running up against. Are we?

And that worked great to get us here. But we're running up against the amount of good data that exists in the world. We're running up against. Are we?

You know, I think that we're starting to run up against the constraints of it. That's a gut feel. I'm not, you know, I'm not looking at particularly in data collection myself, but I think we're starting to run up against the limits of really good data. What's then the problem? So ultimately, that might mean that. hey, we're training larger and larger models.

You know, I think that we're starting to run up against the constraints of it. That's a gut feel. I'm not, you know, I'm not looking at particularly in data collection myself, but I think we're starting to run up against the limits of really good data. What's then the problem? So ultimately, that might mean that. hey, we're training larger and larger models.

You know, I think that we're starting to run up against the constraints of it. That's a gut feel. I'm not, you know, I'm not looking at particularly in data collection myself, but I think we're starting to run up against the limits of really good data. What's then the problem? So ultimately, that might mean that. hey, we're training larger and larger models.

XAI, again, just created the largest GPU cluster of all time, and they're going to try to train larger and larger models.

XAI, again, just created the largest GPU cluster of all time, and they're going to try to train larger and larger models.

XAI, again, just created the largest GPU cluster of all time, and they're going to try to train larger and larger models.

But regardless of how the scaling laws for training larger models or parameter count and accuracy or performance carry out, I'm starting to believe that you could still get better compute not by building a larger engine, to use a metaphor, but by actually putting a bunch of smaller engines together.

But regardless of how the scaling laws for training larger models or parameter count and accuracy or performance carry out, I'm starting to believe that you could still get better compute not by building a larger engine, to use a metaphor, but by actually putting a bunch of smaller engines together.

But regardless of how the scaling laws for training larger models or parameter count and accuracy or performance carry out, I'm starting to believe that you could still get better compute not by building a larger engine, to use a metaphor, but by actually putting a bunch of smaller engines together.

Hebbia, by orchestrating large amounts of inference to answer one single question, ends up kind of building like a Tesla, where Tesla is made of a bunch of smaller engines or a bunch of smaller electromechanical motors that make a lot of torque and a really, really amazing larger engine. Does it not make it incredibly capital inefficient?

Hebbia, by orchestrating large amounts of inference to answer one single question, ends up kind of building like a Tesla, where Tesla is made of a bunch of smaller engines or a bunch of smaller electromechanical motors that make a lot of torque and a really, really amazing larger engine. Does it not make it incredibly capital inefficient?

Hebbia, by orchestrating large amounts of inference to answer one single question, ends up kind of building like a Tesla, where Tesla is made of a bunch of smaller engines or a bunch of smaller electromechanical motors that make a lot of torque and a really, really amazing larger engine. Does it not make it incredibly capital inefficient?

I think the one thing that people in my position will always tell you is that the cost of intelligence will go to zero. The cost of intelligence will go to zero. I think that since Hebbia started, the cost of inference over a fixed number of parameters has decreased by seven orders of magnitude in four years. And so I genuinely believe that scaling compute is like a no brainer.