Sam Altman

But GPT-4, I think we can get to do this.

No, it's a lot of technical leaps in the base model.

One of the things we are good at at OpenAI is finding a lot of small wins and multiplying them together.

And each of them maybe is like a pretty big secret in some sense, but it really is the multiplicative impact of all of them.

And the detail and care we put into it that gets us these big leaps.

And then, you know, it looks like to the outside, like, oh, they just probably like did one thing to get from three to 3.5 to four.

It's like hundreds of complicated things.

So tiny little thing with the training, with the, like everything, with the data organization.

How we like collect the data, how we clean the data, how we do the training, how we do the optimizer, how we do the architect, like so many things.

I heard GPT-4 had 100 trillion.

I can't believe that this came from you.

You know, it doesn't, I don't think it matters in any serious way.

This is like, in some sense, someone said to me this morning, actually, and I was like, oh, this might be right.

This is the most complex software object humanity has yet produced.

And it will be trivial in a couple of decades, right?

It'll be like kind of anyone can do it, whatever.

But yeah, the amount of complexity relative to anything we've done so far that goes into producing this one set of numbers is quite something.

By like number of parameters?

I think people got caught up in the parameter count race in the same way they got caught up in the gigahertz race of processors and like the, you know, nineties and two thousands or whatever.

You, I think probably have no idea how many gigahertz the processor in your phone is, but what you care about is what the thing can do for you.