Dario Amodei

It just tells you about the pace. Yeah. The expectations for when things are going to come out.

Naming is actually an interesting challenge here, right? Because I think a year ago, most of the model was pre-training. And so you could start from the beginning and just say, okay, we're going to have models of different sizes. We're going to train them all together. And, you know, we'll have a family of naming schemes and then we'll put some new magic into them.

And then, you know, we'll have the next, the next generation. The trouble starts already when some of them take a lot longer than others to train, right? That already messes up your time, time a little bit, but yeah, As you make big improvements in pre-training, then you suddenly notice, oh, I can make better pre-trained model, and that doesn't take very long to do.

But, you know, clearly it has the same, you know, size and shape of previous models. Uh, uh, so I think those two together, as well as the timing, timing issues, any kind of scheme you come up with, uh, you know, the reality tends to kind of frustrate that scheme, right? It tends to kind of break out of the breakout of the scheme. It's not like software where you can say, oh, this is like,

you know, 3.7, this is 3.8. No, you have models with different, different trade-offs. You can change some things in your models. You can train, you can change other things. Some are faster and slower at inference. Some have to be more expensive. Some have to be less expensive. And so I think all the companies have struggled with this.

I think we did very, you know, I think, think we were in a good, good position in terms of naming when we had Haiku, Sonnet and Opus. Great start. We're trying to maintain it, but it's not perfect. So we'll try and get back to the simplicity, but just the nature of the field, I feel like no one's figured out naming. It's somehow a different paradigm from normal software. And so...

we just, none of the companies have been perfect at it. It's something we struggle with surprisingly much relative to how trivial it is for the grand science of training the models. So from the user side,

Yeah. Yeah. I definitely think this question of there are lots of properties of the models that are not reflected in the benchmarks. I think I think that's that's definitely the case. And everyone agrees. And not all of them are capabilities. Some of them are, you know, models can be polite or brusque. They can be, you know, very reactive or they can ask you questions.

They can have what feels like a warm personality or a cold personality. They can be boring or they can be very distinctive like Golden Gate Claude was. And we have a whole, you know, we have a whole team kind of focused on, I think we call it Claude character. Amanda leads that team and we'll talk to you about that. But it's still a very inexact science.

And often we find that models have properties that we're not aware of. The fact of the matter is that you can talk to a model 10,000 times and there are some behaviors you might not see. Just like with a human, right? I can know someone for a few months and not know that they have a certain skill or not know that there's a certain side to them. And so I think we just have to get used to this idea.

And we're always looking for better ways of testing our models to demonstrate these capabilities. And And also to decide which are the personality properties we want models to have and which we don't want to have. That itself, the normative question, is also super interesting.

From Reddit. Oh, boy.

So this actually doesn't apply. This isn't just about Claude. I believe I've seen these complaints for every foundation model produced by a major company. People said this about GPT-4. They said it about GPT-4 Turbo. So a couple things. One, the actual weights of the model, right, the actual brain of the model, that does not change unless we introduce a new model.

There are just a number of reasons why it would not make sense practically to be randomly substituting in substituting in new versions of the model. It's difficult from an inference perspective, and it's actually hard to control all the consequences of changing the weights of the model.

Let's say you wanted to fine tune the model to be like, I don't know, to like, to say certainly less, which, you know, an old version of Sonnet used to do. You actually end up changing a hundred things as well. So we have a whole process for it. And we have a whole process for

modifying the model we do a bunch of testing on it we do a bunch of um like we do a bunch of user testing and early customers so it we both have never changed the weights of the model without without telling anyone and it it wouldn't certainly in the current setup it would not make sense to do that now there are a couple things that we do occasionally do um one is sometimes we run ab tests um

Um, but those are typically very close to when a model is being, is being, uh, released and for a very small fraction of time. Um, so, uh, you know, like the, you know, the, the day before the new sonnet 3.5, I agree. We should have had a better name. It's clunky to refer to it. Um, there were some comments from people that like, it's got, it's got, it's gotten a lot better.

And that's because, you know, a fraction were exposed to, to an AB test for, for those one or for those one or two days. Um, the other is that occasionally the system prompt will change, um, on the system prompt can have some effects, although it's on, it's unlikely to dumb down models. It's unlikely to make them dumber.

Um, and, and, and, and we've seen that while these two things, which I'm listing to be very complete, um, happen relatively, happen quite infrequently. The complaints for us and for other model companies about the model change, the model isn't good at this, the model got more censored, the model was dumbed down, those complaints are constant.

And so I don't want to say people are imagining it or anything, but the models are, for the most part, not changing. If I were to offer a theory, I think it actually relates to one of the things I said before, which is that Models are very complex and have many aspects to them.