Nick Heiner

Yeah, so there are a number of model smell things like that, just sort of these weird quirks.

One thing it can come out of is training on synthetically generated data, where the synthetic data is insufficiently diverse.

So we saw a model checkpoint one time that

Like you would ask it to write like a paragraph, like a five paragraph essay.

And every single paragraph was like starting with, you know, like in high school, a very rigid formula you get.

Yeah.

It was like, start with what you want to say in the paragraph and then say it and then end with the conclusion, you know, sort of restating what you said.

And there's a reason that when most of us graduate from 11th grade, like we move beyond that structure.

Yeah.

So it's not actually what you want.

But because this lab had trained on a ton of synthetic data that was generated in precisely that format and said, oh, well, you know, you know, they it's like, well, we don't have quality, but we do have quantity.

So let's just spit out this.

Then, yeah, the model sort of overlearns that that's what good writing looks like.

Yeah, exactly right.

If it's commonly associated with what otherwise looks like good writing, then yeah, you'll sort of see some of those vestigial structures get picked up.

Yeah.

Well, I mean, it's sort of interesting because it gets into like watermarking stuff too, where it is genuinely useful for society to know if something is produced by a model.

Yeah.

And to the extent that labs actually leave that in, like, it's actually kind of nice for them to do that.

Yeah, yeah.