Eliezer Yudkowsky

Produce more outputs like that one.

I agree with this output.

This output is persuasive.

even in the field of effective altruism.

You quite recently had people publishing papers about like, ah, yes, well, you know, to get something at human level intelligence, it needs to have like this many parameters and you need to like do this much training of it with this many tokens according to the scaling laws and at the rate that Moore's law is going, at the rate that software is going, it'll be in 2050.

And me going like,

What?

You don't know any of that stuff.

This is like this one weird model that has all kinds of like, you have done a calculation that does not obviously bear on reality anyways.

And this is like a simple thing to say, but you can also produce a whole long paper

impressively arguing out all the details of how you got the number of parameters and how you're doing this impressive huge wrong calculation.

And I think most of the effective altruists who are paying attention to this issue, the larger world paying no attention to it at all,

you know, or just like nodding along with a giant impressive paper because, you know, you like press thumbs up for the giant impressive paper and thumbs down for the person going like, I don't think that this paper bears any relation to reality.

And I do think that we are now seeing with like GPT-4 and the sparks of AGI that,

Possibly, depending on how you define that even.

I think that EAs would now consider themselves less convinced by the very long paper on the argument from biology as to AGI being 30 years off.

But this is what people pressed thumbs up on.

And if you train an AI system to make people press thumbs up, maybe you get these long, elaborate, impressive papers arguing for things that ultimately fail to bind to reality, for example.

And it feels to me like I have watched the field of alignment just fail to thrive and

except for these parts that are doing these relatively very straightforward and legible problems, like finding the induction heads inside the giant inscrutable matrices.