Eliezer Yudkowsky

Arthur assigns 90% probability to David's statement after hearing his explanation, but assigns a 10% probability to Ernie's statement.

It might seem like these two scenarios are roughly symmetrical.

Both involve taking into account useful evidence, whether strong versus weak authority or strong versus weak argument.

But now suppose that Arthur asks Barry and Charles to make full technical cases with references and that Barry and Charles present equally good cases and Arthur looks up the references and they check out.

Then Arthur asks David and Ernie for their credentials and it turns out that David and Ernie have roughly the same credentials.

Maybe they're both clowns or maybe they're both physicists.

Assuming that Arthur is knowledgeable enough to understand all the technical arguments, otherwise they're just impressive noises, it seems that Arthur should view David as having a great advantage in plausibility over Ernie, while Barry has at best a minor advantage over Charles.

Indeed, if the technical arguments are good enough, Barry's advantage over Charles may not be worth tracking.

A good technical argument is one that eliminates reliance on the personal authority of the speaker.

Similarly, if we really believe Ernie that the argument he gave is the best argument he could give, which includes all the inferential steps that Ernie executed and all of the support that Ernie took into account, citing any authorities that Ernie may have listened to himself, then we can pretty much ignore any information about Ernie's credentials.

Ernie can be a physicist or a clown.

It shouldn't matter.

Again, this assumes we have enough technical ability to process the argument.

Otherwise, Ernie is simply uttering mystical syllables, and whether we believe these syllables depends a great deal on his authority.

So, it seems there's an asymmetry between argument and authority.

If we know authority, we are still interested in hearing the arguments, but if we know the arguments fully, we have very little left to learn from authority.

Clearly, says the novice, authority and argument are fundamentally different kinds of evidence, a difference unaccountable in the boringly clean methods of Bayesian probability theory.

For while the strength of the evidences, 90% versus 10%, is just the same in both cases, they do not behave similarly when combined.

How, oh how, will we account for this?

Well, here's half a technical demonstration of how to represent this difference in probability theory.