Eliezer Yudkowsky

If you don't tell me the sum of the two numbers, and you tell me the first die showed 6, this doesn't tell me anything about the result of the second die, yet.

But, if you now also tell me the sum is 7, I know the second die showed 1.

Figuring out when various pieces of information are dependent or independent of each other, given various background knowledge, actually turns into quite a technical topic.

The books to read are Judea Pearl's Probabilistic Reasoning in Intelligence Systems, Networks of Plausible Inference, and Judea Pearl's Causality.

If you only have time to read one book, read the first one.

If you know how to read causal graphs, then you look at the dice roll graph and immediately see the probability of die 1 and die 2 equals the probability of die 1 times the probability of die 2.

Or the probability of die 1 and die 2 given the sum does not equal the probability of die 1 given the sum times the probability of die 2 given the sum.

If you look at the correct sidewalk diagram, you see facts like the probability of slippery given night does not equal the probability of slippery, or the probability of slippery given sprinkler does not equal the probability of slippery.

But the probability of slippery given night and sprinkler equals the probability of slippery given sprinkler.

That is, the probability of the sidewalk being slippery given knowledge about the sprinkler and the night is the same probability we would assign if we knew only about the sprinkler.

Knowledge of the sprinkler has made knowledge of the night irrelevant to inferences about slipperiness.

This is known as screening off, and the criterion that lets us read such conditional independences off causal graphs is known as de-separation.

For the case of argument and authority, the causal diagram looks like this.

Here the author has the word truth with an arrow pointing to the right, and the words argument goodness, and an arrow pointing to the right, and the words expert belief.

That's truth, arrow, argument goodness, arrow, expert belief.

If something is true, then it therefore tends to have arguments in favor of it, and the experts therefore observe these evidences and change their opinions, in theory.

If we see that an expert believes something, we infer back to the existence of evidence in the abstract, even though we don't know what that evidence is exactly, and from the existence of this abstract evidence, we infer back to the truth of the proposition.

But if we know the value of the argument node, this de-separates the node truth from the node expert belief by blocking all paths between them, according to certain technical criteria for path blocking that seem pretty obvious in this case.

This does not represent a contradiction of ordinary probability theory.

It's just a more compact way of expressing certain probabilistic facts.