Read our related article: Doomy thinking
IF OUR impressive capacity for conscious reasoning is what most clearly sets us humans apart from other species, our capacity for self-deception over the significance of its conclusions must come a close second.
In the real world, this can be serious. To take a topical example, until recently, people who are supposed to know about such things were utterly confident that serious financial crises were a thing of the past. Financial mathematics of supposedly unprecedented sophistication said so. Just look at us now.
Advertisement
So it’s timely to ask if we might be suffering from similar self-deception about other risks we think we understand. A trenchant new analysis by and his colleagues at the Future of Humanity Institute at the University of Oxford gives pause for thought. The team’s paper concerns the risks associated with extremely rare but potentially catastrophic events. They cite the start-up of the Large Hadron Collider, which was challenged on the grounds that the high-energy particle collisions it generates might annihilate the Earth by creating a tiny black hole or a deadly shard of strange matter.
In a spectacular (and intentional) understatement, the physicists who first tried to put numbers on such risks – then in the context of the Relativistic Heavy Ion Collider at Brookhaven, New York, a predecessor to the LHC – referred to potentially “profound implications for health and safety”. Invoking quantum chromodynamics, what we know about the gravitational conditions for creating black holes, and our knowledge of near-Earth high-energy collisions due to cosmic rays, they put the chances of a “dangerous event” at around 10-9 per year.
That sounds pretty safe, and similarly small or smaller numbers have been cited for the LHC. But as Ord’s team argues, this is not as reassuring as it seems: that’s because this figure represents the chance of a dangerous event only if the physicists’ argument is correct. What if it isn’t?
Finding out requires some careful logic. Forget the LHC for the moment. Suppose some argument estimates probability X of an event, good or bad, happening. From a probabilistic point of view, Ord and colleagues point out, we cannot just accept this as the best estimate. The real probability is:
(X Ă— PX) + (Y Ă— PY)
that is, X multiplied by the probability (PX) that the argument is correct, plus the probability the argument is wrong (PY) multiplied by the chance (Y) that the event will happen if the argument is wrong.
This matters hardly at all if X has some reasonably high value such as 50 per cent, or even 1 per cent. But if X is very small, say 10-20, while the chance the argument is wrong is 10-6, then the first figure becomes as good as meaningless. As the authors put it: “If the probability estimate given by an argument is dwarfed by the chance that the argument itself is flawed, then the estimate is suspect.”
In other words, conclusions about extraordinarily small probabilities require equally extraordinary care. Any estimate of the likelihood of an event occurring should take into account the chance that the analysis on which that probability has been assigned is flawed or based on error. This is the principle of epistemological uncertainty – that we can’t know anything with total certainty – in action.
Back to the LHC. Many people, me included, find the most convincing argument for the safety of the collider comes from considering the energy densities frequently created by cosmic rays colliding with particles in the Earth’s atmosphere or elsewhere. If the LHC is likely to be dangerous, we should long since have been annihilated. Even so, as convincing as this is, the same problem recurs. The argument has to be more than convincing: we have to be sure there is only the tiniest chance we could be wrong – which is much harder to establish.
Lots of arguments do turn out to be wrong. To get an idea of how many, Ord and his colleagues looked at the proportion of published scientific papers that are eventually retracted. Using as their source, they found it to be around 1 in 10,000. Given that Medline covers mostly top-ranking journals, they estimate the error rate for all journals to be more like 1 in 1000, which suggests a fairly high chance of error in any argument. And remember, scientists often don’t publish painful and embarrassing retractions unless forced, so the true rate may be even higher.
Of course, the physicists who considered the LHC risks are capable and cautious people so we might think the chance of their argument being wrong is smaller than 1 in 1000. But even if it were a thousand times smaller, only 10-6, this still seems to undermine the authority of their extremely small estimates.
Of course, it is equally important not to be too pessimistic either. The fact is that we have no knowledge at all about the likelihood of an event happening if one or more of its underlying arguments are wrong. Even if the argument does not reach the required level of certainty, this does not mean we are in danger – only that we don’t have any guarantee of safety. Not the same thing at all.
Ord and his colleagues rightly stress that further elaboration of the arguments for the safety of the LHC might well reduce the chance of the overall argument for its safety being wrong. But until this kind of work has been done, they suggest, the current safety report cannot be seen as the final word, which seems entirely reasonable to me.
This is an area where it is crucial to focus on the logic, because our intuitions are no help. Most of us, I suspect, have a gut feeling that certain things “could never happen” and that “people who worry about this are crazy”. Sadly, the fact that we haven’t destroyed ourselves yet is no guarantee that we never will.
It’s easy for any of us to be seduced by the nature of logical thinking and its illusion of certainty. We generally strive to become aware of what former US defence secretary Donald Rumsfeld and the “known unknowns”, but are perilously ignorant of the “unknown unknowns”, and, worse, blithely unaware of our own ignorance. This becomes particularly dangerous when it hides flaws in an argument we are relying on for reassurance that potentially catastrophic events are virtually impossible.
“It’s easy to be seduced by the nature of logical thinking and its illusion of certainty”
It is perhaps an ultimate irony of our human longing for certainty that no amount of effort can definitively prove anything, for we can never discount the possibility that we have made a mistake, and if we enlist others to help us, they too may make mistakes.
Maybe there’s another lesson here: that mathematical certainty isn’t all it’s cracked up to be. We are, after all, biological organisms, and biology never bothers to prove anything. Cell design, for instance, reflects a crazy historical legacy of structures cobbled together to produce workable solutions to thousands of temporary problems. Perhaps we shouldn’t insist that good reasoning conforms to some “pure” proof-making ideal, and accept it is more like those ad hoc biological processes that leave us fallible and vulnerable, but resilient enough to get by in an uncertain world.
Read our related article: Doomy thinking
Profile
is a consultant for żěè¶ĚĘÓƵ, based in France. His books are Nexus, Ubiquity and The Social Atom
