Terry Pratchett was fond of saying that million-to-one chances crop up nine times out of ten. On the face of it, this sentence is mathematically absurd, but in the fantasy world of Pratchett’s Discworld books, powered by the magic of narratives, it makes perfect sense. Of course heroes are always going to face incredible odds, and of course they are almost always going to overcome them, because that is what heroes do.
I was reminded of this malleable concept of probability in recent weeks, as we covered the unfolding story of the possibility of alien life on the exoplanet K2-18b. In brief, Nikku Madhusudhan at the University of Cambridge and his colleagues claim to have found signs of a molecule called dimethyl sulphide (DMS) in the atmosphere of the planet. On Earth, DMS is produced only by living organisms, leading Madhusudhan to call these signs the “strongest evidence yet” for biological activity on another planet.
Advertisement
This was widely reported as a “99.7 per cent chance” of life on K2-18b, which might seem like pretty good odds – but in reality, it’s possible that the chance is actually close to zero.
Like million-to-one chances, 99.7 per cent chances crop up surprisingly often. To understand why, we need to take a look at the way scientists use statistics to tell stories and why, just as on Discworld, the underlying narrative can influence seemingly cold, hard numbers. Let’s start with a basic example. If I hand you a coin and ask you to flip it, what are the odds of it landing on heads? Assuming it is an ordinary coin, you would expect to see heads 50 per cent of the time, but that assumption is important – what if I have doctored the coin somehow, to make heads far more likely? Can you catch me out?
An obvious solution is to flip the coin several times and record the number of heads and tails – with a large enough number of flips of a fair coin, the tally of heads and tails should be pretty close, if not equal. But how large is “large enough”? For example, if you get three heads in a row, you might start thinking I have altered the coin, but the odds of this happening for a fair coin are (½)3, or 1 in 8 – hardly rare.
To answer this question, scientists turn to what is known as “hypothesis testing”. This begins with defining the “null hypothesis” – what, in the absence of anything unusual, we might expect to happen. This is in opposition to the “alternative hypothesis” – the thing we are trying to test. In the case of the coin, the null hypothesis is that it is a fair coin, while the alternative hypothesis is that I have altered it.
Next, we need to think about the probability distribution of our repeated coin flips. If we are flipping a fair coin three times, it’s simple enough to work out all the possible outcomes (HHH, HHT, HTH, HTT, THH, THT, TTH, TTT) and group these – we can see that the probability of three heads is 1 in 8, two heads is 3 in 8, one head is 3 in 8 and tails only is 1 in 8. Writing this out for a much larger number of coin flips would be arduous, but thankfully we don’t have to, because it turns out that for repeated trials of a simple success/failure, you can always use the same equation to calculate the probability distribution. This is known as the binomial distribution. I won’t run through the full maths here, .
With this equation, we use the results of coin flips to tell us something about the coin. Suppose we flip four times and get four heads. Assuming the null hypothesis, that the coin is fair, the probability of this is (½)4 or 6.25 per cent. Should you accept the null hypothesis, and declare the coin fair, or reject it, and accuse me of tampering? The dirty secret at the heart of science is that there is no objective way of making such a declaration – and yet, we do it all the time anyway!
Let me explain. You may be familiar with the phrase “statistically significant”, though it is one we tend to avoid at èƵ, because something that is statistically significant is not necessarily significant in the ordinary sense of the word. What it means is that if we assume the null hypothesis, and the probability of an outcome – known as the p-value – falls below a particular level known as the significance level, we should reject that null hypothesis.
In many areas of science, the significance level is set at 5 per cent, meaning we are seeking a result that allows us to be 95 per cent certain it is not due to random chance, but this is little more than a historical convention. In the four-heads example, my p-value is 6.25 per cent, and so higher than the significance level of 5 per cent. It seems there is not enough evidence to think that there is anything funny about this coin.
But hang on, what if we flip five times, and get five heads? The probability of this is 3.125 per cent, which is below the significance level. Hurrah – we have made a statistically significant discovery! It turns out you should probably accuse me of fiddling with the coin after all – assuming that you agree that a p-value below 5 per cent is strong enough evidence.
Although…it feels a bit strange and arbitrary that just one more coin flip should be enough to change your mind, right? Is it actually significant? Indeed, some unscrupulous scientists might tweak their data analysis to get things just below the significance level, a murky process known as p-hacking. Really, a lot of what we might think of as objective statistics begins to look like it is on shaky ground if you start poking around the foundations.
OK, but what does all of this have to do with aliens? Instead of significance levels, astronomers, like physicists of various stripes, prefer to talk of “sigma” levels – this refers to the Greek letter σ, which is used to denote the standard deviation of a sample, a measure of how much it varies around the mean, or average, value.
Using a normal distribution, or bell curve (again, that relies on assumptions!), 68 per cent of values will be within 1 sigma of the mean and 95 per cent within 2 sigma. This can be directly related to p-values – if you set your significance level at 5 per cent, you need a p-value of beyond 2 sigma to have made a “statistically significant” discovery, because that 5 per cent lies outside the 95 per cent in the centre. As a reminder, if a p-value falls below the significance level, we can reject the null hypothesis, in this case with 95 per cent certainty that the results observed are not due to random chance.
In general, physicists normally demand higher certainty than this. One reason for this is practicality, as testing a new drug on messy humans will always produce weaker evidence than smashing particles together, meaning that biologists insisting on more stringent significance levels could potentially struggle to ever make a statistically significant finding. Another is the culture of the physics community, which again is certainly not an objective way to define anything. For example, the discovery of the Higgs boson was announced with fanfare in 2012 once the data accumulated reached 5 sigma, equivalent to being outside 99.99994 per cent of the normal distribution. By convention, this is a “discovery” in physics, but it is an entirely artificial boundary, and the sigma level for the Higgs had been gradually growing as physicists gathered more data – the choice of when to celebrate has no objective basis.
And now, back to those aliens. Physicists normally classify a 3-sigma result as “interesting” – short of a discovery but meriting further investigation. This is equivalent to a result laying outside of the 99.7 per cent of values within 3 standard deviations of the mean. Ah, now that’s a familiar number! The use of 3 sigma as a significance level explains why 99.7 per cent chances crop up so often, because a statistically significant result in this case means we are 99.7 per cent sure the data we have gathered is not due to random chance.
In the case of K2-18b, what does that actually mean? Here’s where the assumptions – the storytelling – inherent in statistics become so important. The statistical analysis performed by Madhusudhan and his colleagues is complex and involves many steps that I won’t go into here, but the 3-sigma detection – the 99.7 per cent chance – hinges on comparing a model that includes DMS in the planet’s atmosphere with one that doesn’t. Building these models involves a number of assumptions, but you can make others – and indeed, other astronomers have done so, finding that the same data is consistent with a model that is just a straight line, i.e. literally detecting nothing.
At this point, if I were writing a novel, I would pull out an incredible twist ending, just as the rules of storytelling demand – maybe something like “there were no signs of aliens on K2-18b BECAUSE THEY ARE ALREADY HERE ON EARTH!”. But unlike on Discworld, where the element narrativium allows the power of stories to rewrite reality, the stories we tell with statistics are much less powerful. Remember that the next time you see a big discovery.
