THE mathematician and mystic Blaise Pascal offered a famous argument for devout behaviour. Even if you thought it exceedingly unlikely that a vengeful God existed, he said, you’d be prudent and rational to behave as though He did, because it is worth paying the (finite) price of forgoing illicit pleasures in this life as an “insurance premium” to guard against even the smallest probability of something infinitely horrible – eternal hellfire – in the afterlife.
Pascal’s celebrated wager is an extreme version of the precautionary principle widely invoked in health and environmental policy. For example, the long-term consequences of genetically modified plants and animals for human health, and for ecological balance, are manifestly uncertain: a calamitous outcome may seem improbable, but we can’t say it’s impossible. According to proponents of the precautionary principle, the onus should be on the advocates of new techniques to convince the rest of us that such fears are ungrounded – or, at the very least, that the risks are small enough to be outweighed by some specific and substantial benefits.
If this argument were used to impose a blanket prohibition on all experiments and innovations involving explicit risk, science would be paralysed and we would be denied all its benefits. Aviation was once far more hazardous than it is now. Most surgical procedures, even if now routine, were risky and often fatal when they were being pioneered.
Advertisement
But that is not to say these arguments should never be considered for science and technology. Bill Joy, co-founder of Sun Microsystems and the inventor of the Java computer language, has famously suggested we relinquish the research and development that could lead to computers and robots with capabilities that surpass those of humans. He is not primarily concerned about malign misuse of the new technology; his fear is that genetics, nanotechnology and robotics may develop uncontrollably and, one day, take us over. He wishes to stave off the day, which he believes could come during this century, when the biosphere could crumble into “grey goo”, or super-intelligent robots displace us.
My own worries are nearer-term than Joy’s. Before the futuristic capabilities that he fears are attained, society could be dealt shattering blows by the misapplication of technology that exists already, or that we can confidently expect within the next 20 years. Some of these new threats are already upon us. Populations could be wiped out by lethal “engineered” airborne viruses, for example. Perhaps within a decade, some individuals will acquire the power to trigger events on the scale of the worst present-day terrorist outrages. There would be no need of an organised network of Al-Qaida-type terrorists; just a fanatic or social misfit with the mindset of those who now design computer viruses. There are people with such propensities in every country: very few, to be sure, but bio and cyber-technologies will become so powerful that even one could be too many.
Catastrophes could arise, even more worryingly, simply from technical misadventure. Disastrous accidents are possible even in well-regulated institutions: the unintended creation or release of a noxious fast-spreading pathogen, for instance, or a devastating software error.
So what are we to do in the face of these threats? The surest safeguard against a new danger would be to deny the world the basic science that underpins it. Can the more intractable problems stemming from science be staved off by “going slow” in some areas, or by sacrificing some of science’s traditional freedom of enquiry and international openness?
Some regulation is certainly inevitable. New advances are opening up ever more potential applications (especially in biology: human reproductive cloning, genetically modified organisms and the rest) that will be regulated, for reasons of ethics or prudence, or sometimes because of public revulsion – the so-called “yuck factor”. Almost all scientists accept such constraints on the procedures they use (for example, those imposed on animal experiments) but believe that their choice of research topic should in other respects be unconstrained. However, once the genie is out of the bottle, the eventual outcomes may be impossible to control. Hence the argument that restricting some areas of scientific work might be justified, even if that work is in itself safe and ethical, simply because of unease about where it might lead.
Drawing the line
But is this a sensible argument? Governments provide some funding for “curiosity-driven” research, but offer enhanced support, on strategic grounds, to fields that promise valuable or benign spin-offs. So does the converse follow: should support be withdrawn from a line of “pure” research, even an undeniably interesting one, if there is reason to expect its net outcome will be troublesome?
Such a question confronts us with the obvious difficulty that most scientific discoveries can be applied both for good and for ill. Also, the specific applications of academic research generally cannot be foreseen: the inventors of lasers didn’t foresee eye surgery as an early application of their work; the discovery of X-rays was not motivated by a desire to see through flesh.
Nor can scientists be completely stopped from thinking and speculating: their best ideas often come unbidden, during leisure hours. But any academic scientist whose grant has been stopped is aware that funding cuts can slow down a line of research, even if they could never halt it completely.
Self-regulation might be an option. Academic researchers should obviously ensure that none of their experiments poses unacceptable risks, and scientists do sometimes abide by self-imposed moratoria on specific lines of research. A precedent for this was the declaration put forward in 1975 by prominent molecular biologists, to refrain from some types of experiments rendered possible by what was then the new technique of recombinant DNA. This followed a meeting at Asilomar, California, convened by the biochemist Paul Berg of Stanford University. The Asilomar moratorium soon came to seem unduly cautious, but that doesn’t mean it was unwise at the time, because the level of risk was then genuinely uncertain.
James Watson, co-discoverer of DNA’s double helix, regards this attempt at self-regulation as, in retrospect, a mistake. (Watson admits to being “bullish” about the applications of biotechnology, believing that we should be uninhibited about using the new genetics to “improve” humanity. He asks rhetorically: “If biologists don’t play God, who will?”). But another distinguished Asilomar participant, the virologist David Baltimore, believes that it was right “to engage society in thinking about the problems, because we know that society could block us from realising the tremendous benefits of this work unless we square with them and lead them in thinking through the problems”. The Asilomar precedent was encouraging: it showed that an international group of leading scientists could agree a self-denying ordinance, and that their influence on the research community was sufficient to ensure that it was implemented.
There are now even more reasons for exercising restraint, but a voluntary consensus would be far harder to achieve today. The academic community is far larger, and competition, enhanced by commercial pressures, is more intense. To put effective brakes on a field of research, whether for reasons of safety, ethics or because of concerns about uncontrollable spin-offs, would require international consensus. If one country alone imposed regulations, the most dynamic researchers and enterprising companies would migrate to another that was more sympathetic or permissive. This is happening already in stem cell research, where some countries, particularly Britain and Denmark, have established relatively permissive guidelines and may thereby attract a “brain gain”. By offering a still more enticing regime to researchers and to their fledgling biotech industry, Singapore and China aim to leapfrog the competition.
As Joy realises, it wouldn’t be easy to achieve a consensus that a specific type of research was so potentially dangerous that we should forgo it; indeed, even a single enlightened individual would find it hard to know where to draw the line. Humans can seldom “agree, as a species” – the phrase Joy uses – even on what seem far more urgent imperatives. So can relinquishment be sufficiently “fine grained” to discriminate between benign and hazardous projects?
żěè¶ĚĘÓƵs working on the atomic bomb project during the Second World War raised concerns about whether a nuclear explosion could ignite all the world’s atmosphere or oceans. Before the 1945 test of the first atomic bomb, Edward Teller and two colleagues addressed this issue in a report that was (much later) published by the Los Alamos National Laboratory; they convinced themselves that there was a large safety factor. We now know for certain that a single nuclear weapon, devastating though it is, cannot trigger a nuclear chain reaction that would utterly destroy the Earth or its atmosphere.
Similar Promethean concerns were raised three years ago when scare stories in the media suggested that experiments being planned for the giant accelerators at the Brookhaven National Laboratory in the US and at CERN in Geneva could conceivably do us all in. When atoms of lead or gold are crashed together with ultra-high energy, the resultant compression converts them into “quark matter” – something that may previously have existed only in the first microsecond of cosmic history. Could this exotic new form of matter destroy everything it encountered, and thereby trigger a catastrophe? Physicists can conceive of such a scenario, but it would require quarks to behave in ways that are thought to be exceedingly unlikely. Indeed, I don’t know anyone who betrayed the slightest anxiety.
Worst case scenario
These experiments nonetheless offer an interesting case study. Compared to most instances when the precautionary principle is invoked – for instance, the escape of new pathogens into the environment – the potential downside is far more unlikely, but far more catastrophic. How should society guard against being unknowingly exposed by scientists to a not-quite-zero chance of an event with an almost infinite downside? There is no specific countervailing benefit to the rest of us, so the level would surely be lower than the experimenters might willingly accept on their own behalf.
Some would argue that odds of 50 million to 1 against disaster would be good enough: that would put it below the chance that an asteroid large enough to cause global devastation will hit the Earth within the next year. But while we may become resigned to a natural risk such as asteroids or natural pollutants that we can’t do much about, that doesn’t mean that we should acquiesce in an extra avoidable risk of the same magnitude. This would be like arguing that the extra carcinogenic effect of artificial radiation is acceptable if it doesn’t do more than double the risk from natural radiation.
This leads to a quandary. Can scientists ever be sure enough of their reasoning to offer reassurance with the confidence level of many billions to 1? This is the level that people might reasonably demand before sanctioning an experiment that puts the entire Earth at risk. We may offer these odds against the Sun not rising tomorrow, or against a fair die giving 100 sixes in a row, but a scientist would be over-presumptuous to place such confidence in any theories about the extreme temperatures and densities generated when atoms are smashed together with unprecedented energy.
Fortunately, there are empirical grounds for feeling reassured that these accelerator experiments can’t induce a catastrophe. Very fast-moving particles in space (cosmic rays) routinely undergo collisions that resemble what happens in the experiments, and haven’t triggered a runaway process. This may suggest a useful way to draw the line: if we are looking to carry out experiments with poorly understood consequences, and we have no more than subjective probability estimates of the possible outcomes, we should perhaps be circumspect about them only if there is a chance they could generate conditions with no precedent in the natural Universe.
Of course, in contrast to the accelerator experiments, novel techniques and discoveries will generally have manifest short-term usefulness, as well as being steps towards Joy’s long-term nightmare. And even if all the world’s scientific academies agreed that some specific lines of enquiry had a disquieting downside, and all countries imposed a formal prohibition in unison, how effectively could it be enforced? An international moratorium could certainly slow down particular lines of research, even if they couldn’t be stopped completely. And when experiments are disallowed for ethical reasons, enforcement with 99 per cent effectiveness, or even just 90 per cent, is far better than having no prohibition at all.
But when experiments are exceedingly risky, enforcement would need to be close to 100 per cent effective to be reassuring. And that’s hard to imagine. Despite all the efforts of law enforcers, millions of people use illicit drugs and thousands peddle them. In view of the failure to control drug smuggling or even homicides, it is unrealistic to expect that, when relevant expertise is widespread, we can ever be fully secure against bioerror and bioterror. Risks would remain that cannot be eliminated except by measures that are themselves unpalatable, such as universal surveillance.
It is also worth pointing out, however, that the same techniques that could lead to voracious “nanobots” might also be needed to create the nanotech analogue of vaccines that could immunise against them. If no one but clandestine groups were pursuing dangerous research, it would certainly be harder to devise countermeasures because nobody else would have the relevant expertise.
So we now face some difficult decisions. In view of our current scientific and technological capabilities, what is the safest and most responsible way to develop them further? Humanity is more at risk than at any earlier phase in its history, and this is a critical time. Our future as a species may depend on the choices we make in the next hundred years.