IT IS no exaggeration to say that civilisation has entrusted its continued existence to the smooth functioning of technological systems. This trust runs so deep as to be nearly invisible.
We’re now in the early days of a turning point in our dependence upon machines. Last century, we trusted them to do things for us; this century, we’re starting to trust them to decide things for us.
Advertisement
That’s not all bad. Humans have a notoriously patchy record when it comes to decision-making. But relying on technological systems to make decisions for us – especially when risks are involved and our safety is at stake – can have major consequences.
Developments in promise to give some of our technologies extraordinary decision-making skills. With self-driving cars, for example, the experiences of each individual vehicle can be combined, adding together every mile driven, every hazard identified and every accident avoided or not, such as a self-driving car’s recent self-inflicted run-in with a bus.
Within a year of self-driving autos being widely available, such an aggregated artificial intelligence could have orders of magnitude more experience than any human driver. These patterns of improved performance are likely in other systems that have many inputs along with a clear distinction between what’s correct and what’s not, such as in medical diagnostics or financial analysis.
But along with successes will come failures, where machine decisions lead to disaster.
Some will come from faulty programming – no software is perfect, and even the best can have bugs. Others will come from situations or conditions outside of programmed (or learned) recognition, or when multiple correctly functioning systems interact in an undesirable way.
Our dilemma is the result of being too willing to trust technologies to make the right choices – and being unable to recognise an incorrect choice until it’s too late.
It’s here that faith in decision-making technologies can be the most problematic. We may trust too much – one experiment found that people blindly followed a robot into danger in a mock emergency.
We may believe that a machine “knows” more than we do, or can access information we can’t. We’ll need to bring a healthy scepticism into interactions with them.
Civilisation’s adoption of decision-making machines will have its own learning curve so that we realise when trust is appropriate. And we’ll definitely have to figure out how to identify and adapt to situations where our machines are in over their metaphorical heads… and hit the brakes for them.
This article appeared in print under the headline “In machines we trust”
