A PENNY for ’em? Knowing what someone is thinking is crucial for understanding their behaviour. It’s the same with artificial intelligences. A new technique for taking snapshots of neural networks as they crunch through a problem will help us fathom how they work, leading to AIs that work better – and are more trustworthy.
In the last few years, deep-learning algorithms built on neural networks – multiple layers of interconnected artificial neurons – have driven breakthroughs in many areas of artificial intelligence, including natural language processing, image recognition, medical diagnoses and beating a professional human player at the game Go.
The trouble is that we don’t always know how they do it. A deep-learning system is a black box, says Nir Ben Zrihem at the Israel Institute of Technology in Haifa. “If it works, great. If it doesn’t, you’re screwed.”
Advertisement
Neural networks are more than the sum of their parts. They are built from many very simple components – the artificial neurons. “You can’t point to a specific area in the network and say all of the intelligence resides there,” says Zrihem. But the complexity of the connections means that it can be impossible to retrace the steps a deep-learning algorithm took to reach a given result. In such cases, the machine acts as an oracle and its results are taken on trust.
To address this, Zrihem and his colleagues created images of deep learning in action. The technique, they say, is like an fMRI for computers, capturing an algorithm’s activity as it works through a problem. The images allow the researchers to track different stages of the neural network’s progress, including dead ends.
“A deep-learning neural network is a black box. If it works, great. If it doesn’t, you’re screwed“
To get the images, the team set a neural network the task of playing three classic Atari 2600 games: Breakout, Seaquest and Pac-Man. They collected 120,000 snapshots of the deep-learning algorithm as it played each of the games. They then mapped the data using a technique that allowed them to compare the same moment in repeated attempts at a game.
The results look a lot like scans of real brains (pictured below: Seaquest on the left, and Pac-Man). But in this case, each dot is a “game state”, a snapshot of a single game at a moment in time. Different colours show how well the AI was doing at that point in the game.
With Breakout, for example – where the player must knock a hole through a wall of brightly coloured blocks with a paddle and a ball – the team was able to identify a clear banana-shaped region in one map showing every time the algorithm tried tunnelling through the blocks to force the ball to the top of the wall, a winning tactic that the neural network had figured out by itself. Mapping the playthroughs let the team trace how the algorithm successfully applied it in successive games.
Seaquest, where the player has to avoid, collect or destroy various items and pick up underwater divers, is harder for AIs to tackle. Using the maps, the team unravelled numerous failed approaches, like waiting too long to rescue errant divers. The details could be useful when retraining the algorithm, says Zrihem.
Building the perfect game strategy is fun, but scans like these could help us hone algorithms designed to solve real problems, says at the University of Wyoming in Laramie. Clune’s own studies of the inner workings of image-recognition algorithms have led him to create “illusions” that can trick a neural network into thinking something is there when it’s not.
For example, a security algorithm might have a flaw that means it’s easily fooled in certain situations, or an algorithm designed to decide if someone gets a bank loan might be prejudiced against people of a particular race or gender.
“If you’re deploying this technology in the real world, you want to understand how it works and where it might fail,” says Clune. “If we can understand neural networks better, then we can understand their weaknesses and improve their strengths.”
This article appeared in print under the headline “The mind of a machine”
Article amended on 24 February 2016
Correction: Since this article was first published, the spelling of Jeff Clune’s name has been corrected.
