and have won the 2024 Turing award, which is often called the Nobel prize of computing, for their fundamental work on ideas in machine learning that later proved crucial to the success of artificial intelligence models such as Google DeepMind’s AlphaGo.
Barto, who is now retired and lives in Cape Cod, Massachusetts, didn’t even realise he was nominated for the award. “I joined a Zoom with some people and was told and I was just flabbergasted,” says Barto. “I was totally surprised. I was totally unprepared, delighted at the honour, but I had no idea that this was coming.”
The pair will share the $1 million prize for their work on reinforcement learning, in which an AI is “rewarded” and “punished” through trial and error to achieve a goal. This has been studied since AI’s inception – for example, in 1948, Alan Turing first suggested a “pleasure-pain system” for intelligent machines, reminiscent of modern reinforcement learning systems, but until the 1980s received little attention.
Advertisement
Up to that point, research in machine learning was largely focused on symbolic AI, which involves manually teaching a computer the rules of how to learn. Barto and Sutton, who was then Barto’s student, began exploring algorithms and mathematical theories that could replicate Turing’s idea, using neural networks to let an AI work out these rules by itself, rather than the symbolic approach that had previously dominated.
“When I started, it was very unfashionable. I didn’t care, because it was interesting to me,” says Barto. “Not only was it unfashionable, it was considered a dead end to look at neural networks. It’s really surprising and gratifying that it has gotten to the point where a lot of people are working in the area, improving the algorithms and doing applications, many of which are really very beneficial. I’m amazed and pleased to see this evolution.”
“They started the field [of reinforcement learning],” says Chris Watkins at Royal Holloway, University of London. Some of their first reinforcement learning algorithms, such as policy gradient models, which provide a blueprint for AIs to choose their actions as their environment changes, and temporal difference learning, which compares predictions to how a situation unfolds, are still widely used today, says Watkins. For example, they have powered AI breakthroughs such as Google DeepMind’s AlphaGo and AlphaZero, along with advanced robotic systems such as OpenAI’s early work in solving a Rubik’s cube.
Barto and Sutton’s temporal difference algorithm, which was inspired by theories of how animals learned, also understand the dopamine reward system in the brain. In the 1990s, neuroscientists realised that neurons in monkey brains fired in response to unexpected rewards, and worked exactly like the predictions that were part of Barto and Sutton’s algorithms. “It’s the best example of ideas moving back and forth between engineering and natural science ever,” says Sutton.
Sutton hopes that current artificial intelligence research might take more inspiration from the natural world. “We’re doing the obvious idea that an [AI] should learn from experience, just as animals learn from experience, and this is still neglected,” says Sutton. “Modern AIs don’t learn from experience. They learn from a bunch of separate datasets collected by people… Today, we still don’t have machines that will learn from their experience and form an understanding of the world. This is still the obvious thing that remains overlooked.”
