MIDWAY through a poker competition at the Rivers casino in Pittsburgh, Pennsylvania, one player seemed to lose the plot. Opponents watched baffled as Claudico risked large amounts of money with weak cards, or raised the stakes aggressively to win a handful of chips – and then suddenly appeared passive, dithering over decisions and avoiding big wagers.
Yet despite never having set foot in a casino before, Claudico had played more poker games than all of the other players put together. Claudico was a bot. Created by researchers at the nearby Carnegie Mellon University, it had learned poker by playing billions of hands against itself. But for all Claudico’s experience, this was the first time a computer had taken on human professionals at “no limit” Texas hold’em poker, so who knew what might happen? In the end, the humans won the contest – by a whisker. Brains, it would appear, will not have the edge for much longer.
The contest, staged in April 2015, was the latest instalment in almost a century of scientific research into games like poker and Go. It isn’t just about pushing the boundaries of mathematics and artificial intelligence. Poker is a game of hidden information – you can’t see your opponents’ cards and they can’t see yours – and so mimics the uncertainty of many real-life situations, from negotiations and bidding at auctions to share trading and cybersecurity. To triumph in poker, as in life, we must tweak our tactics based on what we know and what our opponents choose to do. Poker bots like Claudico hold lessons for us all about how to cope with risk and make better decisions.
Advertisement
More than a game: As AI becomes more pervasive, can machines take our jobs without ruining our lives?
The first studies of poker strategy took place in the 1920s, when mathematician John von Neumann looked at a very simple two-player version of the game. He realised the players were involved in a tug-of-war, each trying to limit the opponent’s winnings while simultaneously trying to increase their own profit. Von Neumann wondered whether there was a mathematically optimal way to do this. In other words, is there a strategy that will guarantee the best possible result, assuming your opponent is also hunting for the best strategy? Yes there is, his calculations revealed. He called it the “equilibrium” strategy, because if both players adopt it, neither will be able to gain by changing their style of play.
Von Neumann’s discovery made it possible to calculate what the outcome would be if two highly skilled competitors faced each other. Analysing his highly simplified poker, he found that if the player with the opportunity to start the betting had a good hand then they should bet, and if they had an average hand they should not, but choose instead to “check” and wait it out. Intuitively this makes sense. However, he also found that if the player has a poor hand, the optimal strategy requires them to bet too. Successful poker players have long deceived opponents using such bluffing tactics, but von Neumann’s proof showed this approach is no quirk of human psychology. Bluffing is a mathematical necessity.
In recent years, von Neumann’s ideas about optimum strategies have been key to the success of poker-playing computers. When coders created the first online poker bots around 20 years ago, they often gave them specific tactics to follow. As a result, a bot’s ability was limited by its creator’s skill level: a weak player can’t teach a computer a strong strategy. These days, the best bots are taught just the rules of the game, and it is up to them to learn how to win. By playing countless games against themselves, they try to work out the equilibrium strategy. Coders may have little idea of why their bot “thinks” the way it does, so its choices can be a surprise.
Dealing with regret
Claudico’s creators at Carnegie Mellon have a proven track record when it comes to producing talented poker bots. In 2014, their fought off several other bots to win the Annual Computer Poker Competition, a worldwide contest held since 2006. Tartanian7 gradually evolved into Claudico, a name meaning “I limp” in Latin. A poker player is said to “limp” if they bet the minimum for their opening move, rather than raising the stakes or declaring themselves out of the game by folding. Limping is generally seen as a weak strategy: good human players don’t limp, notes Sam Ganzfried, who designed Claudico with colleagues Tuomas Sandholm and Noam Brown. Yet Claudico chooses to limp about 10 per cent of the time. It goes to show that artificial intelligence has the potential to prove popular wisdom wrong.
By searching for the equilibrium strategy, bots like Claudico are refining how they bluff and bet, and getting harder to beat every year. But it is difficult to reach perfection, given the complexity of poker. Even in a two-player version with limited bet sizes – known as “heads-up limit poker” – there are 3.16 × 1017, or 316 million billion, potential situations that could come up in a game. Although this is fewer than in draughts (checkers), the fact that information is hidden in poker has made it hard to identify an optimal strategy.
A major breakthrough arrived when researchers at the University of Alberta in Edmonton, Canada, devised a technique called “counterfactual regret minimisation”. “Regret” here refers to the difference between the expected pay-off of the action taken by , and the potential pay-off if it had acted differently. The technique involves Cepheus tweaking its strategy over the course of billions of hands, lowering its overall regret until it is as small as possible.
This may seem like a rather unambitious goal, given that we tend to think of human behaviour as being positive and forward-looking. Much of economic theory, for example, assumes that people try to maximise potential future gains. But do we? In 2008, economists of the University of Southern Denmark in Odense and at Ca’ Foscari University of Venice, Italy, compared human economic decisions with those made by several computer-learning algorithms and found that . Even if our considerations of regret are subconscious, they could help us make better decisions.
Regret minimisation certainly has benefits for bots. According to Neil Burch, one of the Alberta researchers, its power lies in helping them learn without making many assumptions about the game in question. “It gives them that flexibility that lets them work in a lot of different situations,” he says. In fact, regret minimisation has proved so successful that last year, Burch and his colleagues proclaimed: ““. For the two-player version of Texas hold’em poker with limited stakes, Cepheus had identified a strategy that was so near to the optimal equilibrium that it was in essence unbeatable. Regardless of who the bot faced, it would not lose money in the long run.
Regret-driven learning also leads to some surprising tactics. For example, Cepheus rarely raises the stakes to the limit, even when it is has the best possible hand. It also plays a broader range of hands than a human might, choosing occasionally to play weak cards rather than fold. And Claudico, which also employs regret minimisation for no-limit poker, uses a much wider range of stakes than a human might, from tiny bets to huge raises, notes Sandholm. “Betting $19,000 to win a $700 pot just isn’t something that a person would do,” said one of Claudico’s human opponents at the Brains vs AI contest.
Why don’t humans use as broad a range of tactics as Cepheus and Claudico, given that they are apparently so successful? One explanation is that we have to make simplifications when dealing with the complexity of a game like poker. Rather than considering all possible moves, we tend to mentally bunch similar situations together. We do the same in daily life: we might round numbers up or down to the nearest ten or hundred, or use stereotypes to categorise people. This process of abstraction makes the world easier to handle, but means we can lose to opponents that are using a better approximation of the world than we are.
Abstraction is not our only weakness. Our behaviour often follows patterns, which makes it easier for an opponent to predict what we will do. We also get impatient and frustrated. Indeed, it’s thought that poker websites can distinguish humans from bots because people tend to switch games more often, moving up to the higher-stakes tables when they get bored or overconfident. Bots can tactically exploit human flaws like these, but to do so they need to move away from equilibrium strategies. Cepheus and Claudico will not lose in the long run, no matter how good the opponent, but that makes them inherently defensive. In contrast, bots that can cash in on human weaknesses display other behaviours – ones we might emulate to gain the edge in a negotiation. For example, people struggle when put under pressure, and the Alberta researchers have found that if bots mimic aggression they can force people into errors. “The more actions that you make someone choose, the more chances they have to do something wrong,” says Burch.
“Bluffing is not a quirk of human psychology – it is a mathematical necessity“
Such tactics are increasingly unlikely to be successful against the best poker players, however. They are already learning from their computer counterparts. As they develop their knowledge of game theory and practise against flawless bots, the top poker players are adopting equilibrium-like strategies, which tend to be less aggressive.
Beyond the card tables, when it comes to decisions or negotiations involving hidden information, your opponent must weigh up their options based on what they can observe. As von Neumann showed, the optimal strategy means making this choice as difficult as possible. If your opponent can spot patterns in your behaviour, it gives them valuable information about the cards you are holding, literally or metaphorically. Random choices can help counteract this. But this is something humans are very poor at, in games ranging from rock-paper-scissors to poker. Sometimes the best way to act when making a decision is literally to flip a coin.
The success of bots that try to minimise regret can also teach us something beyond the realm of poker. We may be doing this subconsciously already when negotiating or making decisions, but we might do better if we made it an explicit strategy rather than focusing on maximising future rewards.
Another insight from poker bots is the importance of abstraction. We need to know more about how we approximate the world if we are to make better choices. Our need to simplify complex information means we tend to focus on a narrow range of options, whereas bot research suggests it could be worth employing a much wider range of actions when taking risks.
In future, we may come to view bargaining as more of a science than an art. On a more philosophical level, bots challenge our notions of ourselves. We tend to think deception is an inherently human behaviour, yet poker bots have taught themselves to bluff because it is the optimal strategy. Starting with a blank slate, bots have come up with strategies that challenge dogmas about what ought to succeed. Bots like Claudico and Cepheus often seem to behave strangely, but they don’t know how we feel about them, or care if we think they are contradicting accepted wisdom. They just want to come out on top.
Poker lessons
In 2012, PhD student Will Ma set up a poker course at the Massachusetts Institute of Technology. The classes earned students academic credits, and more than 200 of them attended the packed-out sessions. Here are some of the lessons they learned, which apply to many decisions beyond poker.
• Know the difference between tactics and strategy. Tactics are short-term actions that bring a benefit; strategy is broader and about putting yourself in a better overall position, even if there is nothing to be gained immediately. The best players in poker and life are good at both.
• You can make a good decision and get a bad result, or a bad decision and get a good outcome. In situations that involve risk, it’s important to know whether you got lucky, or whether your strategy is one that brings consistent winnings.
• Don’t be afraid to go all in. Sometimes the optimal tactic will be to raise the stakes substantially. If the situation is in your favour, bet accordingly.
• Work out how many routes to victory you have, and how many your opponent has. It’s not your cards alone that matter; it’s how many ways you could convert them into a winning position.
• Exploit your image. When there is hidden information, your opponents will be looking for ways to find out more about you. From the name you use to how much you talk and how long you take to make a move, the image you put across is an important part of a strategy.
This article appeared in print under the headline “Play your cards right”
