Post by LWPD on Sept 22, 2013 15:38:22 GMT -5
Poker: a game of both chance and skill. Yet how much does success come down to the use of effective psychology? Can the artificial intelligence of the pokerbot machine consistently best even the greatest human bluffers and sharks in the world?
Courtesy of NY Times
The Steely, Headless King of Texas Hold ’Em
By Mike Kaplan
Stroll among the games at the Cosmopolitan, the newest casino on the Las Vegas Strip, and you might be overwhelmed by the latest whooping and flashing gambling machines. All the high-resolution monitors and video effects, devoted to themes ranging from deep-sea-fishing expeditions to Spider-Man to the unsubtlest visions of cash washing over lucky winners, are only the most obvious signs of technology’s move onto the casino floor. Behind the scenes, server-based gaming now enables managers to rapidly alter payouts, raise or reduce betting minimums, even change games themselves. (In just minutes, a bank of slot machines styled for dance clubbers can be rethemed to appeal to church ladies on a Sunday afternoon.) But a few deceptively prim-looking machines represent an even greater technological leap, the biggest advance in automated gambling since Charles Fey introduced the one-armed bandit in 1895. They owe the way they play to artificial intelligence.
The machines, called Texas Hold ‘Em Heads Up Poker, play the limit version of the popular game so well that they can be counted on to beat poker-playing customers of most any skill level. Gamblers might win a given hand out of sheer luck, but over an extended period, as the impact of luck evens out, they must overcome carefully trained neural nets that self-learned to play aggressively and unpredictably with the expertise of a skilled professional. Later this month, a new souped-up version of the game, endorsed by Phil Hellmuth, who has won more World Series of Poker tournaments than anyone, will have its debut at the Global Gaming Expo in Las Vegas. The machines will then be rolled out into casinos around the world.
They will be placed alongside the pure numbers-crunchers, indifferent to the gambler. But poker is a game of skill and intuition, of bluffs and traps. The familiar adage is that in poker, you play the player, not the cards. This machine does that, responding to opponents’ moves and pursuing optimal strategies. But to compete at the highest levels and beat the best human players, the approach must be impeccable. Gregg Giuffria, whose company, G2 Game Design, developed Texas Hold ‘Em Heads Up Poker, was testing a prototype of the program in his Las Vegas office when he thought he detected a flaw. When he played passively until a hand’s very last card was dealt and then suddenly made a bet, the program folded rather than match his bet and risk losing more money. “I called in all my employees and told them that there’s a problem,” he says. The software seemed to play in an easily exploitable pattern. “Then I played 200 more hands, and he never did anything like that again. That was the point when we nicknamed him Little Bastard.”
The pokerbot, which takes on one challenger at a time, can trace its roots to the Norwegian Defense Research Establishment in Kjeller, Norway. Until 2002, that’s where an engineer named Fredrik Dahl worked on artificial intelligence for secret government projects on combat simulations. The job involved using neural networks. Functioning much like an extremely focused, one-dimensional version of the human brain, these complex computer algorithms develop strategies that emerge through so many repetitive mathematical calculations that few humans could reproduce, much less endure them. Dahl’s work on two-sided, zero-sum games, where there is no mutual interest, proved to be useful in developing strategies to win not only wars but also poker games.
He started with backgammon, though. While a student at the University of Oslo, where he concentrated on computer science, he developed a penchant for the game. He once made it to the finals of the Norwegian National Backgammon Championships. “One thing I learned from backgammon is how to handle losses, no matter how well I play,” he says. “It is not a good game for sore losers.”
Rather than be sore, he used computers to improve his play. Dahl created a neural net that predicted the probability of winning a backgammon match from each position on the board, at every possible stage in a game. Any individual situation is easy enough to solve, Dahl says; the challenge was determining all possible situations, giving value to the importance of each one and choosing a play. “The program needed to self-train and discover these strategies itself,” Dahl says.
Dahl spent a year working part time on his program and started selling it by mail order in 1997 for $250 a floppy disk. He figures that he sold thousands of copies, and that its impact was even broader thanks to the pirating of his software. Among those whose play it improved was a Texas-based actuary named Malcolm Davis, whom Dahl describes as a top backgammon player in the world at the time. Davis and Dahl ended up talking over backgammon strategies.
In 2000, Dahl began work on a similar program for poker. He was inspired by the challenge of creating a program that could come up with solutions for a game characterized by incomplete information. Unlike backgammon, in which an opponent’s position is visible, a poker player does not know what another player’s cards are and so cannot follow pure strategies. Instead, he needs to consider a range of hands that his opponent might have and estimate the best response to the various possibilities. The uncertainty, combined with an opponent’s ability to bluff, makes it difficult to write software that plays poker effectively.
Dahl gave his neural nets rudimentary game-playing instructions and programmed them to probe for weaknesses as they played one another. “They computed probability, based on the other player’s actions,” Dahl says. “I had the neural nets play training games and experiment with various approaches.” Eventually, through trial and error, he says, “they learned to be successful.”
Dahl recalls staring at his computer screen, watching his neural nets compete, when he saw one of them make a fairly sophisticated bluff known as floating. You do this by playing passively, initiating no bets and matching the ones that your opponent makes. If, after the turn card is played, your opponent does not bet, you do. His slowing down usually means that he had been overplaying his cards with the hope that you would fold or his hand would improve. Your bet here signals that you’ve just made a strong hand or that you have been inducing him to put as much money as possible in the pot because you have had a superior hand from the start. “At first, I wasn’t even familiar with that strategy,” Dahl says. “Later, I thought it was amazing that the neural net could come up with a known, successful strategy on its own.”
Even as Dahl recognized the improvement and appreciated that his software played better on its five-billionth hand than on its two-billionth, he dismissed any commercial application for it as too difficult. “I started doing the poker project for fun,” he says. “Then, in 2003, Malcolm talked me into doing it for real.”
Doing it for real meant creating a machine that could be put in casinos. As originally designed by Dahl, the brain of his game adjusted to the opposition. If, for example, an opponent folded a lot, it played aggressively; if it faced aggressive play, it tried to trap. Casino commissions, however, mandate that a gaming machine cannot change its playing style in response to particular opponents. A poker game must play a World Series of Poker champion the same way it does a neophyte, so Dahl’s machine would not be able to learn from its experience in a casino.
“The neural net’s learning needs to be frozen” is how Dahl puts it. Yet it would also have to play well enough so that few humans could consistently beat it. For Dahl, now 46 and a natural pessimist, this was a predicament he did not think he could overcome. “I thought that if it is vulnerable to even one person’s strategy, that is a huge problem — then other people learn how to do it, and the machine collapses,” Dahl says.
Dahl eventually retooled his neural net so that it would teach itself to play a perfectly defensive game. Rather than steer it to study its opponent and try to capitalize on weaknesses, the net was directed to make itself as hard to beat as possible. “Ordinarily, you figure out weaknesses in your opponent and find ways to exploit those weaknesses,” Dahl says. “But because our program needs to be stable, it can’t do that. So instead it does everything it can to prevent itself from being exploited. The theory behind it is almost paranoid.”
Dahl’s game grew more unpredictable over time, as the neural nets learned. Eventually, it got to the point where, over thousands of hands, they would each orchestrate the optimal number of bluffs; but in any one hand, the program might do anything. What’s more, a second neural net, which plays in a slightly different style, was introduced to reinforce the machine’s unpredictability; when an opponent has a reduced stack of chips, a third net takes over and plays in a manner customized for that situation. Like three tag-team fighters, the nets alternate against an opponent. At random moments, the machine’s mode of play might change the level of its aggressiveness.
By 2006, after thousands of neural nets, tweaked repeatedly, had played billions of hands, Dahl recruited gifted poker-playing friends to take on his game. It won frequently enough to hold up in a casino environment, he thought. Malcolm Davis then brought it to the attention of Bob Hamman, a frequent backgammon opponent of his and the bridge partner of Bill Gates.
Though Hamman ranks among the world’s top bridge players, he plays cards only as a hobby. He makes his living by insuring promotional contests (like those at company outings where attendees might win $10,000 for sinking a 30-foot putt, for example). Intrigued by the software’s potential, Hamman tested it against both other poker-playing programs and a young bridge master named Justin Lall. “Justin also happens to be a skilled poker player,” Hamman says. “He’s skilled enough that if you think you want to make a living playing against Justin, you might want to reconsider. He said it’s a good game. He found it captivating. He came close to beating it.”
Soon after, in October 2006, Hamman called Gregg Giuffria, a neighbor of Gates’s at the Del Mar Country Club in Southern California, where both had homes. Giuffria was once better known as a member of the hard-rock band Angel, but now he ran a company that made gambling machines. After a bit of small talk, Hamman told him about Dahl’s software. “It’s real smart,” he said. “I thought it was only interesting. But then you play against it and realize that it’s bluffing you. All of a sudden, you’re talking to steel and glass like it’s human.”
By the time Giuffria heard from Hamman, he had already wandered far from arena stages. He took the first step in 1990, 15 years after Angel’s first album came out, when he had a life-changing dinner with Lee Iacocca, the former Chrysler chairman. (Giuffria’s wife, April, knew Iacocca as a family friend.) “I thought I was not the dumbest guy at the table,” Giuffria says, “but the dumbest guy on the planet.” He suddenly saw himself as a 39-year-old “white boy chasing rock ‘n’ roll, with hip-hop coming in — it was time for me to reinvent myself.”
Days later, he cut his hair and, on the advice of Iaccoca, began analyzing patents that the Defense Department was allowing to be released to the public sector. The hope was that Giuffria would discover an unexploited business opportunity and maybe Iaccoca would partner with him to develop it. Giuffria came across a company called Summit Systems that held a patent for a mathematical process that had an application for slot machines. Iaccoca passed, so Giuffria used royalties from his music career to acquire rights to the patent from the moribund company and eventually helped sell them to International Game Technology, now the world’s largest manufacturer of slot machines. “In one afternoon,” Giuffria says, “I made more money from that patent than I had in 18 years of touring, writing songs and getting gold records with Angel.”
That success hooked Giuffria on the gambling industry. He and Iacocca collaborated on casino developments around the country. Later, Giuffria built a Hard Rock casino near New Orleans and got into creating gambling machines. “It’s all entertainment,” Giuffria says, when describing the transition from rocker to gaming entrepreneur.
Giuffria liked the sound of what Hamman had to offer. What’s more, Giuffria had not forgotten the valuable lesson he learned in 1998 when he passed up a chance to invest in Triple Play Draw Poker, a video-poker machine that turned out to be one of the most successful games of its type. Had Giuffria made that earlier investment, he’d be a billionaire today.
Not long after Hamman first called, he and Malcolm Davis flew from Dallas to meet Giuffria in Las Vegas. “They showed me a gray screen on a computer,” he says. “It dealt cards, you played for units, it tallied who won and lost. It was in a very stark form, but, like Bob said, as soon as it started bluffing against me, I realized that this was the most incredible thing I had ever seen. I wouldn’t let it walk out the door.”
Giuffria decided to develop the card-playing software into an actual casino machine, knowing that it would cost more than $5 million to devise a prototype with a case, monitor, graphics and sound effects.
For starters, he and his tech guys needed to test each neural net. But because these had been developed through self-training and not created by humans, there was no source code — the computer instructions written out by programmers — to analyze. You couldn’t track the logic behind the system’s actions. “We had to take a black-box approach,” says Bob Honeycutt, Giuffria’s lead engineer on the project. They had to look at the results without being able to know how they were produced. Honeycutt customized math-based programs that look for probabilities to play poker against Fredrik Dahl’s neural nets. Honeycutt’s software lost. The neural nets showed no patterns or anomalies.
Next, local poker pros dropped by to test it out. Mike (the Mouth) Matusow, famous for his loquaciousness, “came over and treated the machine like it was one of his buddies,” Giuffria says. “He’d say, ‘What, are you drawing down on me with a pair of 4s?’ Some pros came in here, sure that they can beat the machine, and then left angry when they couldn’t. It really upset people.”
The visceral reactions pleased Giuffria, who saw value in the personification of the machine. “A lot of people who play this are interested in live poker, but they are too intimidated to jump in there,” says Anthony Lucas, professor of casino management at the University of Nevada in Las Vegas. “[This] gives them a chance to play without running the risk of being judged or embarrassed for making a bad move. This is private. Nobody criticizes your strategy.” In addition, he says, it’s perfect for a generation that likes to lose itself in Angry Birds: “If you prefer not to interact or socialize with people, you can play this game the same way you would watch TV or go online.”
Convinced that he had a market — and confident that players wouldn’t worry about the legitimacy of the “cards” dealt by the machine any more than they do about the house’s dice being loaded — Giuffria moved ahead. His team created digitized cards, chips and green felt and added realistic sound effects.
Before Giuffria showed his machine to I.G.T., which had the right of first refusal to whatever games he created, he sent it to the New Jersey-based Gaming Laboratories International, a kind of Underwriters Laboratories for casino gambling. “We asked them to punch holes in it,” Giuffria says. It usually takes about a month to test a game, but G.L.I. kept Dahl’s neural nets for about nine months.
Giuffria asked how humans would do against it. “Oh,” he recalls the lab telling him, “they’ll get killed.”
Because a never-beatable game will not succeed in a casino, the machine was programmed to occasionally play in a weak, passive style, seeming to reduce the game’s edge and re-engaging casual players. The result is that this game “gets accused of having leaks,” Giuffria says; posters gloat online about its weaknesses. Inevitably, he adds, the take-away is: " ‘Of course I will beat it.’ They don’t know that it might be one of the hands that falls into a gray area where the machine takes a dive deliberately.”
Giuffria seems to take the knocks personally. Nevertheless, he does not correct the mistaken impressions. “No way did I want to put out there that not only do we have a machine they can’t beat, but we’ve spent two years trying to dummy it down so that it doesn’t beat humans all the time. That would have made enemies. We kept our mouths shut.”
Natasha Dow Schull, an associate professor at M.I.T. and the author of “Addiction by Design: Machine Gambling in Las Vegas,” describes the experience of playing Texas Hold ‘Em Heads Up Poker as feeling as if you’re taking on “a really good medieval automaton” with “a little man behind the curtain.” Giuffria seems to prefer not to break this humanizing spell, so that players who experience the occasional wins will believe that the machine is beatable. “Gamblers continually overestimate their abilities,” says Mike Dixon, a psychology professor at the University of Waterloo in Ontario with a specialization in gambling. “It’s like hitting a half-court shot occasionally and thinking it makes you into Carmelo Anthony.”
The game has developed a small but fervent fan base since its slow-drip release began two years ago, with about 200 machines so far located in Las Vegas, Mississippi and California (all states where live versions of poker thrive). When it appeared on the casino floors of Bellagio and Aria, Las Vegas casinos where some of America’s highest-stakes poker games take place, Texas Hold ‘Em Heads Up Poker captured the imaginations of big-money players.
A very few poker professionals, particularly those who specialize in one-on-one limit Texas Hold ‘Em, actually have a shot at winning regularly. Michael Reed, a pro gambler from Pittsburgh, credibly claims to fall into that category. So far, over the course of 500 hours, he says, he averages about $135 per hour in profits, playing at the $20- or $40-bet level. This means that by making the maximum bets throughout a hand, he can win or lose $500 in seconds.
He has not seen many others turning a profit. “Over all, this machine crushes people,” he says. “The machine is far too aggressive and steals far too many pots.”
How does Reed manage to overcome a machine that has been so hard to beat? He says it “gains its edge by being the aggressor. It almost never check-calls, or simply matches an opponent’s bet without a raise. The bot gives credit to your hand when you raise and reraise.” Unseasoned players, Reed says, have a habit of folding hands that might seem inferior. Reed has discovered that playing connecting cards like 7 and 8 can have unexpected value. “If a high card comes on the flop, the machine often folds to a bet from you, believing that you have made a high pair. So you have the middle range [of flopped cards] from which you can make hands, and the high range from which you can bluff. If you don’t bet, the bot will want to.” It’s been estimated that Reed is among 100 or so people in the world who can steadily beat the machine.
I relayed this to Bill McBeath, Aria’s former president and a high-stakes poker player in his own right, and he laughed. McBeath said that when somebody tells him that he can beat the machine, his reply is: " ‘Come back and see me in a year, pal.’ The game does not have a statistical advantage over players, but the imperfections of human capabilities mean that it inevitably wins.” I started to explain Reed’s experiences, and McBeath cut me off. “Look, there was a 24-year-old who had beaten it for a while. Now he’s broke. And I think this machine had something to do with his demise.”
Only 200 Texas Hold ‘Em Heads Up Poker machines are now in circulation. Triple Play Draw Poker can have 100 machines in a single casino. As part of the effort to get closer to this level, the new Hellmuth machine and another new one created with Johnny Chan (who ranks No. 2 in World Series wins) have more than just the endorsements of the poker pros. Their styles and personalities have also been added to the machines. The Pro Series will incorporate neural nets that seem to play according to Hellmuth’s tight but forceful approach or the aggressive strategy that inspired Chan’s nickname, The Orient Express.
Brian Perego, a movie-industry veteran, recorded Chan and Hellmuth repeating their favorite lines — “I dodge bullets, baby!” is one of Hellmuth’s — and expressing various reactions that will be played on the machines’ video monitors. “When a hand starts to get serious, Phil might put on his sunglasses,” Perego says. “It will be fun, but there will be psych-out moves, just like at a real poker table.”
The technology behind Dahl’s game has the potential to do a lot more than simply taking money from casino customers. Dahl could see it being adapted to make credibility assessments, like deciding who should get a loan, for example, by analyzing applicants in comparison with databases of borrowers who repaid their loans and those who did not.
Hector Levesque, a professor of computer science at the University of Toronto who specializes in A.I., also sees the bigger picture for this sort of technology. “Pushing hard on statistics and learning [via neural nets] can have a big impact,” he says, while cautioning that that should not be mistaken for having the ability to think in the multidimensional, contextual sense. “The biggest technology coming out of A.I. is getting examples of reasonable behavior and learning from statistics. Google search does so well because it has to do with statistics and adjusts from that. Same with the driverless car. Airline pricing is basically a game, played against people who want to buy tickets. The airline comes up with a price that is not high enough to discourage customers but still maximizes profits. Poker and games are not so far from things that have real economic bite if you can train the neural nets properly.”
For now, however, Dahl’s interest is in whether Texas Hold ‘Em Heads Up Poker can withstand the onslaught that will come with wider exposure. Back in Vegas one Sunday afternoon, I caught up with Johnny Chan on the gaming floor of MGM Grand Hotel and Casino, where he was about to play in a baccarat tournament. “I get e-mails all the time, with people challenging me and wanting to play me heads-up,” Chan said. “They’ll get a chance to do it with this machine.”
I asked him if he thought anyone could win big over the long haul, beyond what Mike Reed has managed to grind out. “Nothing is impossible,” Chan said. Then he added that some people see Texas Hold ‘Em Heads Up Poker as a potential gold mine. “You probably have five groups out there right now, testing the machine, writing down all the results, getting every play blow by blow and figuring out what to do on every hand. One hundred percent, I’m sure there are teams working on it.”
The pokerbot just put its sunglasses on.
Courtesy of NY Times
The Steely, Headless King of Texas Hold ’Em
By Mike Kaplan
Stroll among the games at the Cosmopolitan, the newest casino on the Las Vegas Strip, and you might be overwhelmed by the latest whooping and flashing gambling machines. All the high-resolution monitors and video effects, devoted to themes ranging from deep-sea-fishing expeditions to Spider-Man to the unsubtlest visions of cash washing over lucky winners, are only the most obvious signs of technology’s move onto the casino floor. Behind the scenes, server-based gaming now enables managers to rapidly alter payouts, raise or reduce betting minimums, even change games themselves. (In just minutes, a bank of slot machines styled for dance clubbers can be rethemed to appeal to church ladies on a Sunday afternoon.) But a few deceptively prim-looking machines represent an even greater technological leap, the biggest advance in automated gambling since Charles Fey introduced the one-armed bandit in 1895. They owe the way they play to artificial intelligence.
The machines, called Texas Hold ‘Em Heads Up Poker, play the limit version of the popular game so well that they can be counted on to beat poker-playing customers of most any skill level. Gamblers might win a given hand out of sheer luck, but over an extended period, as the impact of luck evens out, they must overcome carefully trained neural nets that self-learned to play aggressively and unpredictably with the expertise of a skilled professional. Later this month, a new souped-up version of the game, endorsed by Phil Hellmuth, who has won more World Series of Poker tournaments than anyone, will have its debut at the Global Gaming Expo in Las Vegas. The machines will then be rolled out into casinos around the world.
They will be placed alongside the pure numbers-crunchers, indifferent to the gambler. But poker is a game of skill and intuition, of bluffs and traps. The familiar adage is that in poker, you play the player, not the cards. This machine does that, responding to opponents’ moves and pursuing optimal strategies. But to compete at the highest levels and beat the best human players, the approach must be impeccable. Gregg Giuffria, whose company, G2 Game Design, developed Texas Hold ‘Em Heads Up Poker, was testing a prototype of the program in his Las Vegas office when he thought he detected a flaw. When he played passively until a hand’s very last card was dealt and then suddenly made a bet, the program folded rather than match his bet and risk losing more money. “I called in all my employees and told them that there’s a problem,” he says. The software seemed to play in an easily exploitable pattern. “Then I played 200 more hands, and he never did anything like that again. That was the point when we nicknamed him Little Bastard.”
The pokerbot, which takes on one challenger at a time, can trace its roots to the Norwegian Defense Research Establishment in Kjeller, Norway. Until 2002, that’s where an engineer named Fredrik Dahl worked on artificial intelligence for secret government projects on combat simulations. The job involved using neural networks. Functioning much like an extremely focused, one-dimensional version of the human brain, these complex computer algorithms develop strategies that emerge through so many repetitive mathematical calculations that few humans could reproduce, much less endure them. Dahl’s work on two-sided, zero-sum games, where there is no mutual interest, proved to be useful in developing strategies to win not only wars but also poker games.
He started with backgammon, though. While a student at the University of Oslo, where he concentrated on computer science, he developed a penchant for the game. He once made it to the finals of the Norwegian National Backgammon Championships. “One thing I learned from backgammon is how to handle losses, no matter how well I play,” he says. “It is not a good game for sore losers.”
Rather than be sore, he used computers to improve his play. Dahl created a neural net that predicted the probability of winning a backgammon match from each position on the board, at every possible stage in a game. Any individual situation is easy enough to solve, Dahl says; the challenge was determining all possible situations, giving value to the importance of each one and choosing a play. “The program needed to self-train and discover these strategies itself,” Dahl says.
Dahl spent a year working part time on his program and started selling it by mail order in 1997 for $250 a floppy disk. He figures that he sold thousands of copies, and that its impact was even broader thanks to the pirating of his software. Among those whose play it improved was a Texas-based actuary named Malcolm Davis, whom Dahl describes as a top backgammon player in the world at the time. Davis and Dahl ended up talking over backgammon strategies.
In 2000, Dahl began work on a similar program for poker. He was inspired by the challenge of creating a program that could come up with solutions for a game characterized by incomplete information. Unlike backgammon, in which an opponent’s position is visible, a poker player does not know what another player’s cards are and so cannot follow pure strategies. Instead, he needs to consider a range of hands that his opponent might have and estimate the best response to the various possibilities. The uncertainty, combined with an opponent’s ability to bluff, makes it difficult to write software that plays poker effectively.
Dahl gave his neural nets rudimentary game-playing instructions and programmed them to probe for weaknesses as they played one another. “They computed probability, based on the other player’s actions,” Dahl says. “I had the neural nets play training games and experiment with various approaches.” Eventually, through trial and error, he says, “they learned to be successful.”
Dahl recalls staring at his computer screen, watching his neural nets compete, when he saw one of them make a fairly sophisticated bluff known as floating. You do this by playing passively, initiating no bets and matching the ones that your opponent makes. If, after the turn card is played, your opponent does not bet, you do. His slowing down usually means that he had been overplaying his cards with the hope that you would fold or his hand would improve. Your bet here signals that you’ve just made a strong hand or that you have been inducing him to put as much money as possible in the pot because you have had a superior hand from the start. “At first, I wasn’t even familiar with that strategy,” Dahl says. “Later, I thought it was amazing that the neural net could come up with a known, successful strategy on its own.”
Even as Dahl recognized the improvement and appreciated that his software played better on its five-billionth hand than on its two-billionth, he dismissed any commercial application for it as too difficult. “I started doing the poker project for fun,” he says. “Then, in 2003, Malcolm talked me into doing it for real.”
Doing it for real meant creating a machine that could be put in casinos. As originally designed by Dahl, the brain of his game adjusted to the opposition. If, for example, an opponent folded a lot, it played aggressively; if it faced aggressive play, it tried to trap. Casino commissions, however, mandate that a gaming machine cannot change its playing style in response to particular opponents. A poker game must play a World Series of Poker champion the same way it does a neophyte, so Dahl’s machine would not be able to learn from its experience in a casino.
“The neural net’s learning needs to be frozen” is how Dahl puts it. Yet it would also have to play well enough so that few humans could consistently beat it. For Dahl, now 46 and a natural pessimist, this was a predicament he did not think he could overcome. “I thought that if it is vulnerable to even one person’s strategy, that is a huge problem — then other people learn how to do it, and the machine collapses,” Dahl says.
Dahl eventually retooled his neural net so that it would teach itself to play a perfectly defensive game. Rather than steer it to study its opponent and try to capitalize on weaknesses, the net was directed to make itself as hard to beat as possible. “Ordinarily, you figure out weaknesses in your opponent and find ways to exploit those weaknesses,” Dahl says. “But because our program needs to be stable, it can’t do that. So instead it does everything it can to prevent itself from being exploited. The theory behind it is almost paranoid.”
Dahl’s game grew more unpredictable over time, as the neural nets learned. Eventually, it got to the point where, over thousands of hands, they would each orchestrate the optimal number of bluffs; but in any one hand, the program might do anything. What’s more, a second neural net, which plays in a slightly different style, was introduced to reinforce the machine’s unpredictability; when an opponent has a reduced stack of chips, a third net takes over and plays in a manner customized for that situation. Like three tag-team fighters, the nets alternate against an opponent. At random moments, the machine’s mode of play might change the level of its aggressiveness.
By 2006, after thousands of neural nets, tweaked repeatedly, had played billions of hands, Dahl recruited gifted poker-playing friends to take on his game. It won frequently enough to hold up in a casino environment, he thought. Malcolm Davis then brought it to the attention of Bob Hamman, a frequent backgammon opponent of his and the bridge partner of Bill Gates.
Though Hamman ranks among the world’s top bridge players, he plays cards only as a hobby. He makes his living by insuring promotional contests (like those at company outings where attendees might win $10,000 for sinking a 30-foot putt, for example). Intrigued by the software’s potential, Hamman tested it against both other poker-playing programs and a young bridge master named Justin Lall. “Justin also happens to be a skilled poker player,” Hamman says. “He’s skilled enough that if you think you want to make a living playing against Justin, you might want to reconsider. He said it’s a good game. He found it captivating. He came close to beating it.”
Soon after, in October 2006, Hamman called Gregg Giuffria, a neighbor of Gates’s at the Del Mar Country Club in Southern California, where both had homes. Giuffria was once better known as a member of the hard-rock band Angel, but now he ran a company that made gambling machines. After a bit of small talk, Hamman told him about Dahl’s software. “It’s real smart,” he said. “I thought it was only interesting. But then you play against it and realize that it’s bluffing you. All of a sudden, you’re talking to steel and glass like it’s human.”
By the time Giuffria heard from Hamman, he had already wandered far from arena stages. He took the first step in 1990, 15 years after Angel’s first album came out, when he had a life-changing dinner with Lee Iacocca, the former Chrysler chairman. (Giuffria’s wife, April, knew Iacocca as a family friend.) “I thought I was not the dumbest guy at the table,” Giuffria says, “but the dumbest guy on the planet.” He suddenly saw himself as a 39-year-old “white boy chasing rock ‘n’ roll, with hip-hop coming in — it was time for me to reinvent myself.”
Days later, he cut his hair and, on the advice of Iaccoca, began analyzing patents that the Defense Department was allowing to be released to the public sector. The hope was that Giuffria would discover an unexploited business opportunity and maybe Iaccoca would partner with him to develop it. Giuffria came across a company called Summit Systems that held a patent for a mathematical process that had an application for slot machines. Iaccoca passed, so Giuffria used royalties from his music career to acquire rights to the patent from the moribund company and eventually helped sell them to International Game Technology, now the world’s largest manufacturer of slot machines. “In one afternoon,” Giuffria says, “I made more money from that patent than I had in 18 years of touring, writing songs and getting gold records with Angel.”
That success hooked Giuffria on the gambling industry. He and Iacocca collaborated on casino developments around the country. Later, Giuffria built a Hard Rock casino near New Orleans and got into creating gambling machines. “It’s all entertainment,” Giuffria says, when describing the transition from rocker to gaming entrepreneur.
Giuffria liked the sound of what Hamman had to offer. What’s more, Giuffria had not forgotten the valuable lesson he learned in 1998 when he passed up a chance to invest in Triple Play Draw Poker, a video-poker machine that turned out to be one of the most successful games of its type. Had Giuffria made that earlier investment, he’d be a billionaire today.
Not long after Hamman first called, he and Malcolm Davis flew from Dallas to meet Giuffria in Las Vegas. “They showed me a gray screen on a computer,” he says. “It dealt cards, you played for units, it tallied who won and lost. It was in a very stark form, but, like Bob said, as soon as it started bluffing against me, I realized that this was the most incredible thing I had ever seen. I wouldn’t let it walk out the door.”
Giuffria decided to develop the card-playing software into an actual casino machine, knowing that it would cost more than $5 million to devise a prototype with a case, monitor, graphics and sound effects.
For starters, he and his tech guys needed to test each neural net. But because these had been developed through self-training and not created by humans, there was no source code — the computer instructions written out by programmers — to analyze. You couldn’t track the logic behind the system’s actions. “We had to take a black-box approach,” says Bob Honeycutt, Giuffria’s lead engineer on the project. They had to look at the results without being able to know how they were produced. Honeycutt customized math-based programs that look for probabilities to play poker against Fredrik Dahl’s neural nets. Honeycutt’s software lost. The neural nets showed no patterns or anomalies.
Next, local poker pros dropped by to test it out. Mike (the Mouth) Matusow, famous for his loquaciousness, “came over and treated the machine like it was one of his buddies,” Giuffria says. “He’d say, ‘What, are you drawing down on me with a pair of 4s?’ Some pros came in here, sure that they can beat the machine, and then left angry when they couldn’t. It really upset people.”
The visceral reactions pleased Giuffria, who saw value in the personification of the machine. “A lot of people who play this are interested in live poker, but they are too intimidated to jump in there,” says Anthony Lucas, professor of casino management at the University of Nevada in Las Vegas. “[This] gives them a chance to play without running the risk of being judged or embarrassed for making a bad move. This is private. Nobody criticizes your strategy.” In addition, he says, it’s perfect for a generation that likes to lose itself in Angry Birds: “If you prefer not to interact or socialize with people, you can play this game the same way you would watch TV or go online.”
Convinced that he had a market — and confident that players wouldn’t worry about the legitimacy of the “cards” dealt by the machine any more than they do about the house’s dice being loaded — Giuffria moved ahead. His team created digitized cards, chips and green felt and added realistic sound effects.
Before Giuffria showed his machine to I.G.T., which had the right of first refusal to whatever games he created, he sent it to the New Jersey-based Gaming Laboratories International, a kind of Underwriters Laboratories for casino gambling. “We asked them to punch holes in it,” Giuffria says. It usually takes about a month to test a game, but G.L.I. kept Dahl’s neural nets for about nine months.
Giuffria asked how humans would do against it. “Oh,” he recalls the lab telling him, “they’ll get killed.”
Because a never-beatable game will not succeed in a casino, the machine was programmed to occasionally play in a weak, passive style, seeming to reduce the game’s edge and re-engaging casual players. The result is that this game “gets accused of having leaks,” Giuffria says; posters gloat online about its weaknesses. Inevitably, he adds, the take-away is: " ‘Of course I will beat it.’ They don’t know that it might be one of the hands that falls into a gray area where the machine takes a dive deliberately.”
Giuffria seems to take the knocks personally. Nevertheless, he does not correct the mistaken impressions. “No way did I want to put out there that not only do we have a machine they can’t beat, but we’ve spent two years trying to dummy it down so that it doesn’t beat humans all the time. That would have made enemies. We kept our mouths shut.”
Natasha Dow Schull, an associate professor at M.I.T. and the author of “Addiction by Design: Machine Gambling in Las Vegas,” describes the experience of playing Texas Hold ‘Em Heads Up Poker as feeling as if you’re taking on “a really good medieval automaton” with “a little man behind the curtain.” Giuffria seems to prefer not to break this humanizing spell, so that players who experience the occasional wins will believe that the machine is beatable. “Gamblers continually overestimate their abilities,” says Mike Dixon, a psychology professor at the University of Waterloo in Ontario with a specialization in gambling. “It’s like hitting a half-court shot occasionally and thinking it makes you into Carmelo Anthony.”
The game has developed a small but fervent fan base since its slow-drip release began two years ago, with about 200 machines so far located in Las Vegas, Mississippi and California (all states where live versions of poker thrive). When it appeared on the casino floors of Bellagio and Aria, Las Vegas casinos where some of America’s highest-stakes poker games take place, Texas Hold ‘Em Heads Up Poker captured the imaginations of big-money players.
A very few poker professionals, particularly those who specialize in one-on-one limit Texas Hold ‘Em, actually have a shot at winning regularly. Michael Reed, a pro gambler from Pittsburgh, credibly claims to fall into that category. So far, over the course of 500 hours, he says, he averages about $135 per hour in profits, playing at the $20- or $40-bet level. This means that by making the maximum bets throughout a hand, he can win or lose $500 in seconds.
He has not seen many others turning a profit. “Over all, this machine crushes people,” he says. “The machine is far too aggressive and steals far too many pots.”
How does Reed manage to overcome a machine that has been so hard to beat? He says it “gains its edge by being the aggressor. It almost never check-calls, or simply matches an opponent’s bet without a raise. The bot gives credit to your hand when you raise and reraise.” Unseasoned players, Reed says, have a habit of folding hands that might seem inferior. Reed has discovered that playing connecting cards like 7 and 8 can have unexpected value. “If a high card comes on the flop, the machine often folds to a bet from you, believing that you have made a high pair. So you have the middle range [of flopped cards] from which you can make hands, and the high range from which you can bluff. If you don’t bet, the bot will want to.” It’s been estimated that Reed is among 100 or so people in the world who can steadily beat the machine.
I relayed this to Bill McBeath, Aria’s former president and a high-stakes poker player in his own right, and he laughed. McBeath said that when somebody tells him that he can beat the machine, his reply is: " ‘Come back and see me in a year, pal.’ The game does not have a statistical advantage over players, but the imperfections of human capabilities mean that it inevitably wins.” I started to explain Reed’s experiences, and McBeath cut me off. “Look, there was a 24-year-old who had beaten it for a while. Now he’s broke. And I think this machine had something to do with his demise.”
Only 200 Texas Hold ‘Em Heads Up Poker machines are now in circulation. Triple Play Draw Poker can have 100 machines in a single casino. As part of the effort to get closer to this level, the new Hellmuth machine and another new one created with Johnny Chan (who ranks No. 2 in World Series wins) have more than just the endorsements of the poker pros. Their styles and personalities have also been added to the machines. The Pro Series will incorporate neural nets that seem to play according to Hellmuth’s tight but forceful approach or the aggressive strategy that inspired Chan’s nickname, The Orient Express.
Brian Perego, a movie-industry veteran, recorded Chan and Hellmuth repeating their favorite lines — “I dodge bullets, baby!” is one of Hellmuth’s — and expressing various reactions that will be played on the machines’ video monitors. “When a hand starts to get serious, Phil might put on his sunglasses,” Perego says. “It will be fun, but there will be psych-out moves, just like at a real poker table.”
The technology behind Dahl’s game has the potential to do a lot more than simply taking money from casino customers. Dahl could see it being adapted to make credibility assessments, like deciding who should get a loan, for example, by analyzing applicants in comparison with databases of borrowers who repaid their loans and those who did not.
Hector Levesque, a professor of computer science at the University of Toronto who specializes in A.I., also sees the bigger picture for this sort of technology. “Pushing hard on statistics and learning [via neural nets] can have a big impact,” he says, while cautioning that that should not be mistaken for having the ability to think in the multidimensional, contextual sense. “The biggest technology coming out of A.I. is getting examples of reasonable behavior and learning from statistics. Google search does so well because it has to do with statistics and adjusts from that. Same with the driverless car. Airline pricing is basically a game, played against people who want to buy tickets. The airline comes up with a price that is not high enough to discourage customers but still maximizes profits. Poker and games are not so far from things that have real economic bite if you can train the neural nets properly.”
For now, however, Dahl’s interest is in whether Texas Hold ‘Em Heads Up Poker can withstand the onslaught that will come with wider exposure. Back in Vegas one Sunday afternoon, I caught up with Johnny Chan on the gaming floor of MGM Grand Hotel and Casino, where he was about to play in a baccarat tournament. “I get e-mails all the time, with people challenging me and wanting to play me heads-up,” Chan said. “They’ll get a chance to do it with this machine.”
I asked him if he thought anyone could win big over the long haul, beyond what Mike Reed has managed to grind out. “Nothing is impossible,” Chan said. Then he added that some people see Texas Hold ‘Em Heads Up Poker as a potential gold mine. “You probably have five groups out there right now, testing the machine, writing down all the results, getting every play blow by blow and figuring out what to do on every hand. One hundred percent, I’m sure there are teams working on it.”
The pokerbot just put its sunglasses on.
