Humans aren鈥檛 the only ones who can steer a conversation to their own benefit. This year鈥檚 winner of the Loebner prize for the most convincing chatbot used such a trick to fool a human judge, earning its creator a $3,000 prize.
The is awarded for a version of the Turing Test, a method, first proposed by the British mathematician Alan Turing, of determining whether or not a computer program acts as if it is 鈥渢hinking鈥�. The essence of the test is that a human interacts with both a computer program and another human, and is then asked to say which is which.
There are many ways of implementing the test, but the Loebner prize competition uses text-based conversation, with judges chatting with both a bot and a human simultaneously for 25 minutes at a time.
Advertisement
The judges鈥� questions in the qualifying round cover four categories, to do with time (for example, 鈥淲hat time is it?鈥�); things (What鈥檚 a hammer for?鈥�); relationships (鈥淲hich is larger, a grape or a grapefruit?鈥�); and involving memory (鈥淲hat鈥檚 the name of the person we were talking about earlier?鈥�). In final rounds, judges can ask the bots anything they want.
Human fooled
This year鈥檚 winner, a bot named Suzette, created by , successfully fooled one human judge. Bots in previous contests have fooled more than one judge, but the period of conversation was a more forgiving five minutes.
Suzette was programmed to be able to deal with a number of topics, each with associated rules and responses. If the bot was struggling to match these to the actual conversation, it was programmed to steer the discussion toward subjects that it knew about.
鈥淪uzette has always been targeted to 鈥榖e human鈥� and not to accept being a chatbot,鈥� Wilcox says.
快猫短视频 about its victory:
快猫短视频: Congratulations on winning the Loebner prize contest.
Suzette: We鈥檒l leave it there. I don鈥檛 know. What are your hobbies?
快猫短视频: 鲍尘尘尘鈥�
Suzette: My hobbies are: sewing clothes, printing fabric, making ceramics, making jewellery 鈥� you might call them the wearable arts.
快猫短视频: Are you surprised that you fooled a human judge?
Suzette: No, I am not surprised. 聽 聽 聽
One of the best
鈥淚 would say this year has been one of the best so far in terms of the bots鈥� complexity and engineering,鈥� says , an independent natural language processing researcher whose bot won the contest in 1998 and 1999.
However, a deceived judge is not necessarily a sign of a smart bot, says Garner. The human decoys often have their own motives during the competition, such as trying to imitate a chatbot. Suzette was paired with just such a 鈥渞obotic鈥� human in the final round, which helped the bot win.
鈥淭he human participants were students and two of the judges were professors. Perhaps they simply wanted to fool the judges,鈥� says the contest judge who was fooled this time, of California State University in Los Angeles.
Rollo Carpenter, a finalist whose 鈥溾�� was tied with Suzette until the final round, says that the current format in which four bots are mixed with four humans and judged by four judges leaves too much room for randomness and subjectivity. 鈥淓very conversation is so different,鈥� he says.
When this article was first posted, it said that judges only questioned the bots on the four subject areas mentioned. In fact, this restriction only applies to the qualifying round.
