Cutting-edge artificial intelligence models such as OpenAI鈥檚 GPT-4V and Google鈥檚 Gemini struggle to solve clever wordplay puzzles involving both images and text.
Rebus puzzles typically require a puzzler to identify a word represented by an image, to add or subtract letters from that word and to combine the result with words identified from other images to arrive at a solution.
For instance, a rebus might include a picture of the planet Venus followed by the letters 鈥榰s鈥� with a line running through them 鈥� instructing the puzzler to retain only 鈥淰en鈥�. If this is followed by a picture of an ice cube, the puzzler should be able to combine 鈥淰en鈥� and 鈥渋ce鈥� to solve the puzzle: Venice.
Advertisement
鈥淲e all saw during the the demonstration of some rebus puzzle-solving ability,鈥� says at the ML Alignment & Theory Scholars research programme in California. 鈥淎nd then we thought that this is actually potentially a good benchmark, because it evaluates not only its vision capabilities but also its language abilities.鈥�
Panickssery and his colleagues developed such a benchmark test by hand-crafting 333 rebus puzzles covering categories such as films, music composers, major cities, food and Christmas songs. Such puzzles are intended to test how well multimodal AIs 鈥� models capable of interpreting different types of information 鈥� can perform skills such as image recognition and multi-step reasoning.
The researchers tested several leading AI models developed by commercial companies, as well as some open-source AIs. OpenAI鈥檚 GPT-4V performed the best, with a 24 per cent solve rate, with Google鈥檚 Gemini Pro coming in second at 13 per cent. Open-source AIs such as performed even worse, with accuracy scores below 2 per cent.
Each of the AI models ran into certain challenges. For instance, Gemini Pro had trouble with phonetic rebus puzzles that required it to understand the sound a word makes when it is spoken.
鈥淎 person who looks through some of the puzzles may see that they are very challenging,鈥� says at New York University, who was not involved in the study. 鈥淚t鈥檚 good to see that the current models are actually getting some of them and are thinking through and recognising them correctly.鈥�
Tong sees the rebus puzzle benchmark as 鈥渧ery interesting鈥� and one of many independent benchmarks currently being proposed to test different AI capabilities.
But even if AI puzzle-solving capabilities improve in the future, Panickssery cautions against seeing that as an indicator of AI having achieved general visual and language intelligence. Instead, he describes the puzzles as 鈥渁 good test for very basic skills鈥�.
arXiv
