Telling artificial intelligence models to “think” step by step when carrying out a task can improve their performance so much that they can outperform humans at jobs AIs usually struggle with.
Using the phrase “let’s think step by step” to cajole AIs into taking more logical decisions was first suggested in presented at a computational neuroscience conference. Such “chain-of-thought” prompting encourages these models, which include GPT-3, a text-generating AI developed by OpenAI, to logically consider tasks in the same way humans do.
Now, at Stanford University in California and his colleagues at Google Research have put chain-of-thought prompting to the test, asking AI models to tackle 23 tasks that AI historically hasn’t been better than humans at.
Advertisement
The tasks included sorting a list of words into alphabetical order and imagining where you would end up after following a number of navigational steps.
Three different AI models – Codex, InstructGPT and PaLM 540B – were asked to complete the tasks without any prompting on how to think, and also after being instructed to think step by step.
Prompting the models improved their performance by between about 25 and 32 per cent, on average, across all the tasks.
The results were also compared with how well humans performed each task in experimental conditions. Without the chain-of-thought prompt, the AI models were only better than humans in between four and six of the 23 tasks, depending on which model was used. With the prompt, the AIs were better than humans in between 10 and 17 of the tasks.
Prompting made the AI models perform worse than when unprompted in three tasks. One of these measured sarcasm, while the other two required prior knowledge, such as common presuppositions about how the world works.
The research shows that using chain-of-thought prompting works in several different data sets, domains and settings, says at the University of California, Santa Barbara.
“Moving forward, prompt engineering will still be critical for enabling GPT-3 for different tasks,” says Wang. He believes that chain-of-thought prompting improves AI performance because it mimics the way that the models are trained, by being presented with data sequentially.
“It’s easier to elicit knowledge from large language models this way,” he says. He compares it to how people interact. “For humans, sometimes I also need students to prompt me in a particular order, so it’s easier for me to give them what they need.”
Reference:
