A robot can learn how to correctly wash dishes or pack a bag of trail mix by listening to human voice commands. That could pave the way for future workplace or household robots capable of learning in the moment as they interact with people.
It is 鈥渁lmost impossible鈥� to train home-helper robots that can perfectly manage chores in every situation, says at Stanford University in California. 鈥淪o we do want to have this method of naturally interacting with a robot and giving feedback to the robot on the fly.鈥�
Shi and her colleagues developed two artificial intelligence modules. One plans out robotic actions using language instructions, and the other translates those actions into motor control movements. The researchers first trained the robot by having human operators directly control two robotic arms to perform tasks while narrating each step out loud. The recorded audio was then transcribed and synchronised with the robot鈥檚 movements to create labelled training data for the robot to learn from.
Advertisement
The researchers also kept some data where the human operators made mistakes but corrected themselves. 鈥淚nevitably the robot will also make mistakes,鈥� says at Stanford University. 鈥淎nd if it makes a mistake, this data shows it how to correct the mistake.鈥�
Then they challenged the robot to perform tasks such as placing items in a plastic Ziploc bag, using a metal scoop to pour specific trail mix ingredients into a bag and using a sponge to scrub stuck gummy bear candies off a plate. A microphone allowed human observers to further instruct the robot as it worked: they first issued a trigger command such as 鈥渟top鈥� or 鈥減ardon鈥� before providing details such as 鈥淚 want some M&Ms鈥� or 鈥渃lean the right side鈥�.
Those verbal corrections were used afterwards to fine-tune and improve the robot鈥檚 planning capabilities. Such fine-tuning boosted the robot鈥檚 success rate by about 20 per cent on average when it attempted the three tasks again by itself.
This custom-trained AI approach also outperformed OpenAI鈥檚 鈥� a generic AI model capable of processing both visual and language information 鈥� in planning out actions for the robot. GPT-4V did reasonably well in its reasoning steps but made errors in understanding the spatial relationships between the robot and surrounding objects.
Still, the researchers say that it might be possible to take a similar vision-language model and fine-tune it with additional training to enable robots to understand even more language commands.
arXiv
