A robot from Google has achieved a level of wide-ranging capability that hasn’t been seen before. It can converse with you like a chatbot, answer questions about pictures and even get the right snacks for you from a drawer.
The robot uses a version of a language model called PaLM, which Google researchers first created last year. This is similar to the GPT-3 model that powers ChatGPT, but has more parameters – the number of variables that can be tweaked to adjust the model’s output.
The original model was also designed to be a generalist. PaLM has since been adapted for specialised tasks, as Med-PaLM, for instance, which answers medical questions, or PaLM-SayCan, which attached PaLM to a robot to improve its ability to carry out tasks.
Advertisement
Now, at Google and his colleagues have developed SayCan further to create PaLM-E, a robot equipped with a version of PaLM that has been trained on images as well as text from the internet. This allows it to better comprehend and perform tasks when asked to do so, partly because it can recognise objects more easily.
“It can, for example, figure out in a new kitchen whether a particular unfamiliar object is a snack or not, because of what it’s seen in other sources of data from the internet that have nothing to do with robots,” says Levine.
This means you can ask the robot – via instructions typed into an interface – to get you a bag of rice snacks.
The key to PaLM-E, says Levine, is that the system can store representations of both words and images in the same abstract computational space. “The main innovation here is to take the foundation provided by language models and augment it with the ability to read in not just text but also image observations,” he says.
Once the language and image part of the model has received a command or answered a question, PaLM-E can use this to make plans for the robot. It maps the information to various actions it can make in the physical world, such as moving from place to place or grasping things with its arm.
The precise physical actions the robot can carry out aren’t hugely different from what other robots can do. However, its ability to plan, reason and generalise using its language and image capacity is unique, says Levine, and means it can outperform these other robots in different, real-world environments.
The robot can also converse, and respond to queries about images, in the question-and-answer style that ChatGPT uses, for which it has a similar performance on many benchmarking tasks.
“It’s a very nice integration of a large language model with embodiment aspects – with perception, with scene description and robotic components,” says at the University of Manchester, UK.
However, if the robot is ever to carry out as many tasks as humans manage, it might require a more complex system, says Cangelosi, because of how the different parts of the human brain interact and work with each other. “There is an assumption that it’s all language based, that our mind is a pure linguistic engine, which is not the way it is,” he says.
For instance, when you hear verbs like “kick” or “lick”, because they involve action, , whereas in PaLM-E, it is a one-way relationship from language to the robot moving. Until the model has calculated its answer using language, the motor system is inactive, he says.
arXiv
