Image-generating artificial intelligence models like DALL-E and Midjourney often have difficulty creating human hands, with many otherwise photorealistic pictures given away by hands with the wrong number of fingers or in impossible poses 鈥� so now researchers have created an AI dedicated to just drawing hands.
Most text-to-image AIs in use today are based on a technology called diffusion, which has become adept at creating photorealistic images, but they struggle to reproduce the shape of the human hand because images in their training data have hands in wildly different poses and fingers can be hidden behind objects or other fingers.
Zhiyang Guo at the University of Science and Technology of China in Hefei and his colleagues have created an AI model based on a different technology, called neural radiance fields (NeRF). This is a way to model 3D shapes using a neural network and it has previously been used by driverless car company and Google Research to create large, seamless 3D models of a city.
Advertisement
The researchers trained their HandNeRF model on an open-source data set of hand images. They could then use it to create realistic images of entirely new hand poses from any desired angle.
, co-founder of AI start-up Hugging Face, says that while NeRF models seem capable of producing realistic hands, this technology is notoriously difficult to train and the model probably can鈥檛 create entire images by itself, unlike diffusion.
鈥淏oth are usable for creating images, but they鈥檙e very different techniques with different pros and cons. So a naive combination of this NeRF technique with diffusion models wouldn鈥檛 be possible to do in a plug-and-play way,鈥� he says.
at the University of Lugano, Switzerland, says that because diffusion models have no concrete concept of the shape of objects, but only their appearance from training set images, they are easily confused. But NeRFs work by developing a framework of a 3D object, so are better at understanding how they can be seen from various viewpoints.
Combining the two could be helpful, he says, as the output of a diffusion model that includes hands could be compared with a NeRF representation of a hand to ensure that it is logical, possible and includes the correct number of digits.
鈥淎 hand showing three raised fingers asking for three coffees, or a hand pointing one finger to the air, would still be described as 鈥榟uman hand鈥� and thus it鈥檚 difficult for the model to discern the number of total fingers an actual human hand has,鈥� says Condor Lacambra. 鈥淣eRFs will help us model reality faster, more efficiently and with a higher degree of quality, but are not traditionally used to create completely new content.鈥�
arXiv
