Artificial intelligence has been used to make avatars that appear capable of translating text into sign language. But one of the researchers behind the demonstration videos says they are being overhyped 鈥� and experts in Deaf culture and sign language describe the AI translations as often incomprehensible.
鈥淲e are very much in the experimental and research stages where AI is being tested,鈥� says at Gallaudet University, a university for deaf and hard of hearing students in Washington DC, who wasn鈥檛 involved in the research. 鈥淚t is absolutely not perfect yet.鈥�
The videos are certainly , but they 鈥渁re not the direct output from our model and are actually a lot of trouble to make 鈥� they are just for demonstration purposes鈥�, says at Rutgers University in New Jersey.
Advertisement
First, Fang and his colleagues created their Prompt2Sign data set by downloading sign language videos from the internet and public video repositories. Their software tool then processed each video to extract a 2D skeletal framework representing a person鈥檚 upper body, arms and fingers as coloured lines. The team used this to create a large data set of sign language vocabulary, consisting of body poses for eight different sign languages tagged with the text of what those poses mean.
The researchers then trained an AI model called SignLLM on that dataset, enabling it to translate, for example, written German into German Sign Language. They found that it performed better than average, according to scoring by a common machine translation evaluation.
To show what this technology might enable in the future, Fang and his colleagues then used a developed by the Chinese tech company Alibaba to manually convert SignLLM鈥檚 sign language poses into nine videos demonstrating short American Sign Language translations. The photorealistic and photogenic avatars in those videos attracted some attention and praise on social media and in tech newsletters.
But Malzkuhn, a third-generation Deaf researcher, described the sign language translations in the videos as varying in quality from semi-comprehensible to 鈥渞eally unintelligible鈥�, despite showing 鈥渁 degree of potential鈥�. She also pointed to the avatars鈥� facial expressions 鈥� another crucial element of sign languages 鈥� as lacking. Translations of this quality would 鈥渇orce the end user to spend a lot of their mental energy just trying to figure out what the avatar is saying鈥�, she says.
There is still promise in the idea of AI-generated sign language translations, says Malzkuhn, whose research lab is developing signing avatars for children鈥檚 stories. Such tools could provide greater accessibility for those who are deaf or hard of hearing. For example, the UK company SignApse has created train station announcements in British Sign Language by using AI to blend together pre-recorded videos of human translators into a versatile signing avatar. And the New Zealand company Kara Technologies is creating signing avatars based on specific pre-recorded phrases and words for use during emergency situations.
鈥淪uccessful companies and research groups are not going to try to bite off more than they can chew at any one time,鈥� says , also at Gallaudet University. 鈥淏idirectional translation using AI is not feasible as a first goal.鈥�
Anyone working on sign language technologies should also involve the Deaf community early on, says Malzkuhn, citing the motto 鈥渘othing about us without us鈥�. None of the SignLLM researchers are proficient in any sign languages, says Fang, but they 鈥渆xpect to introduce professionals to assist with proofreading and improvement in future work鈥�.
鈥淚 really would caution anyone who doesn鈥檛 know sign language to be thoughtful about getting into this work, because they don鈥檛 really know what they鈥檙e looking at. Even if they think they do, they would need to bring in a Deaf native signer of a language that they鈥檙e working on to be able to tell them what is being said and how to notate it,鈥� says Malzkuhn. 鈥淏ut if we could shift their strategy and put them on a better path, then the steps that could be advanced along could still be beneficial.鈥�
arXiv
