Someone who types while on a video call may be giving away more than they realise. A computer model can work out the words that the person is typing just by tracking the movement of their shoulders and arms in the video stream.
鈥淭here are significant, oftentimes not very easily discernible shoulder movements that occur when typing,鈥� says Murtuza Jadliwala at the University of Texas at San Antonio. 鈥淲e thought if we are able to model them scientifically, we should be able to infer different keystrokes by looking at the video data.鈥�
Jadliwala and his colleagues developed a model to do just that. They mapped the movements onto a keyboard and cross-referenced the results against a dictionary of commonly typed words, finding they could identify the correct word being typed 75 per cent of the time. Their experiments were conducted both in lab conditions and using real-life video call data.
Advertisement
First, the computer model removes the background information from a frame of a聽video call. It then detects the outer edges of the shoulder by analysing each frame using an image-processing technique called optical flow, which traces how pixels change in a video and maps arm movements onto a keyboard.
The model has varying levels of success depending on a user鈥檚 typing skills. Touch typers are more difficult to discern than those who 鈥減eck鈥� at their keyboard, for whom the model could recover 83 per cent of words correctly. Those who wore some sort of sleeve were also less susceptible to being analysed accurately.
The video-calling software used also affects results: 3.4 per cent more words were recovered on Skype calls than on Zoom, which the researchers say may be due to the way each app compresses video.
The results are alarming, says Alan Woodward at the University of Surrey, UK. 鈥淵ou don鈥檛 need that many characters to fill in gaps in words,鈥� he says.
Blurring your background, skipping frames in the video and pixellating your shoulders and arms are ways to mitigate this issue, says Jadliwala, though Woodward reckons that defeats the purpose. 鈥淭he whole point of a video call is to see people,鈥� he says.
Reference:
