This article is being made free to view thanks to sponsorship from P&G

Artificial intelligence is getting its teeth into lip reading. A project by Google鈥檚 DeepMind and the University of Oxford applied deep learning to a huge data set of BBC programmes to create a lip-reading system that leaves professionals in the dust.

The AI system was trained using some 5000 hours from six different TV programmes, including 快猫短视频night, BBC Breakfast and Question Time. In total, the videos contained 118,000 sentences.

First the University of Oxford and DeepMind researchers trained the AI on shows that aired between January 2010 and December 2015. Then they tested its performance on programmes broadcast between March and September 2016.

By only looking at each speaker鈥檚 lips, the system accurately deciphered entire phrases, with examples including 鈥淲e know there will be hundreds of journalists here as well鈥� and 鈥淎ccording to the latest figures from the Office of National Statistics鈥�.

Here is a聽clip from the database without subtitles:

And here鈥檚 the same clip with subtitles provided by the AI system:

AI shows the way

The AI vastly outperformed a professional lip-reader who attempted to decipher 200 randomly selected clips from the data set.

The professional annotated just 12.4 per cent of words without any error. But the AI annotated 46.8 per cent of all words in the March to September data set without any error. And many of its mistakes were small slips, like missing an 鈥榮鈥� at the end of a word. With these results, the system also outperforms all other automatic lip-reading systems.

鈥淚t鈥檚 a big step for developing fully automatic lip-reading systems,鈥� says at the University of Oulu in Finland. 鈥淲ithout that huge data set, it鈥檚 very difficult for us to verify new technologies like deep learning.鈥�

Two weeks ago, a similar deep learning system called 鈥� also developed at the University of Oxford 鈥� outperformed humans on a lip-reading data set known as GRID. But where GRID only contains a vocabulary of 51 unique words, the BBC data set contains nearly 17,500 unique words, making it a much bigger challenge.

In addition, the grammar in the BBC data set comes from a wide diversity of real human speech, whereas the grammar in GRID鈥檚 33,000 sentences follows the same pattern and so is far easier to predict.

The DeepMind and Oxford group says it will release its BBC data set as a training resource. , who is working on LipNet, says he is looking forward to using it.

Lining up the lips

To make the BBC data set suitable for automatic lip reading in the study, video clips had to be prepared using machine learning. The problem was that the audio and video streams were sometimes out of sync by almost a second, which would have made it impossible for the AI to learn associations between the words said and the way the speaker moved their lips.

But by assuming that most of the video was correctly synced to its audio, a computer system was taught the correct links between sounds and mouth shapes. Using this information, the system figured out how much the feeds were out of sync when they didn鈥檛 match up, and realigned them. It then automatically processed all 5000 hours of the video and audio ready for the lip-reading challenge 鈥� a task that would have been onerous by hand.

The question now is how to use AI鈥檚 new lip-reading capabilities. We probably don鈥檛 need to fear computer systems eavesdropping on our conversations by reading our lips because long-range microphones are better for spying in most situations.

Instead, Zhou thinks lip-reading AIs are most likely to be used in consumer devices to help them figure out what we are trying to say.

鈥淲e believe that machine lip readers have enormous practical potential, with applications in improved hearing aids, silent dictation in public spaces (Siri will never have to hear your voice again) and speech recognition in noisy environments,鈥� says Assael.

Going higher

BEEPING monitors. Stick-on electrodes. The finger clip that takes your pulse. Modern medicine comes with a cacophony of sensors. Oxehealth, a company spun out from the University of Oxford, is aiming to replace them all with just one 鈥� a camera.

uses camera data to measure heart rate, respiration and blood oxygenation from a distance. The company is now trying out the technology in the real world, in hospitals, psychiatric wards and police stations.

One place the tech is being tested is at the , Oxford鈥檚 main teaching hospital. Patients who will need a spell in intensive care after their surgery are being asked to participate in the trial: Oxehealth鈥檚 cameras will monitor their vital signs in parallel with conventional sensors.

鈥淣ormal monitoring involves sticking things on patients,鈥� says Peter Watkinson, an intensive care doctor for Oxford University Hospital Trust. 鈥淚t makes it less easy for them to get out of bed, have a shower or go to the loo. Every time you roll over in bed you pull the thing off.鈥�

The Oxehealth software watches for the tiny changes in video frames as a patient鈥檚 chest rises and falls when they breathe, for example. It also tracks subtle changes in the pinkness of their skin, using that to infer their pulse.

Watkinson thinks camera monitoring will make it possible to catch patients whose condition is deteriorating before their symptoms are obvious. At the moment, clinicians have to decide which patients they need to monitor 鈥� usually those who are recovering from a procedure or are already very ill. But the cameras would be able to keep tabs on everyone in their vicinity.

鈥淚t would be much more reliable at picking up patients at risk of deterioration throughout the hospital,鈥� says Watkinson.

Oxehealth鈥檚 Oliver Gibson says the company has also been running studies with London鈥檚 Metropolitan Police Service and Broadmoor high-security psychiatric hospital. Here the firm is 鈥渢rying to solve the problem of making sure someone is safe when they鈥檙e in a secure room鈥�.

鈥淲ith camera monitoring you can catch patients who are deteriorating before symptoms are obvious鈥�

In April, Broadmoor installed cameras in one of its rooms. Staff would sit there during the day, while patients would sleep in it at night. Both were wired up with sensors to validate the readings the cameras took.

Broadmoor鈥檚 patients need regular monitoring, says the hospital鈥檚 director Robert Bates 鈥� especially at night. 鈥淲e usually check on patients every 15 minutes,鈥� he says. It can be hard to tell whether someone is breathing just by looking through their door, so staff have to approach patients to make sure they鈥檙e OK. 鈥淚t鈥檚 upsetting for the patients because they鈥檙e being regularly disturbed,鈥� says Bates.

The camera system could offer a less intrusive means of keeping tabs on residents. 鈥淚t will also help us monitor our patients鈥� physical health after administration of medication,鈥� says Bates. 鈥淎nti-psychotics can have an impact on patients鈥� physical health.鈥�

Gibson says the Broadmoor study confirmed the cameras鈥� accuracy. 鈥淲e were getting a breathing rate accuracy where 94 per cent of measures were within two breaths per minute of a contact device,鈥� he says. 鈥淎nd 94 per cent of heart rate measures were within three beats per minute.鈥�

Oxehealth is not alone in hoping to free patients from their wires. at RWTH Aachen University in Germany is setting up a wireless monitoring ward in the university hospital. Six beds in the geriatrics department will have cameras monitoring their occupants鈥� vitals.

Leonhardt says the system could one day be installed in elderly people鈥檚 homes.鈥漈he reach will be large as we have an ageing population,鈥� he says.

The biggest advantage of camera monitoring may come after patients are discharged. Wiring people up to keep track of their health is impractical once they鈥檝e left hospital, but camera-based systems could work.

鈥淧erhaps people could go home earlier from hospital and we could monitor them remotely,鈥� says Watkinson. If chronically ill people could be monitored from home, he says, they could might avoid coming into hospital at all.

This article appeared in print under the headline 鈥淎 watchful eye鈥�

Chip device

Live in the US? There鈥檚 a 50:50 chance that you鈥檙e in a police face recognition database, according to from the Center on Privacy & Technology at Georgetown Law in Washington DC. The findings suggest that about a quarter of all police departments in the US have access to face recognition technology.

That police are using face recognition technology is not a problem in itself. In a world with a camera in every pocket, they would be daft not to. But face recognition can be used far more broadly than fingerprint recognition, which means it carries a higher risk of tagging innocent people.

Fingerprints are difficult to work with. Prints from known criminals can only be gathered in controlled environments at police stations, and dusting for prints is so time consuming that it is only done at relevant crime scenes. This narrows down the number of people in the sights of any one investigation.

It鈥檚 much easier to build huge databases of identified photographs. The majority of the 117 million faces in the police datasets come from state driving licenses and ID cards. And when trying to solve a crime, gathering faces is as easy as pointing a camera at the street. People attending protests, visiting their church, or just walking by can all have their faces 鈥渄usted鈥� without ever knowing it.

Tech isn鈥檛 colourblind

That means most of the faces in the database are the innocent public, not hardened crooks, giving police forces a bigger canvas on which to make mistakes. 鈥淚t鈥檚 uncharted and frankly dangerous territory,鈥澛� said , who led the Georgetown report, .

And face recognition software is far from perfect. Under ideal conditions, which are rarely achieved in reality, face recognition is less accurate than fingerprint recognition, says of 聽Michigan State University.

Facebook鈥檚 face recognition software has made headlines for 鈥溾��, but systems dealing with grainy CCTV images are nowhere near this good. A big database of innocent people could actually make it harder to fight crime, because the software may start turning up more false matches than human investigators can check.

There are next to no regulations on how the police use this technology, or how much weight they give to its results. Face recognition鈥檚 mystique is strong enough that, without guidance, officers may overvalue the software鈥檚 output, and unconsciously favour evidence that matches its results.

Face recognition systems are also likely to be biased against black people. Since black people are arrested more often than white people, black faces are over-represented in the mugshot databases. This means that innocent black people are more likely to be linked to a crime by face recognition than innocent white people.

Face in the crowd

At the same time, has shown that commercial face recognition software is less accurate at analysing the faces of black people, women and children, compared with white men. So not only is the software likely to point the finger at a larger number of black people, it also points less accurately at black people than anyone else.

None of the four main companies selling face recognition technology 鈥� Cognitec, NEC, 3M Cogent and Morpho 鈥� are open about how their software works, or what datasets they use to train it. 鈥淭hey will not tell you what is the size of their database or where they get it from,鈥� says Jain. 鈥淭hat鈥檚 all proprietary.鈥�

Not even the FBI knows what it鈥檚 doing. In May this year, the US Government Accountability Office (GAO) on the FBI鈥檚 face recognition programme, stating the agency had not tested to see how often errors occurred. By conducting better tests, the GAO said, the FBI could be more sure that its system 鈥減rovide leads that help enhance, rather than hinder, criminal investigations鈥�.

Like all forensic techniques, face recognition has the power to catch criminals police might otherwise miss. But to do so, its results must be transparent and reliable 鈥� otherwise you might as well just pick someone out of the crowd.

鈥淲e鈥檝e operated our water systems by the seat of our pants for the past century鈥�

Beeping monitors. Stick-on electrodes. The finger clip that takes your pulse. Modern medicine comes with a cacophony of sensors. Oxehealth, a company spun out from the University of Oxford, is aiming to replace them all with just one 鈥� a camera.

uses camera data to measure heart rate, respiration and blood oxygenation from a distance. It鈥檚 now trying out the technology in the real world, in hospitals, psychiatric wards and police stations.

One place the tech is being tested is at the , Oxford鈥檚 main teaching hospital. Patients who will need a spell in intensive care after their surgery are being asked to participate in the trial: Oxehealth鈥檚 cameras will monitor their vital signs in parallel with conventional sensors.

鈥淣ormal monitoring involves sticking things on patients,鈥� says Oxford University Hospital Trust intensive care doctor Peter Watkinson. 鈥淚t makes it less easy for them to get out of bed, have a shower or go to the loo. Every time you roll over in bed you pull the thing off.鈥�

The Oxehealth software watches for the tiny pixel changes in video frames as a patient鈥檚 chest rises and falls when they breathe, for example. It also tracks subtle changes in the pinkness of their skin, using that to infer their pulse. The software works with regular video cameras and runs on a computer connected locally or over the internet. It could share results with doctors on screen, or trigger an alarm if vitals drop beyond a specified range.

No obvious symptoms

Watkinson thinks camera monitoring will make it possible to catch patients whose condition is deteriorating before their symptoms are obvious. At the moment, clinicians have to decide which patients they need to monitor 鈥� usually those who are recovering from a procedure or are already very ill. This means doctors may miss the signs that someone is becoming sicker. But the cameras would be able to keep tabs on everyone in their vicinity.

鈥淚t gives the opportunity in the future for pervasive monitoring throughout the hospital,鈥� says Watkinson. 鈥淚t would be much more reliable picking up patients at risk of deterioration throughout the hospital.鈥�

Oxehealth鈥檚 Oliver Gibson says the company has also been doing studies with London鈥檚 Metropolitan Police Service and Broadmoor high-security psychiatric hospital. Here the firm is 鈥渢rying to solve the problem of making sure someone is safe when they鈥檙e in a secure room鈥�.

In April this year, Broadmoor installed cameras in one of its rooms, and then monitored volunteer patients and staff. Staff would sit in the room during the day, while patients would sleep there at night. Both were wired up with sensors to validate the readings the cameras took.

Broadmoor鈥檚 patients need regular monitoring, says the hospital鈥檚 director Robert Bates 鈥� especially at night. 鈥淲e usually check on patients every 15 minutes,鈥� he says. It can be hard to tell whether someone is breathing just by looking through their door, so staff have to approach patients to make sure they鈥檙e OK. 鈥淚t鈥檚 upsetting for the patients because they鈥檙e being regularly disturbed,鈥� says Bates.

Ditch the wires

The camera system could offer a less intrusive means of keeping tabs on residents. 鈥淚t will also help us monitor our patients鈥� physical health after administration of medication,鈥� says Bates. 鈥淎nti-psychotics can have impact on patients鈥� physical health.鈥�

Gibson says the Broadmoor study confirmed the cameras鈥� accuracy. 鈥淲e were getting a breathing rate accuracy where 94 per cent of measures were within two breaths per minute of a contact device,鈥� he says. 鈥淣inety-four per cent of heart rate measures were within three beats per minute.鈥�

Oxehealth is not alone in its ambition to free patients from their wires. at RWTH Aachen University in Germany is setting up a wireless monitoring ward in the university hospital. Six beds in the geriatrics department will have cameras monitoring their occupants鈥� vitals.

鈥淲e鈥檙e setting it up now,鈥� says Leonhardt. 鈥淲e鈥檒l find out what鈥檚 good for elderly homes in general. The reach will be large as we have an ageing population.鈥�

The biggest advantage of camera monitoring may come after patients are discharged. Wiring people up to keep track of their health is impractical once they鈥檝e left hospital, but camera-based systems could work.

鈥淧erhaps people could go home earlier from hospital and we could monitor them remotely,鈥� says Watkinson. If chronically ill people could be monitored from home, he says, they could perhaps avoid coming into hospital at all.

鈥淭he maps need to be there for the autonomous cars to be able to do what they need to do鈥�

Small beginnings

鈥淎 car has far more computational power than a phone and much better sensors鈥�