THE phrase “the camera never lies” has never been so wrong. Artificially intelligent smartphones are now editing pictures in real time to create images that can’t be produced by conventional cameras. These enhancements, known as computational photography, are changing the way we view the world.
The goal of digital photography was once to approximate what our eyes see. “All digital cameras, including ones on smartphones, have always had some sort of processing to modify contrast and colour balance,” says Neel Joshi, who works on computer vision at Microsoft Research.
Advertisement
Computational photography goes beyond this, automatically making skin smoother, colours richer and pictures less grainy. It can even turn night into day.
“Take a selfie and a Meitu phone can automatically remove dark circles, widen your eyes or slim your jaw”
These photos may look better, but they raise concerns about authenticity and trust in an era of fakeable information. “The photos of the future will not be recorded, they’ll be computed,” says Ramesh Raskar at the MIT Media Lab.
Digital cameras channel light through a lens to hit a sensor, which converts the light into electrical signals. Processing software then translates these signal into images. The better the lens and the bigger the sensor, the nicer the photo, which is why professionals prefer bulkier digital single-lens reflex (DSLR) cameras over smartphones.
Early computational photography efforts focused on improving these professional cameras, says Raskar. But as sales of smartphones started outpacing those of standalone cameras, more energy went into using digital enhancements to compensate for the poor quality of phone photos. Now phones are catching up with and even overtaking their rivals. “Forget about trying to mimic DSLR cameras,” says Raskar. Smartphones can now fake lighting, for example. “We can do so much more that would not be considered photography to start with.”
Phone companies have given their cameras multiple lenses and more advanced hardware. At last month’s Mobile World Congress in Barcelona, Spain, Nokia released its 9 PureView with five cameras that can produce photos up to 240 megapixels in resolution. Huawei’s P30 Pro, due to be launched next week, apparently has a hardware zoom .
Many of these AI-equipped smartphone cameras take bursts of photos, using multiple lenses to pick up a variety of information. AI algorithms can then combine the best of these images, or create new images that no single camera captured, like a photo with adjustable focus.
Some create pictures with a “high dynamic range” of lumosity, for example, by merging dark and light areas from different photos. This allows you to photograph an outdoor scene with a blue sky in the background instead of an over-exposed white sky.
Night vision
Smartphones can also emulate long-exposure photography, in which a camera shutter is opened for an extended time to let in more light. Night Mode on Huawei smartphones and the Google Pixel’s Night Sight are two such examples, using a burst of multiple images to produce sharp, bright photos when taken in dim settings, without the need for flash. Night Sight also has algorithms that take artificial lighting into account, estimating the true colour of both the light and the objects in the scene and correcting them accordingly.
Taking photos in dark settings has conventionally been a trade-off: let in too little light through the shutter, and the resulting picture looks grainy. But a long exposure time requires holding still – particularly difficult with a handheld smartphone – because movement of the camera or what is in frame results in a streaking of objects known as motion blur.
To minimise blur, Night Sight uses accelerometers to track both the movement in the scene and of your hands before the picture is taken, and adjusts the camera’s shutter speed accordingly. If it detects the phone is stable, like on a tripod, it uses longer exposures. To compensate for movement, it takes a burst of darker photos more rapidly and merges them to a sharper result. The pictures can look startlingly different to reality, with evening scenes appearing lit by daylight (see photos).
“If a whole generation grows up not trusting photos or videos, what do we have left?”
Giving cameras an understanding of what they are looking at is key. The image-processing system on Huawei smartphones, known as Master AI, can recognise more than 1500 different objects, says Peter Gauden, a spokesperson for the company. This allows it to create a “bokeh” effect, for example, in which an object in the foreground can appear in focus, but the background is blurred.
Huawei’s cameras also automatically segment an image into components, adjusting the colours, exposure and contrast of each element. When the system detects greenery, for example, it ramps up the saturation to what it thinks is most aesthetically pleasing.
Fakery and image manipulation predate digital photography, but the growing popularity of instant-editing cameras raises concerns about authenticity, particularly as photographs are used in news reports or as evidence in courts of law.
“Everybody has a different stake in the believability of their image depending on the context in which they’re working,” says Michelle Henning, a cultural historian at the University of West London, but she points out that this has always been the case. “Photography has never just represented the world as we see it.”
Yet there does seem to have been a recent shift. “If a whole generation grows up not trusting photos or videos, what do we have left?” says Raskar. Digital watermarking and cloud-based encryption are possibilities, he says, as is using the blockchain technology behind bitcoin.
Image-editing technologies also introduce the social temptation to continuously beautify one’s image. “It’s become a necessity of contemporary social life, typically for younger people, to have to have their face and their photograph circulating,” says Henning.
Chinese company Meitu, best known for its photo “beautification” apps with some 350 million active monthly users, has recently introduced AI-powered photo editing. Software on its apps and phones can analyse and adjust faces in real time.
Take a selfie with a Meitu phone and it locates facial landmarks, builds a virtual mesh over your face, and segments the skin so it can adjust features accordingly, removing imperfections like dark circles and pimples to give your skin a homogeneous soft glow. By warping the mesh, it moulds your face to aesthetic proportions, widening your eyes or slimming your jaw.
Such alterations might be doing more than just prettying up your Instagram feed. Tijion Esho, a London doctor who performs cosmetic procedures, says he has seen a rise in people requesting treatments based on edited photographs of themselves.
“People were creating very chiselled jaw lines, ones that couldn’t be created [in reality] because it wouldn’t be possible to move your head with that level of definition in the neck,” he says. “In some cases, people wanted their eyes widened in a way that some filters do.”
Unrealistic beauty
Esho believes cameras that edit pictures in real time could drive unrealistic beauty standards, especially if the person using the phone doesn’t know what it is doing behind the scenes.
Pamela Rutledge, director of the Media Psychology Research Center in California, isn’t so sure. “Capturing and sharing beauty is not a new phenomenon,” she says. Prettier images don’t necessarily make people feel worse about their real-life appearance, she adds.
One that people edit selfies out of the desire for a more ideal online presentation, rather than a dissatisfaction with their appearance. However, a , with the effect being more pronounced for women – and the concern is that this will be compounded when unreal images become ubiquitous.
The next decade will see major advances in computational photography, says Raskar. Beyond better-looking photos that are easier to take, he believes there will be a shift towards pictures that capture the emotions and essence of a scene. “I think people really want to create magical experiences, and that will not be done purely by recording the photo – they’ll be done by computation.”
Other clever things your phone’s camera can do
Smart cameras don’t just make your pictures look nice – they can do many other very handy jobs.
Google Translate’s camera feature lets you point your phone at a sign in a foreign country or words in a book (see picture), superimposing an on-screen translation of the text to make it appear in your own language.
Microsoft’s Pix app detects if you are taking a picture of a whiteboard or receipt, automatically straightening and cropping the image to the text contained within. The firm’s Excel app can capture printed data tables and automatically put them into a spreadsheet, doing away with the need for tedious data entry.
Samsung’s Bixby virtual assistant has a calorie counter built into its camera (see picture below). Point it at what is on your plate and it will give you an estimate of the food’s energy content.
MEITU’s MakeupPlus app lets you virtually apply more than 3000 cosmetic products in real time, then order them for real.
