¿ìè¶ÌÊÓƵ

Hollywood producers must long for actors and actresses who never throw
tantrums, who never miss a shoot because of a hangover, and who never move
away to take up a better offer elsewhere. Their wishes are close to coming
true, as electronic clones of real actors and actresses are being made ready
to perform any time, anywhere, without the need for stand-ins, make-up
– or even payment.

Among the first to demonstrate the potential of digitally cloned actors
will be French film maker Didier Pourcel in his adaptation of the Jules
Verne novel Twenty Thousand Leagues under the Sea. In the classic story
of the submarine and the octopus, Captain Nemo and his crew will be portrayed
by digital clones based on, among others, the French actor Richard Bohringer.
The clones are nothing more than digital images stored on a computer, but
on screen they will look like human actors, performing against a computer-generated
background.

To construct a digital clone of Bohringer, Pourcel begins by recording
a static 3D image of the actor, using a technique called laser range finding
with structured light. Bohringer sits on a circular platform for 15 seconds
while a laser scanner of the type used in supermarket checkouts circles
around him, illuminating his head with a vertical strip of light. Laser
light reflected from the curves and contours of the actor’s face is picked
up by a video camera at a rate of 60 frames a second. This information is
digitised and fed to a computer program, which extracts information about
the contours of the head from the images and converts it into the 3D coordinates
needed to build a computer model. Meanwhile, a second video camera in the
scanner is focused on the actor’s head a little to the side of the laser
stripe. This camera captures information about the colours and texture of
the head and face, which the computer combines with the graphics model to
produce the clone.

With a static image of the actor’s head scanned in, the next step is
to make it talk and show emotions. This is a problem that has taxed computer
artists for years. Movement of eyes, eyebrows, mouth, cheeks and other features
interact in subtle ways that are particularly hard to model on a computer.
Until now, most 3D computer models used in film and television were constructed
out of polygons – usually triangles. The finer the detail, the more polygons
are required. A simple model of a face, for example, requires around 300
polygons; for a more lifelike model, around 1400 are needed. Colour, texture
and shading are added to the model in the process known as rendering. This
involves a very large number of calculations: it takes up to 10 minutes
to render a single frame of TV quality even on a high-powered PC.

So far, no mathematical formula has been devised that satisfactorily
mimics the actions of the 20 or so major connected muscles in the human
face. On the screen, the pre-programmed movement looks mechanical, and faces
lack expression. This is fine for the The Terminator or the pseudopod in
The Abyss but not so good if you want the image to look human.

Life drawing

Digital clones get round this problem by taking the facial movements
of a live actor and mapping them onto the computer model. One device to
do this, known as the Facial Waldo, was developed by SimGraphics of Pasadena,
California (Technology, 13 March 1993). This consists of a helmet containing
a number of sensors that touch key areas of an actor’s face and measure
the movements that occur when the actor talks or smiles. This information
is fed into a computer program that animates a computer-generated face.
You can see the end result on the Saturday morning children’s programme
Live and Kicking on BBC₁, in which Ratz, a cartoon cat, interacts
in real time with the presenter, showing realistic expressions. Ratz is
animated by comedian Paul Brophy wearing a Facial Waldo.

A system that does the same job as the Facial Waldo, but without the
awkward mechanical sensors, has been developed by France’s National Institute
of Audiovisual Techniques (INA), a government-sponsored education and research
organisation in Bry-sur-Marne on the outskirts of Paris. It was demonstrated
in Monte Carlo in February at Imagina, the annual festival of computer graphics
and communications technology. In the INA system, images of the actor’s
face are captured by a video camera and digitised at a rate of 10 frames
a second. The eyes and mouth are picked out from the images by a program
that recognises their shape and colour, and which tracks the movement of
key points as the actor’s expression changes. This is then used in the computer
modelling and animation process, which produces a series of animated frames
that can be captured on film or video tape.

The facial movements are made more realistic through a series of computer-generated
springs that represent the network of facial muscles. There are 10 springs
to control the mouth alone. Using the image analysis data, the computer
calculates the facial muscle movements and translates them into a system
of forces which are applied to the springs. So, for example, a smile may
involve moving springs four and five upwards. This produces a series of
deformations in the model which accurately mimics the speaker’s original
movements, including the wrinkling of the eyes and the creasing of the cheeks
that proved difficult to model and coordinate with older modelling techniques.
Once a person’s face movements are captured as mathematical formulae, it
is possible to use the same calculations to produce facial expressions on
any computer model.

Medialab, a Paris-based company specialising in 3D computer graphics,
is working with the Institute of Spoken Communication (ICP) in Grenoble,
on a slightly different approach to giving human features and expressions
to 3D computer-generated characters. As in the INA system, the shape of
the actor’s lips is captured on video and sent to a computer which calculates
the coordinates of certain key points that characterise the lip shapes.
But while the INA technique uses image analysis software to pick out the
shape of the lips, the Medialab/ICP system uses conventional video techniques
for this task. The actor’s lips are painted blue, because blue is easy to
separate out of a video image, and this allows them to be tracked by the
software. The resulting data are used to animate the computer character’s
lips.

Joining the dots

A similar technique is being used to capture Bohringer’s face movements
for Pourcel’s Twenty thousand leagues under the sea. Dots of blue tape are
stuck to key points on Bohringer’s face, such as the mouth and around the
eyes. The movement of these dots as Bohringer speaks his lines can then
be extracted from the video image.

Realistic expressions for computer-generated actors are not the only
goal for research on the portrayal of facial movement. Medialab and INA
believe their systems will play an important role in improving the quality
of video-conferencing – an extension of the telephone in which callers
can see as well as talk to each other. These video-based communications
require huge volumes of data. A single frame of TV-quality video requires
about 2400 kilobits of digital data, and Britain’s PAL standard for video
runs at 25 frames a second. Yet even high-capacity ISDN telephone lines
can carry only 64 kilobits of data a second.

A number of image compression techniques are available for reducing
the amount of data that has to be sent down the line (‘Big squeeze for video’,
¿ìè¶ÌÊÓƵ, 28 August 1993). But when images are compressed sufficiently
to be sent over ISDN lines, people’s movements appear jerky. INA believes
its image analysis program can overcome the blur by separating the information
that is relevant to the movement of the face from the information relating
to colour, texture, and so on. Only the data relating to movement is transmitted
continuously, while the information on colour and texture is sent at the
beginning of the conferencing session. INA claims that its system can send
video images portraying facial expressions in real time over conventional
computer networks or telephone lines.

The problem of reproducing human expressions is also being tackled by
Fuji Television’s Computer Graphics Centre in Japan. Fuji’s researchers
want to explore how human clones could be used in conventional media such
as live theatre. In particular, they have looked at ways of overcoming what
Hiroshi Sakomoto of Fuji describes as ‘the cold inhuman image of computer
²µ°ù²¹±è³ó¾±³¦²õ’.

The researchers have created a play telling the story of an advertising
executive who creates a computer graphics character called Kaori. As he
works to give Kaori greater authenticity, she suddenly comes to life. At
first she appears only on a giant screen on stage, but eventually a human
actor takes over and Kaori appears to step out of the screen and onto the
stage. As it is the same actor who has been animating the computer clone,
the facial movements and voice are identical. An eternal triangle then develops
involving Kaori, the executive and his real-world girlfriend, and the brokenhearted
Kaori eventually returns to her electronic isolation in the computer.

Kaori in her computer graphical form has to respond to live events,
so she has to be animated in real time. Fuji’s system, which mapped a video
image of the actress’s face onto the computer graphics model, was similar
to the technique used by Medialab and the ICP. The actress wore blue lipstick,
allowing the computer to extract information about her mouth movements,
which were mapped onto a 3D computer graphics model of her face. Only the
mouth was animated automatically; Kaori’s other features had to be animated
manually by stage assis-tants using a joystick and a special keyboard.

Fuji’s play was devised specifically to explore the potential of digital
technology in future forms of entertainment. Its digi-tally cloned actress
successfully won the audience’s sympathy for Kaori, but bringing the character
to life was no mean feat. More than 200 cast and crew were involved in
the production.

Many questions raised by digital clones still have to be answered. To
what degree is a digital clone an independent entity? Who owns the clone?
Since anyone’s face can be mapped onto a computer graphics model and animated,
should we be worried about people creating impostor clones for propaganda
or other malevolent purposes? How will the film industry be affected by
actors and actresses who need never age, or who can phone in their performance
from an exotic beach to the director on location in a cold, wintry landscape?

Bohringer appears to have no qualms about sharing the future with his
clone. He is pleased to contribute to the advancement of the technology,
he says, likening the move to donating his body to medical science. But
we will have to wait and see whether the new breed of digital clones will
come to haunt us.

Clive Davidson is a freelance writer.