Game Developers Crossing the Uncanny Valley

Meet “Ira.”

Ira isn’t an actual human being—he’s just a computer model—but you’d be forgiven for not being able to tell the difference.

“Not only does Ira look real, but this is all generated in real time,” Nvidia, the graphics chip company, suggested in a recent video demonstration. “His skin has bumps and blemishes that move and stretch as Ira blinks and grimaces. Those imperfections make Ira far more real than the fake, rubbery faces generated by previous generations of graphics technology.”

Ira’s appearance and movements are based on those of a real person, recorded by the USC Institute for Creative Technologies, which has developed one of the world’s most advanced methods of motion capture. His facial movements are renders using some of the most powerful software and hardware on the market.

But you won’t see Ira, or any digital character that looks quite as realistic, in any video game that exists today.

Nearly 30 years after Super Mario Bros., video game graphics have advanced to heights that once seemed impossible. Modern sports games are fueled by motion capture of actual athletes. Narrative-driven adventures can seem more like interactive movies than games.

But gaming’s increasing realism brings a side effect—a game can now fall into the “uncanny valley,” a term coined by robotics professor Masahiro Mori of the Tokyo Institute of Technology in 1970.

Mori “hypothesized that a person’s response to a humanlike robot would abruptly shift from empathy to revulsion as it approached, but failed to attain, a lifelike appearance,” states the introduction to an English translation of Mori’s original essay on the topic. “This descent into eeriness is known as the uncanny valley.”

Not everyone may experience “revulsion,” especially as we become more comfortable with animated humans from movies and games. But games can suffer when characters just look… weird. A digitally rendered athlete often looks pretty natural as he’s hitting a home run or catching a touchdown pass, but the sense of realism is broken when the athlete woodenly celebrates with teammates and shows off his sad facsimile of a human face. In story-driven games, characters that look almost fully human often betray their true nature when performing complicated movements or physically interacting with other characters.

“To go from good-looking, real-time graphics to something that is photo-realistic, in my mind that gap is still quite large, unfortunately,” Nick Thibieroz, worldwide manager of gaming engineering at AMD, told Slashdot. “It will take a few years if not more to get to a place where that approaches photo realism for real time rendering.” That may not happen with the current console generation of the PS4 and Xbox One.

“I think every game out there right now has an uncanny valley problem,” added Mark Daly, senior director of content development at Nvidia. “What they do is hide it. It’s pretty rare that you actually see the face of a character in most games. They usually have some kind of mask on, a helmet with a visor on to hide the facial motions, or it’s not a real human.”

Illusions of Realism

There’s nothing wrong with evading the uncanny valley by creating non-human characters. Realism isn’t necessary for a game to be fun or visually stunning.

But in games where human-like characters are necessary, the uncanny valley can be an even bigger problem than in animated movies. Gamers control characters rather than just watching them, creating more opportunities for the illusion of realism to falter.

In video games, “there’s almost no control over how the characters are going to be viewed,” said Karl F. MacDorman, associate professor in the Human-Computer Interaction program of the School of Informatics at Indiana University. “Oftentimes you’re able to view them from any direction. You can rotate around the arena and look at the characters from different directions. There’s not the same control over the presentation, the posing, the lighting, how it’s rendered and so on that you would have in film, so I think the uncanny valley in theory could be a larger problem in games.”

MacDorman added a caveat to that point: The emotional problem of the uncanny valley is a “loss of empathy” with the lead character. “We lose our ability to really see the other character as a fellow human, to be able to emotionally project ourselves into the situation the other character is feeling.”

Because of that, the uncanny valley may be less of a problem in games where story plays second fiddle to gameplay. Even in games where the story is crucial, the fact that the player directly controls the hero makes the player’s identification with that character stronger, making it harder to trigger a loss of empathy, MacDorman said.

Our brains are extremely good at processing human faces and detecting emotions from facial expressions, so even small imperfections won’t go unnoticed. We’re “over-trained” in observing humans, but if we became similarly over-trained in observing digital doubles, that too might help us overcome the uncanny valley, MacDorman said.

A Vexing Problem, But Many Strategies to Solve It

While human-like characters are an option for some games, others require them. Sports simulators, many created by Electronic Arts (EA), are perhaps the best example of this, because they attempt to recreate real players. EA uses motion capture to increase the accuracy of athletic movements, but the unrealistic faces display the limitations of technology.

“Some of these EA sports games are the worst violators of the uncanny valley,” MacDorman said. “They have the most horrific looking characters that really look like vacant possession, walking zombies.”

We weren’t able to interview anyone at EA, but Daly noted that gamers shouldn’t be surprised when digitally recreated athletes aren’t completely lifelike.

“If you have a basketball player or a tennis player or baseball player, they don’t usually wear a mask, and they have to be human,” Daly said. “If I’m going to make a realistic looking basketball player I can’t use all the tricks that these fantasy games use…. Not only do they have to make a guy but they have to make a guy that looks exactly like somebody we know…. They take on what is by far the hardest problem.”

Daly explained that game developers today use several techniques for designing characters, with 3D animation software products like Maya and 3ds Max. At the most basic level, there is “simple, rigid body animation,” he said. “That skeletal animation is the foundation of everything. Here’s where the problems come up: When you go from skeletal animation to adding flesh and skin onto the character, now you’ve got technique #2, which is called skinning. Basically, you put a body glove over this skeleton and now there’s a mesh wrapped around these skeletal bones and that mesh is tied to the bones, so as soon as a bone moves the mesh moves with it.”

Techniques for adding flesh to the rigid skeleton tend to be “a hack, if you will,” Daly added. Volume in the joint areas is lost, and elbows end up looking like a collapsed hose.

“If you look at any game, it’s pretty rare that you see a character raise his entire arm above his head because that’s probably the worst case for this skinning operation, because your whole shoulder basically collapses,” he said.

These shortcomings are covered over using a third technique involving “blend shapes,” which combine shapes together to create other shapes. These can be used all around the body to restore volume and make a character’s motions look more natural.

Blend shapes are crucial in creating realistic faces. “If all we did was rely on skin weighting, you’d look like a South Park character when you talked,” Daly said.

That necessitates a variety of blend shapes based on different human expressions. “You create anywhere from a dozen to two dozen face shapes, like big smiles and frowns and big scowls,” Daly said. “For pretty much every extreme shape your face could make, we store a mesh describing the face in that position. Now if I’m a good animator I can make pretty much any facial expression using some combination of these blend shapes.”

With Ira, Nvidia used about 24 blend shapes, with around eight contributing to each expression. But the success of Ira couldn’t have been achieved without the performance of a real actor.

Evolving Motion Capture

When a motion capture actor performs, “they put these highly reflective dots on your joints and on your bones and they videotape you, and they process that video after the fact to figure out where your joints and bones were and they build a 3D skeleton out of that,” Daly said.

Motion capture is “the best we have today to carry out realistic animations,” Thibieroz said. Still, it has limits.

Classic motion capture “just gives you the skeletal information,” Daly continued. “Because this whole facial thing is such a problem they’ve made attempts to use mocap techniques to create blend shapes. They’ll put tiny little dots on someone’s face, and the more dots you have, the better you are at capturing the different facial expressions.” Face paint designed for motion capture also helps the cameras pick up more detailed expressions from actors’ faces.

But it’s a new kind of motion capture, the one used to create Ira, that has Daly excited. It was developed by Paul Debevec at the USC Institute for Creative Technologies, whose work has been used in movies such as Spider-Man 2, King Kong, The Curious Case of Benjamin Button, and Avatar.

Daly says that Debevec’s “Light Stages” are able to capture far more detailed information on human features than standard motion capture, which is more about recording body movements than facial expressions. With Light Stages, an actor is placed inside a sphere with multiple cameras and special lighting that lets Debevec “pull texture information from the skin,” Daly said. “He can give you the color, he can give you the bump maps that describe wrinkles. He can even give you the specular bump map that will describe down to like a tenth of a millimeter the little cracks and pores in your skin. It’s pretty amazing.”

The cameras are able to “capture facial geometry, surface detail, and lighting information of an actor without any of the traditional tricks of face markers or special makeup,” Nvidia notes.

Light stage data helped provide the realistic blend shapes that contributed to Ira. That data is fed into an Nvidia software product called FaceWorks, which does the rendering. Developers can test a demo version of FaceWorks now, and play around with adjusting Ira’s facial features. Eventually, this software or something like it could become a crucial tool in video game creation.

The game publisher Activision has already collaborated with Debevec, signaling that his work may help future games cross the uncanny valley. No games using light stage technology are out today, though. Even with increasing GPU power and expanded storage, the amount of data needed to create photorealistic characters that move and talk like real humans is forbidding.

“If I’m running at 30 frames a second, I want to capture the mesh of the face and all the textures of the face for every 30th of a second for the entire length of a performance,” Daly said. “So if somebody talks for ten seconds, that’s 300 meshes that I need to store just for a 10-second performance, and these meshes can be huge.”

Console Graphics: Not Photo-Realistic, But Pretty Darn Good

While that helps explain why the uncanny valley still exists in gaming, technology’s advance should help make characters more realistic in games on the newest generation of consoles.

A video that shows the upcoming baseball simulator, Sony’s MLB 14: The Show, helps to illustrate this idea. The game is coming out both on the 7-year-old PS3 and the brand-new PS4, which is powered by an AMD Radeon GPU:

As you can see, player faces look far more realistic on the PS4, right down to how hair moves when they walk.

“We have 1,500 different players that our artists built, so for PS4 we have to decide what do we want to do to make them look better,” said Patrick Hager, senior lead graphics programmer on the game. “On PS3 they’re modeled as kind of a Ken doll plastic molded piece. For PS4 we wanted to capture a more realistic version, so instead of modeling a giant molded piece we’re actually drawing up to 40,000 individual hairs. It feels much more like a real person.”

In today’s games, Thibieroz often notices shortcomings in character’s eyes, or teeth that look a bit too perfect. He said one of the most important areas of development right now is in lighting, with many developers using real-time global illumination software that makes sure every scene has realistic lighting, and that the light on characters’ faces changes appropriately each time they move.

While Daly and Thibieroz agree that motion capture is generally necessary to create the most realistic characters, the efforts of game designers are not to be overlooked.

“Perfect art means very high polygon counts, very high resolution textures, and very good, creative animations,” Thibieroz said. “It’s not only about the engineering and the shaders and the lighting techniques. It’s more about art, I would say.”

Besides the difficulties of recording the nuances of human faces, traditional motion capture struggles to record physical interactions between characters. If one motion capture actor hugs another, the markers on their bodies that cameras track are hidden.

“Multiple character animations tend to have a fair amount of human artists in the loop to try to clean up some those mocap, errors, and that’s why some of it looks a little odd,” Daly said.

New and better tools such as FaceWorks will help, but the human element of creating realistic human animations is going to remain crucial. As Daly said, “I don’t care how good the tool is, you’re never going to get past the fact that you need a very talented animator.”


Image: Nvidia