Skip to main content

If (Virtual) Reality Feels Almost Right, It's Exactly Wrong

How adding touch to VR can lead to an “uncanny valley” of sensations—and what we can do about it

This article was published in Scientific American’s former blog network and reflects the views of the author, not necessarily those of Scientific American


We can all remember the crisply beveled edges of our cheery-yellow No. 2 pencil, the cool, smooth feel of a chalk-powdered blackboard, the gritty red bricks of the schoolhouse walls. Surely that all wasn’t just an illusion?

No, of course not.

But—as it turns out—it kind of is.


On supporting science journalism

If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.


The sense of touch (and indeed, all of our senses) is more or less illusory because no sensation stands alone. Stretch out your hands and snap your fingers. This, of course, feels very real. But you’re seeing your fingers, hearing your fingers, and feeling your fingers—and all of these sensations fully correspond.

Now, what if they didn’t?

Virtual Reality (VR) is a great tool for revealing this strange and otherworldly foundation of our everyday sensory perceptions. Sneaky (but, of course, highly ethical) experimentalists such as ourselves can render a completely computer-generated world. If we hand you a pair of controllers that can vibrate on command, we can play tricks.

We can even use this simple apparatus to produce a sensation of touch that “feels like” it originates in the completely empty space between your outstretched hands—an experiment we discuss in our paper in the current issue of Science Robotics.

This VR equivalent of a carnival trick is known as funneling. Each controller provides the user with synchronous vibrotactile stimuli of different amplitudes. In the virtual environment, we show the participant a wooden dowel that they grip with each hand. The dowel is only imaginary, of course—the hands are not physically linked, and the dowel does not exist. We then show a small white marble that seems to knock on the dowel at different locations as we vary the vibratory sensations.

Figure 1: The participant holds the wooden dowel with the Virtual Reality controllers as he gets synchronous vibrations of different amplitudes. The result is a spatialized touch illusion, a touch that “feels like” it originates in the completely empty space between your outstretched hands. Credit: Mar Gonzalez-Franco, Christopher C Berger and Ken Hinckley

This is where it gets weird. The apparent spatial location of the touch sensation will persist, even if the marble is invisible to the eyes of the beholder. Your brain fills in these gaps and inconsistencies—the vibratory sensations felt by your hands, the (at times imaginary) little white marble knocking on the dowel in the physically empty space in-between. A particular spatial location that happens to be completely outside of your own body.

Figure 2: Participants were able to estimate the location with high accuracy of the touch both when the marble was visible (green dots) and when the marble was invisible (red dots). Credit: Mar Gonzalez-Franco, Christopher C Berger and Ken Hinckley

This is a neat perceptual illusion, but why does it matter?

Well, in virtual environments (or robotic tele-operation tasks), producing higher-fidelity haptic (that is, tactile) sensations of this sort come with the (oft-unstated) assumption that such “improvements” will always yield more a realistic and immersive experience. Using this funneling illusion when the marble was imaginary we can demonstrate an “uncanny valley” effect for haptic sensations, in which the distorted sensations of touch produced in our experiments that were almost right.

As a result, they were exactly wrong, and the immersion was broken.

But this uncanny valley can be shifted or eliminated by subtly manipulating the experimental conditions. For example, if the dowel appears to be inside a smoky-looking cylinder, thus partially obscuring the location of the knocking (but invisible) marble, this provides a plausible reason for any discrepancy between the perceived location of the haptic sensation, versus what we see with our eyes—and the simulation feels more realistic.

Likewise, the sensation can be manipulated based on whether we are simply holding the virtual dowel—thus experiencing the knock of the marble as a passive participant—or actively moving the dowel up and down, thus becoming the ‘agent’ of the sensation through our own actions.

Although demonstrated in our curious VR testbed at present, these effects are rooted in human perception. As such, understanding them more deeply helps us build better and more convincing virtual environments.

About Mar Gonzalez-Franco, Christopher C Berger and Ken Hinckley

Mar Gonzalez-Franco, PhD is a Researcher at Microsoft. Her work is at the intersection of Virtual Reality and Human Perception. In her research she fosters strong immersive experiences using different disciplines: computer graphics, computer vision, avatars and haptics. All while studying human behavior, perception and neuroscience. Follow her on Follow her on Twitter @twi_mar. Christopher Berger, PhD is a Neuroscientist at the California Institute of Technology working on developing novel sensory experiences using sensory augmentation and substitution devices by conducting and applying research that probes the basic psychophysical and neural mechanisms of sensory perception. Follow him on Twitter @c_c_berger. Ken Hinckley, Ph.D. is Principal Research Manager of the EPIC (Extended Perception, Interaction, and Cognition) research group at Microsoft Research. His work explores how to reduce the impedance mismatch between humans and technology. This often involves new ways of interacting with devices via sensors, multi-modal inputs, and rich sensory cues. Follow him on Twitter @ken_hinckley.

More by Mar Gonzalez-Franco, Christopher C Berger and Ken Hinckley