Skip to main content

How the Brain Processes Images

A visit to China reminds a neuroscientist that no matter how differently different cultures see the world, they process images in the same way

This article was published in Scientific American’s former blog network and reflects the views of the author, not necessarily those of Scientific American


No matter where we call home, where we were raised, or what we ate for breakfast, our brains process information pretty much the same as anyone else in the world.  Which makes sense—our genomes are 99.6-99.9% identical, which makes our brains nearly so. Look at a landscape or cityscape and comparable computations occur in your brain as in someone from another background or country.

Zhangjiajie National Forest Park, China. Credit: Chensiyuan, via Wikimedia Commons under GFDL

Consider my recent walk through China’s Zhangjiajie National Forest Park, an inspiration for James Cameron’s Avatar. Some of our first steps into the park involved a 1,070 foot ascent in the Bailong elevator, the world’s tallest outdoor elevator. Crammed within the carriage were travelers from Japan, India, China, the U.S.A., and Korea.


On supporting science journalism

If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.


Bailong elevator, Zhangjiajie National Forest Park. Credit: Ivan Dubya via Wikimedia Commons under creative commons license.

No matter our origin, the Wulingyuan landscape didn’t disappoint: the towering red and green rock formations stretched towards the sky as they defied gravity. Gasps and awes were our linguistic currency while our visual cortices gleefully fired away.

The approximately 3000 quartzite sandstone pillars, with their unusual red and green contrasts, mesmerized our visual centers, demanding our attention.

One of the brain’s earliest visual processing centers, V1, lies at the middle of the back of our head. V1 identifies simple forms like vertical, horizontal, and diagonal edges of contrasting intensities, or lines. Look at a vertical line, and neurons that are sensitive to vertical lines will fire more quickly; look at a horizontal line, and our horizontal neurons buzz away.

Downstream visual centers (conveniently called V2, V3, V4) weave together these basic visual forms to create the beginnings of a visual tapestry. More complex forms like squares and circles are assembled; motion and color are added to the picture.

The content of visual information—for example, the density and direction of the lines—determines how much of our attention a visual stimulus grabs. Studies have shown that a high density of lines, for example, grabs our attention in what is known as bottom-up orienting.

The brain’s orienting network selects important, behaviorally relevant information from what we see, hear, and touch. Imagine walking down Shanghai’s Bund in the cool evening air. As we gaze across the river at the Pudong District’s glowing skyscrapers we choose to shift our attention from skyscraper to skyscraper. This is called goal-directed, or top-down orienting.

Shanghai's Pudong District, seen from the Bund. Credit: Zhang Zhang via Wikimedia Commons under Creative Commons license

Contrast this deliberate shift of attention with the reflexive, almost forced shift of attention when, staring at the skyline, a motorcycle taxi honks, demanding that we acknowledge we’re in his way. This is called stimulus-driven, or bottom-up orienting.

Visual scenes with busy, crisscrossing lines grab and keep our attention by engaging stimulus-driven orienting networks—those line neurons fire fast enough to collect their slice of consciousness. Bright, bold, or contrasting colors do the same.

The combination of towering vertical lines and other-worldly fluorescent lights were the substance of the Bund’s nighttime enchantment, which attracted a global cadre of visitors—visitors that responded with smiles of approval or thoughtful gazes.

By day, Shanghai’s M50 art district also deserves attention. One of the artists, Qing Sima, appeared particularly attuned to lines.

Whether he’s painting trees billowing in the wind, “lost” abandoned buildings, or reeds in a swamp (in which he concealed a cuboid), his pieces strongly feature geometric shapes, lines.  In some pieces, he teases you with mere shadows of lines, forcing your brain piece them together as you will.

For the last 12 years, Sima has repeatedly painted Shanghai’s most dramatic architectural display of lines: the Jin Mao tower.  Completed in 1999, the Jin Mao tower captures traditional Chinese architecture and modern design within an exoskeleton of crisscrossed aluminum pipe. In his series of paintings, Sima details Jin Mao’s nearly 1380 feet of brain-tickling lines.

When I asked how Sima planned each project, he told me that before each painting, he stakes out the tower and snaps a photo to use as a guide. His favorite view is at about 45 degrees from one of the faces:

“The angle is very important. I take a photo to keep my perspective. I am very careful, like a scientist,” he told me with a grin, “But I do it by eye, by feel, by experience. I reduce the details using the photo to focus my lines and give me perspective. Taking away some details allows me to express emotion, making it more abstract.”

Sima’s trick was to blur the upper edges and set the tower within an intensely solid blue, yellow or red background that evoked awe, turmoil, passion. By subtracting detail and adding a tinge of atypical color, Sima had hit his mark.

Standing in Sima’s half-basement studio, I wondered if the act of perception, of inference from incomplete information formed the substance of emotion, whether emotional networks were really extrapolation or inference circuits, evolution’s quick solution for patchwork processing. Indeed, while combinations of lines are strong snaggers of attention, scenes that are ominous or emotionally provocative are the most potent mesmerizers.

Sima’s choice of a solid, intense color combined with dense combinations of lines fascinated me.  With the help of Baidu’s translation app, we discussed a classic study of visual attention by neuroscientists at John Hopkins University. The researchers wanted to know whether image features like color, intensity (average RGB luminance), and line orientation affected bottom-up attention. They used an eye-tracking task to measure how much each feature attracted participants’ eyes and discovered that, in cityscapes, color intensity and line orientation were most important in luring participant’s brains to attention.

“Pretty cool eh?” I said. We chuckled at how Sima’s paintings were, in no small fashion, a neuroscience experiment. 

Pointing at his head, Sima replied, “Same brain.”

 

Further Reading on Attention

Barron DS, Castellanos FX (In Press). “Attention Networks.” In: Pfaff DW, Volkow N, Calderon DP (eds.) Neuroscience in the 21 st Century, 2 nd Edition. Springer Press. A primer on how neuroscientists study the attention system and why attention is divided into orienting, alerting, and executive networks. Emphasis is also placed on what and how lesion-deficit, behavioral, and neuroimaging data can teach us about the brain.

Corbetta, M., & Shulman, G. L. (2011). Spatial neglect and attention networks. Annual Review of Neuroscience, 34, 569–599. doi:10.1146/annurev-neuro- 061010-113731. An in-depth yet highly readable review of attention networks and how they relate to the effects of stroke.

Petersen, S. E., & Posner, M. I. (2012). The Attention System of the Human Brain: 20 Years After. Annual Review of Neuroscience, 35(1), 73–89. doi:10.1146/annurev-neuro-062111-150525. A more in-depth review of the brain’s attention system, with emphasis on the latest studies.