This article was published in Scientific American’s former blog network and reflects the views of the author, not necessarily those of Scientific American
Why do groups stop innovating well when they grow large? How can large teams or companies or research groups innovate faster and better?
Over the past two decades, scientists have been turning to the science of emergence, the study of surprising collective behaviors, to help us understand a broad range of systems: how birds flock, fish swim, brains work, diseases erupt, ecosystems collapse, and, more recently, how groups of people behave. Now we can use similar principles to help us design teams that are better suited for nurturing radical new ideas—the breakthroughs we desperately need to treat cancer, for example, or reverse climate change.
The recent resurgence of emergence traces back to a classic 1972 essay by the physicist and Nobel laureate Philip Anderson: “More Is Different.” Anderson railed against a widespread view that the only science that mattered was the science of the small—“fundamental” laws of nature—and argued the merits of understanding the science of the many: emergent behaviors.
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
The essay, still worth reading today, notes that we could never explain emergent behaviors, such as superconductivity, in which all electrical resistance inside metals suddenly disappears, starting from theories that apply at much smaller distance scales, such as quantum electrodynamics. At each level of the hierarchy of nature, Anderson wrote, “entirely new laws, concepts, and generalizations are necessary, requiring inspiration and creativity to just as great a degree as in the previous one.” In other words, he continued, the whole “is not only more than, but very different from, its parts.” More is different.
Anderson focused on fascinating properties of matter, such as the metal-insulator transition or superconductivity. Researchers extended these ideas to any complex system, meaning a whole made of many interacting parts that follow certain rules or principles. The key is to identify properties of the whole that are not sensitive to the details of those parts. Analyzing the interactions between large groups of buyers and sellers, for example, can help us identify collective properties of financial markets. (The Nobel Prize–winning economist Paul Krugman noted that “When Adam Smith wrote of the way that markets lead their participants, ‘as if by an invisible hand,’ … what was he describing but an emergent property?”)
Even more recently, Jim Sethna identified emergent behaviors among “humans in mosh and circle pits at heavy metal concerts,” and Jonathan Touboul used similar techniques to explain “the hipster effect”: why anti-conformists all look the same.
A recent analysis applies similar principles to help us understand the behavior of teams and companies. That analysis gives us new insights about designing more innovative organizations.
Here’s how it works: The key insight in all these applications, from superconductivity to financial markets, is to start with a model of the underlying system that is just simple enough. In other words, a model that captures enough real-world properties of the interactions so that we can see the key features of the system we wish to study, but not so much that the problem becomes intractable. A friend once summarized this as “keep it simple, but not simplistic.”
A key feature we wish to understand in any complex system is the sudden change between two types of emergent behaviors: a phase transition. Water, for example, will suddenly change from liquid to solid. Metals will suddenly change from ordinary conductors to superconductors. Why?
Systems snap when the tide turns in a microscopic tug-of-war. Binding energy ties to lock water molecules into rigid formation. Entropy encourages those molecules to roam. As temperature decreases, binding forces get relatively stronger, and entropy forces get relatively weaker. When the strengths of those two forces cross, the system snaps. Water freezes.
All phase transitions are the result of two competing forces, like the tug-of-war between binding energy and entropy in water. And that’s where we begin with teams and companies: when people organize into any kind of group with a mission, and a reward system tied to that mission, they also create two competing forces—two forms of incentives. We can think of the two competing incentives, loosely, as stake in outcome and perks of rank.
As structure changes, one grows stronger and the other grows weaker. When groups are small, for example, everyone’s stake in the outcome of the group project is high. At a small biotech, if the drug works everyone will be a hero and a millionaire. If it fails, everyone will be looking for a job. The perks of rank—job titles or the increase in salary from being promoted—are small compared to those high stakes.
As teams and companies grow larger, the stakes in outcome decrease while the perks of rank increase. When the two cross, the system snaps. Incentives shift from encouraging a focus on projects and outcomes to encouraging a focus on politics and promotion. A simple—but not simplistic—model of incentives inside organizations allows us to calculate when this transition will occur.
What does this mean for innovation?
The most important breakthroughs—the ones that change the course of science, business or history—are fragile. They are rarely announced with blaring trumpets and a red carpet, dazzling everyone with their brilliance. Instead, they often arrive covered in warts—the failures and seemingly obvious reasons they could never work that make them so easy to dismiss. They pass through long dark tunnels of skepticism and uncertainty, their champions dismissed as crazy. For lack of any better term, I call them loonshots.
In the first phase of team organization mentioned above, when stake in outcome dominates, incentives favor uniting around these early stage-projects. Individuals have so much collectively at stake in the outcome of their mission, that they will come together to rescue those projects from their inevitable stumbles and wrong directions. We can call this the loonshot phase.
In the second phase, when perks of rank dominate, incentives favor a focus on careers and promotion. Early-stage projects covered in warts are rejected in favor of ideas that raise the fewest objections. Those are typically franchise projects: the next generation of an already-established product or program (the next statin drug, the next Avengers movie). We can call this the franchise phase.
The sudden change between these two emergent behaviors is a phase transition.
The bad news about these changes in organizations is that phase transitions are inevitable. All liquids freeze. The good news is that understanding the forces that cause a transition allows us to manage it. Water freezes at 32 degrees Fahrenheit. On snowy days, we toss salt on our sidewalks to lower that freezing temperature. We want the snow to melt rather than harden into ice.
We use the same principle to engineer better materials. Adding a small amount of carbon to iron creates a much stronger material: steel. Adding nickel and tungsten to steel creates some of the strongest alloys we know: the steels used inside jet engines and nuclear reactors.
Understanding the analogous control parameters that govern the transition inside teams and companies helps us engineer more innovative organizations.
None of these insights would be possible by analyzing the behavior of individuals on their own, just as we could never understand why liquids flow or solids shatter by analyzing individual molecules on their own.
Innovating well is an emergent behavior. More is different.