Skip to main content

Harnessing Serendipity

Innovation is critical to sustained economic growth—and mathematics can help us understand how it works 

This article was published in Scientific American’s former blog network and reflects the views of the author, not necessarily those of Scientific American


Imagine that you are a budding chef, and your objective is to create new culinary experiences. To do so, you add ingredients to your store cupboard in any order you choose such that you can construct as many recognized recipes (culinary innovations) as possible. But is it possible to have a strategy of innovation, given the quirkiness of what constitutes a valid recipe? And how would you factor in the seemingly imponderable serendipity of today's ingredient choices paying off later in unexpected ways?

These were some of the questions we posed in a recent collaboration between the BCG Henderson Institute and the London Institute of Mathematical Sciences. By modeling innovation as a search for valuable combinations of components (products), informed only by the knowledge of what has already worked for yourself and competitors, we aimed to get behind the important but murky topic of innovation strategy. Innovation is critical to sustained economic growth and has been variously explained as luck, special vision or, at the other extreme, just another business process to be optimized for efficiency. We looked both deductively at the recombinatorial math of innovation, and empirically at the performance of different strategies in real world innovation spaces where the innovation game has already played out, including language, gastronomy and technology.

The mathematics of innovation


On supporting science journalism

If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.


We made an exciting discovery. We found that it is indeed possible to have an information advantaged strategy of innovation. Innovations can be characterized by their complexity—the number of unique components that they contain. Ingredients can also be characterized by the average complexity of the recipes they occur in. And innovation spaces have a characteristic distribution for the complexity of valid recipes, which determines how the innovation process unfolds. We found that impatient strategies—ones focused on simple ingredients and recipes with an immediate pay off were most successful early on. This echoes the “minimum viable product” strategies which are popular with tech start ups.

Later in the evolution of innovation spaces, patient strategies, which focus on more complex recipes and ingredients and have a postponed pay off, are more successful. These are more like the sustained research programs of large enterprises operating in mature spaces. Both strategies always beat random strategies for choosing ingredients. And by monitoring the average complexity of the recipes in an unfolding innovation space, one can spot the cross over point of these strategies, and thereby construct an adaptive strategy which optimally combines the attributes of patient and impatient strategies.

Understanding serendipity and disruption

So, how can we understand the mysterious idea of serendipity? In each innovation space we looked at, the popularity of some ingredients increased over time as the complexity of product recipes grew. In other words, components that looked worthless initially could become more prevalent in later innovations. Garlic for example, is such a “serendipitous” ingredient, which hardly occurs in simple recipes but is very common in more complex ones. The chart below shows the popularity of ingredients across all recipes logged in popular recipe databases, as more ingredients are added to the chef’s store cupboard and the upward sloping lines represent the serendipitous ones.

Credit: BCG Henderson Institute and London Institute of Mathematical Sciences

Our model also explains phenomena like the “disruption” which many enterprises fear in today’s fast moving technological environment. We found that when separate innovation spaces are merged, the complexity clock was often reset, such that impatient strategies and simple recipes once again predominated, requiring incumbents to also rapidly adjust their approach to innovation.

Applying the math of serendipity

The ideas can be applied to design too. We can consider the user journey for a new piece of software or product as an innovation process in which the user selects and combines different functionalities to achieve their ends. This then begs the design related question of what the optimal sequence of functionalities to build into the product is, in order that the user can have the immediate gratification of simple applications, a manageable learning journey and also a broad range of later possible applications.

We also applied these ideas in situ with an award-winning chef. Together with Chef Rueben Major of the Vancouver Urban Winery, we hosted an innovation dinner for 60 executives and academics at the TED 2017 conference in Vancouver. We chose recipes to reflect the career journey of the chef, which mirrored well the patterns observed in our research (see photo below).

Thai Curry. Credit: TED2017

The meal kicked off with Swedish pancakes, a simple recipe which comprised simple ingredients such as eggs and flour, a dish reminiscent of the chef’s childhood kitchen. We then went on to savor a complex South East Asian curry as the main course, containing many ingredients characteristic of complex recipes as a main course, followed by a dessert that served as a culinary representation of a disruption caused by the introduction of new technologies.

In the past, many business functions based their decisions mainly on tradition or intuition. Many, such as the marketing or HR departments, are far along on the journey to become more analytical and evidence based. Our findings give us hope that innovation too can be at least partially tamed by applying information based approaches such as the ones discovered in our research.

References:

Martin Reeves, Thomas Fink, Ramiro Palma, and Johann Harnoss (2017): "Harnessing the Secret Structure of Innovation."Sloan Management Review March 2017, http://sloanreview.mit.edu/article/harnessing-the-secret-structure-of-innovation/

Thomas Fink, Martin Reeves, Ramiro Palma, Robert Farr (2017): "Serendipity and Strategy in Rapid Innovation." LIMS working paper, see https://arxiv.org/abs/1608.01900