Skip to main content

To Fix the Reproducibility Crisis, Rethink How We Do Experiments

Scientists are taught to vary one factor at a time, but a so-called multifactorial approach could be more reliable

Credit:

Getty Images

This article was published in Scientific American’s former blog network and reflects the views of the author, not necessarily those of Scientific American


It is now widely acknowledged by the scientific community that science is suffering from a reproducibility crisis. An article in Scientific American,Science under Scrutiny: The Problem of Reproducibility

, describes the issues well.

These problems are not just academic. The inability of scientists to recreate one anothers' experiments and reproduce results delays drug development, increases demands on resources and drives up research costs. In 2015 it is estimated that the amount spent on irreproducible preclinical research in the U.S. was $28 billion.


On supporting science journalism

If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.


Although given that we are all taught to do science a certain way from school onward, should we be surprised experimental findings are difficult to reproduce?

Taking an engineering approach to biology could improve reproducibility

Do you remember being taught the scientific method in middle school and what it meant to perform a fair test? We were all taught to alter only one thing at a time so you could see what effect each factor had, in turn.

This reductionist, one-factor-at-a-time (OFAT) approach is meant to ensure we can distinguish what each factor is doing and exclude interfering results coming from other changes in the experimental conditions. The implication, however, that OFAT is the only way to do science is wrong. Using carefully designed experiments it is possible to look at many factors simultaneously.

Working in this multifactorial way is particularly important for biology, where organisms have evolved over perhaps billions of years. Evolution has no imperative to make simple, easily understood systems. Every living thing is the result of huge numbers of component parts, interacting in complex, often unpredictable ways with one another and their environment.

Thus, to understand biological systems enough to be confident in any given experimental conclusion, we must observe not just a particular effect but understand how that might change as the context of the experiment changes. For example, a change of lab, a difference in which biological strain is used, or how that strain has been stored or grown prior to the experiment could have a dramatic effect on its results.

Multifactorial experiments are key to optimizing scientific investigations

Think of trying to understand how cells make a product, such as a therapeutic antibody, for example. We need to think not just about how we might use genetic engineering to promote production, but also what environment we are going to grow those cells in. What should we feed the cells? What pH should we grow them at? What temperature? And what about when the best temperature depends on the pH, or for that matter how you genetically engineered the cells in the first place?

Understanding holistically how all the factors combine to influence our cells using a multifactorial experimental approach provides much greater insight.

This multifactorial "design-of-experiment" (DoE) approach is not new. It was initially developed in the 1920s by statistician Ronald A. Fisher to help optimize crop growth but has rarely been used for biological investigations since then. It is telling, however, that the Food and Drug Administration asks for multifactorial investigations of pharmaceutical production processes to make sure they are consistent when lives rely on reproducible biology, then we insist that scientists use methods that provide true, robust understanding.

Multifactorial experiments could represent a step change in reproducibility

It is a testament to the genius and capability of biologists that we have come so far in our understanding of biology with the OFAT approach. But imagine how much farther we could go by approaching biology from an engineer's perspective of performing multifactorial experiments.

As the next generation of therapies are developed, such as gene and cell therapies, multifactorial DoE experiments will help scientists to track and understand the multitude of factors that impact on the biological processes used to generate the therapeutics. Once they understand the processes holistically in this way, they can systematically optimize them to deliver lifesaving therapies more reproducibly, in less time and at less cost.