Does anyone who follows physics doubt it is in trouble? When I say physics, I don’t mean applied physics, material science or what Murray-Gell-Mann called “squalid-state physics.” I mean physics at its grandest, the effort to figure out reality. Where did the universe come from? What is it made of? What laws govern its behavior? And how probable is the universe? Are we here through sheer luck, or was our existence somehow inevitable?

In the 1980s Stephen Hawking and other big shots claimed that physics was on the verge of a “final theory,” or “theory of everything,” that could answer these big questions and solve the riddle of reality. I became a science writer in part because I believed their claims, but by the early 1990s I had become a skeptic. The leading contender for a theory of everything held that all of nature’s particles and forces, including gravity, stem from infinitesimal, stringy particles wriggling in nine or more dimensions.

The problem is that no conceivable experiment can detect the strings or extra dimensions. In 1991, I raised the issue of testability with string theorist Edward Witten, whom some say is the greatest living physicist. He emphasized the “incredible consistency, remarkable elegance and beauty” of string theory. “Good wrong ideas are extremely scarce,” he assured me, “and good wrong ideas that even remotely rival the majesty of string theory have never been seen.”

Basically Witten was saying that string theory is too beautiful to be wrong. When I interviewed him in 2014, he said he was still confident that string theory is “on the right track,” even though it remains as lacking in evidence as ever.

Strings are only one of several popular physics concepts that can be neither experimentally verified nor falsified. Others include multiverse theories, which hold that our universe is only one of many, and the theory of cosmic creation called inflation. The persistence of such highly speculative theories, I argued in The End of Science, suggests that physics is crashing into insurmountable limits.

But what the hell do I know? I’m just a science journalist. For a far more authoritative and up-to-date critique of physics, check out the fascinating, painfully honest new book Lost in Math: How Beauty Leads Physics Astray, by Sabine Hossenfelder. Hossenfelder, who specializes in quantum gravity, became a physicist two decades ago because she believed that physics represents our best hope of understanding reality. With its combination of mathematical logic and empirical evidence, physics helps us overcome wishful thinking and other biases. Ideally.

Credit: Sabine Hossenfelder Wikimedia (CC BY-SA 4.0)

Lost in Math tells the story of Hossenfelder’s disillusionment, her realization that subjective factors, such as an obsession with beauty, have infected physics. Aesthetic considerations guide physicists’ judgments of strings, inflation, supersymmetry, multiverses and the many different interpretations of quantum mechanics.

Physicists seem to adhere to Keats’s old aphorism that truth equals beauty. In the absence of data, that principle reduces physics to a matter of taste, not truth. You like string theory and Bach, I prefer loop-space theory and the Beatles. “I’m not sure anymore that what we do here, in the foundations of physics, is science,” Hossenfelder writes. “And if not, why am I wasting my time with it?”

What sets Lost in Math apart from other books by physicists is that Hossenfelder, who writes a popular blog, “Backreaction,” acts like a reporter. In my favorite parts of her book, she visits other physicists and talks about their field’s problems. Some of her interviewees, who include the distinguished elders Steven Weinberg, Frank Wilczek and George Ellis and the renegade Garrett Lisi, are surprisingly candid. Wilczek worries that physics has become “post-empirical,” a situation that he calls “appalling, really appalling.”

Physicists’ seeming lack of concern with empirical evidence and testability, Ellis argues, is bad not only for physics but for science as whole. “[S]cience is having a difficult time out there,” he says, “with all the talk about vaccination, climate change, GMO crops, nuclear energy, and all of that demonstrating skepticism about science. Theoretical physics is supposed to be the bedrock, the hardest rock, of the sciences, showing how it can be completely trusted. And if we start loosening the requirements over there, I think the implications are very serious” for other fields.

Hossenfelder lists cognitive biases that, in addition to aesthetic preferences, have undermined physics. They include confirmation bias, the sunk-cost fallacy and the social desirability bias. Explaining the latter, Hossenfelder remarks, “You don’t tell the tribal chief your tent stinks if behind you stand a dozen fellows with spears.” Hossenfelder nonetheless bravely declares that “this tent stinks.”

Cognitive biases, of course, beset not just physics but science as a whole. They help to explain science’s replication crisis and diminishing returns. “Almost all scientists today have an undisclosed conflict of interest between funding and honesty,” Hossenfelder writes.

And yet, perhaps because she is subject to the sunk-cost fallacy, Hossenfelder cannot abandon the endeavor in which she has invested so much time. She ends her book with a burst of optimism.  She rejects the claim that physics “was the success story of the last century, but now is the century of neuroscience or bioengineering or artificial intelligence (depending on whom you ask).” She says, “I got a new research grant. There’s much work to do. The next breakthrough in physics will occur in this century. It will be beautiful.”

There is something almost unbearably poignant and, yes, beautiful about humans striving to accomplish a grand goal after repeated failures. But as Hossenfelder asks, “How long is too long to wait” for physicists to succeed? Good question. At what point, if ever, will physicists conclude that they cannot complete their quest, because the riddle of existence is unsolvable?

Further Reading:

Was I Wrong about The End of Science?

Is Science Hitting a Wall?

How Physics Lost Its Fizz

Stephen Hawking's "new" theory of everything is the same old CRAP

Is speculation in multiverses as immoral as speculation in subprime mortgages?

Is David Deutsch's Vision of Endless Understanding Delusional?

Is Science Infinite?

Meta-Post: Horgan Posts on Physics and Cosmology

See my 2016 Q&A with Hossenfelder as well as my interviews with other physicists: Scott AaronsonDavid DeutschGeorge EllisMarcelo Gleiser, Garrett LisiPriyamvada NatarajanMartin Rees, Carlo RovelliLee Smolin, Paul Steinhardt, Steven Weinberg, Edward Witten, Peter Woit and Stephen Wolfram