Can social science ever become as rigorous, as "hard," as, say, nuclear physics? I explored this question in a recent post, which I wrote in part as a response to The Physics of Wall Street: A Brief History of Predicting the Unpredictable, by James Owen Weatherall. I've known Jim since 2005, when I started working at Stevens Institute of Technology. Jim had just earned bachelors and masters degrees in physics and philosophy from Harvard, and he was taking graduate courses in mathematics and physics at Stevens. He helped me establish the Center for Science Writings at Stevens, which hosts talks by prominent science writers.

Jim and I shared many of the same obsessions. We argued endlessly about the limits of science, whether physicists will ever find a final theory, whether super strings are the real deal or just science fiction with equations. Jim looked younger than many of my undergraduate students, but he was so smart that he really gave my brain a workout. Jim eventually earned a doctorate in mathematics and physics from Stevens and a Ph.D. in philosophy from the University of California at Irvine, where he now teaches philosophy.

While working on his doctorates, Jim started writing about physics for Scientific American and other publications. His work attracted attention, and he signed a contract with a major publisher to write a book about physics and economics. The Physics of Wall Street is a thoroughly researched history of modern physics and finance, with lucid explanations of fractals, derivatives and other esoteric topics. It's also a great read, filled with compelling characters, high drama and provocative ideas. I recently emailed Jim questions related to his book. Here is our exchange:

Horgan: You've studied both string theory and derivatives. Which was tougher to understand?

Weatherall: String theory. No question. Derivatives can get complicated, but I don’t think there are any deep mysteries behind how the various contracts work. As I see it, the problems with derivatives stem from the fact that we price them using models that are based on assumptions and simplifications, and not everyone in the industry pays close enough attention to the details of those assumptions. On the other hand, I do think there are mathematical problems connected to the social sciences that are just as difficult as any that arise in physics. Didier Sornette, one of the physicists I interviewed for my book, told me that he was drawn to economics because the problems were so much more difficult than in physics! But these problems aren’t really connected to derivatives contracts.

Horgan: Speaking of string theory, has it yielded any ideas that might help mutual fund managers?

Weatherall: Not that I know of—though Jim Simons, one of the co-founders of Renaissance Technologies, whose Medallion Fund is the most successful hedge fund ever, made very important early contributions to string theory. So perhaps Medallion has drawn on some of his early work, though I doubt it. (And Simons isn’t very forthcoming about Medallion’s strategies!) That said, one of the ideas I talk about in the book, due to Eric Weinstein and Pia Malaney, is connected to high energy particle physics—specifically to Yang-Mills gauge theory, which is the basis for the Standard Model of Particle Physics. Weinstein and Malaney, and some others such as Lee Smolin from the Perimeter Institute, have explored how the notion of “path dependence” in Yang-Mills theory might be used to construct a better measure of how cost of living changes over time.

Horgan: If physics can help economics, can it also help other social sciences? Say, sociology and political science?

Weatherall: I think there are a lot of fascinating historical connections between physics and economics. For instance, the first American to win a Nobel Prize in economics, Paul Samuelson, was deeply influenced by the work of J. Willard Gibbs, a 19th century physicist who helped invent thermodynamics—and turn chemistry into a rigorous mathematical theory. Building on Gibbs, Samuelson used notions like “equilibrium” and “entropy” to explain economic phenomena. Meanwhile, the first Nobel laureate in economics, Jan Tinbergen, had a PhD in physics, and introduced the term “model” into economics, in analogy to its use in physics. But I don’t really think it’s right to say that physics can help economics, so much as to say that there has been a rich exchange of ideas between physics and economics over the last century, and that financial professionals could benefit from learning to think about the relationship between their mathematical models and the world in the way physicists are trained to. As for sociology and political science—I think that people like Nate Silver have recently shown us the power of predictive modeling of social and political phenomena. He’s not a physicist, of course, but I think he’s deeply sensitive to the sorts of issues about modeling assumptions that I argue are so important. (Of course, there’s a long history of mathematical sociology, political science, and psychology—Silver didn’t invent this!)

Horgan: Do you think a physicist—all other factors being equal--would make a good President?

Weatherall: If all other factors really are equal, and one person is a physicist, then yes I would prefer the physicist! But I think being president is a pretty tough job, and I don’t know of any physicists who have all the other qualities I would hope for, such as great leadership skills and a deep understanding of the legislative process.

Horgan: In Isaac Asimov's science fiction novel Foundation, a mathematician named Hari Seldon invents "psychohistory," a theory that accurately predicts the future of societies. Do you think psychohistory will ever be possible? I.e., can social science ever become as rigorous and predictive as, say, nuclear physics?

Weatherall: No. But frankly, I am skeptical about the idea of a "final theory" in physics, too, both because I think it may be beyond our reach and also because it isn’t clear to me that such a theory would be very useful to us for the purposes we care most about. A final theory of economics or of finance, or of sociology for that matter, seems even more far-fetched. Indeed, the reason we have had any success in modeling in the social sciences is that we aren’t trying to find the ultimate predictive model.

But there’s another issue that comes up in this question, concerning rigor. I think rigor is extremely important. But I don’t think that the difference between economics and nuclear physics comes down to rigor, at least not in the way I think you have in mind. If you read an economics textbook, you will see lots of mathematics, with axioms and theorems and fully rigorous proofs. You would never find that in a nuclear physics textbook. If anything economics is more rigorous than nuclear physics. But rigor isn’t what you need if you want to come up with useful solutions to the problems we care about. In fact, I think that some economists have been blinded by the rigor of their work: if the mathematics is right, the theories must be true. But the relationship between mathematical theories and the world is more complicated than that.

Horgan: Why are you so critical of Nassim "Black Swan" Taleb's view of financial modeling?

Weatherall: I sometimes wonder if, at the end of the day, Taleb and I disagree about anything (other than how to express ourselves). He is absolutely right about the importance of black swans—events that are completely unforeseeable, and which change everything when they occur—and of so-called “fat-tailed” probability distributions, which help us account for the likelihood of extreme events. But I think the considerations he raises, many of which I also discuss in The Physics of Wall Street, should make us cautious and modest in our attempts to understand complex systems such as financial markets. I do not think they show that we should give up on mathematical modeling altogether. No model is perfect, but surely thinking about how black swans can affect us will help us make our modeling better—not because we can ever account for every unforeseen possibility, but because the recognition that there are unforeseen possibilities can guide us in how to build extra caution into our practices.

Horgan: Of all the physicists who've delved into economics, who most impresses you?

Weatherall: This is a tough question. The people I wrote about in The Physics of Wall Street are uniformly brilliant and creative, but they are all remarkable for different reasons, which makes it difficult to compare them. On the one hand, there’s Ed Thorp, who proved mathematically that card counting can be used to beat blackjack, and then went on to start the first modern quantitative hedge fund. I think he has an utterly unique way of applying mathematical reasoning to the real world that I find very impressive. But then if you think of someone like Benoît Mandelbrot, who spent his whole career fighting uphill battles against the academic establishment as he tried to better account for the amazing complexity of the natural world (and financial markets), it’s hard not to be amazed. So I don’t have a simple answer.

Horgan: The New York Times reviewer of your book harrumphed that "the world’s economic problems are far too complex to be reduced to a matter of physics and mathematics." Your response?

Weatherall: Statements like this leave me speechless. I don’t understand how anyone could think they know in advance which problems are impossible, and which are merely very difficult. Mathematics is an extraordinarily powerful tool. To simply dismiss its applicability to any subject out of hand seems silly to me—all the more so if the dismissal comes at the end of an otherwise positive review of a book that gives detailed examples of how physicists and mathematicians have made concrete contributions to our understanding of economic problems. And of course, leaving physicists aside, there are lots of economists out there who are using mathematics to understand the world’s economic problems. If tools from mathematics and mathematical modeling can’t help us understand the world’s economic problems, what else do we need? Or is the idea that economics is simply beyond the ken of human understanding? If it’s the latter, then I guess I am just more optimistic, or at least, I don’t think we gain much by throwing in the towel.

Horgan: Do you think economics could still be transformed by some paradigm-shifting genius—a future Hari Seldon—or will progress be incremental?

Weatherall: I would say I know more about finance than about economics more generally, so what I say here should be taken with a grain of salt. But my sense is that economics is too varied a field for the paradigm language to apply very effectively: economics has long been characterized by competing “schools”, such as New Keynesianism, Post-Keynsianism, New Classicism, Austrian economics, etc. And, especially since these tend to have political associations, it is hard to imagine a new idea coming in and leading to a complete revolution. That said, there have been a few dramatic innovations in the last 60 or 70 years that have changed wide swaths of economics. One is game theory, which was developed in the 1940s and 1950s by mathematical physicist John von Neumann, economist Oskar Morgenstern, mathematician John Nash, and others. Game theory provided new mathematical tools for analyzing strategic scenarios, which proved extremely useful to economists. Another is the introduction of ideas and methods from the behavioral sciences, spearheaded by people such as psychologist Daniel Kahneman. This movement, known as “behavioral economics,” has successfully questioned many basic economic assumptions about rational action. I think developments such as these are the closest we will come to paradigm shifts in economics—and yes, I think they are still possible! Note, though, that these “revolutions” have both come from people trained in fields other than economics. This provides some evidence of the potential for outsiders to make significant progress in economic thinking.