I'm writing this post for two reasons. One is to recommend a new book by Columbia astrobiologist Caleb Scharf (who also writes a terrific Scientific American blog, "Life, Unbounded"), and the other is to defend an old book of mine.

Scharf's book is The Copernicus Complex: Our Cosmic Significance in a Universe of Planets and Probabilities. I loved Scharf's book, and so I brought him to my school, Stevens Institute of Technology, to talk about it this week. Here's how I introduced him:

"The Copernicus Complex addresses some of the deepest questions humans have ever asked. How weird are we? Was our existence highly probable, or improbable? Even miraculous? You can break this question down into more specific questions: How probable was our universe? Our galaxy? Solar system? Planet? How probable was life? And how probable were creatures like us, who can ponder their probability? For thousands of years, anyone could speculate about our weirdness, from Plato to pot-smoking college kids, because no one had any answers. It was just philosophy, pure guesswork. But new scientific discoveries are informing and guiding estimates of our probability. This is the story that Caleb Scharf tells brilliantly in The Copernicus Complex."

"Pure guesswork" was a gratuitous jab at my Stevens philosophy buddies. Otherwise, I meant everything I said in my introduction, especially the last sentence. Scharf provides an exceptionally clear, entertaining, up-to-date report on observations and conjectures bearing on what I call our "weirdness."

Scharf delves into exoplanets (4,000 discovered so far), and also microbiomes, Bayesian inference, planetary dynamics, multiverses, the anthropic principle, the Fermi Paradox, the RNA world theory and lots of other fascinating topics. All this work is provoking reconsideration of the Copernicus Principle, which holds, as Scharf puts it, that "we are not the center of all existence; we are not 'special.'"

Now I come to the second reason for writing this post. In The Guardian, journalist Tim Radford praises Copernicus Complex as "an intoxicating collection of questions answered with other questions. A couple of decades ago, physicists spoke confidently of a theory of everything, and one or two even proposed an end to science. All has now changed. The mysteries have multiplied."

I don't know any physicists who proposed "an end to science" two decades ago, but I did. In my 1996 book The End of Science, I argued that "pure" science, which I defined as the "quest to understand the universe and our place in it," may be drawing to a close. "Further research may yield no more revelations or revolutions but only diminishing returns."

I addressed the same questions that Scharf does in Copernicus Complex. I examined physicists' efforts to explain how our universe came to be, and why it took a form that allowed us to exist. I also looked at attempts by "chaoplexologists," notably Stuart Kauffman (also mentioned by Scharf), to estimate the probability of life, including intelligent, multicellular creatures like us. (I coined the term "chaoplexity" to describe research into chaos and complexity, which are virtually indistinguishable.)

I did not anticipate the thrilling recent surge in discoveries of exoplanets, which have revealed that planetary systems are quite common. Those observations, as I said when introducing Scharf, are "informing and guiding"—that is, enriching--discussions about our probability. But answers to the biggest questions remain as elusive as ever. Scientists still don't have a clue why our universe has the form we observe, or how life began on the Earth some 3.6 billion years ago, or whether life exists elsewhere. During his talk at Stevens, Scharf acknowledged that we may never observe exoplanets in sufficient detail to know, with certainty, that they harbor life.

In the "Afterword" of the paperback version of The End of Science, I noted that scientists, lacking data that can fully resolve questions about our probability, choose answers for philosophical and even emotional reasons. I wrote:

"Unfortunately, you cannot determine the probability of the universe or of life on Earth when you have only one universe and one history of life to contemplate. Statistics require more than one data point. The utter lack of empirical data does not stop scientists and philosophers from holding strong opinions on these matters. On one side are inevitabilists, who take comfort in theories portraying reality as the highly probable and even necessary outcome of immutable laws. Most scientists are inevitabilists. Perhaps the most prominent was Einstein, who rejected quantum mechanics because it implied that God 'plays dice with the universe.' On the other side are the anti-inevitabilists--notably Karl Popper, Stephen Jay Gould and Ilya Prigogine--who see scientific determinism as a threat to human freedom. We are either pawns of destiny or wildly improbable flukes. Take your pick."

I hope this situation changes, but I fear it won't, in spite of continued research in cosmology, astrobiology, chaoplexity and other fields. Scharf writes toward the end of his book: "So are we unusual or not?... Neither side is yet a winner. But we are much, much closer to an answer than we have ever been in the history of the human species; we are on the cusp of knowing."

We may be on the cusp of knowing, and yet still infinitely far away.

Postscript: For Caleb Scharf's response to this post, see "The Cusp of Knowing and the Evolution of Science."