About the SA Blog Network

Doing Good Science

Doing Good Science

Building knowledge, training new scientists, sharing a world.
Doing Good Science Home

Evaluating scientific claims (or, do we have to take the scientist’s word for it?)

The views expressed are those of the author and are not necessarily those of Scientific American.

Email   PrintPrint

Recently, we’ve noted that a public composed mostly of non-scientists may find itself asked to trust scientists, in large part because members of that public are not usually in a position to make all their own scientific knowledge. This is not a problem unique to non-scientists, though — once scientists reach the end of the tether of their expertise, they end up having to approach the knowledge claims of scientists in other fields with some mixture of trust and skepticism. (It’s reasonable to ask what the right mixture of trust and skepticism would be in particular circumstances, but there’s not a handy formula with which to calculate this.)

Are we in a position where, outside our own narrow area of expertise, we either have to commit to agnosticism or take someone else’s word for things? If we’re not able to directly evaluate the data, does that mean we have no good way to evaluate the credibility of the scientist pointing to the data to make a claim?

This raises an interesting question for science journalism, not so much about what role it should play as what role it could play.

If only a trained scientist could evaluate the credibility of scientific claims (and then perhaps only in the particular scientific field in which one was trained), this might reduce science journalism to a mere matter of publishing press releases, or of reporting on scientists’ social events, sense of style, and the like. Alternatively, if the public looked to science journalists not just to communicate the knowledge claims various scientists are putting forward but also to do some evaluative work on our behalf — sorting out credible claims and credible scientists from the crowd — we might imagine that good science journalism demands extensive scientific training (and that we probably need a separate science reporter for each specialized area of science to be covered).

In an era where media outlets are more likely to cut the science desk than expand it, pinning our hopes on legions of science-Ph.D.-earning reporters on the science beat might be a bad idea.

I don’t think our prospects for evaluating scientific credibility are quite that bad.

Scientific knowledge is built on empirical data, and the details of the data (what sort of data is relevant to the question at hand, what kind of data can we actually collect, what techniques are better or worse for collecting the data, how we distinguish data from noise, etc.) can vary quite a lot in different scientific disciplines, and in different areas of research within those disciplines. However, there are commonalities in the basic patterns of reasoning that scientists in all fields use to compare their theories with their data. Some of these patterns of reasoning may be rather sophisticated, perhaps even non-intuitive. (I’m guessing certain kinds of probabilistic or statistical reasoning might fit this category.) But others will be the patterns of reasoning that get highlighted when “the scientific method” is taught.

In other words, even if I can’t evaluate someone else’s raw data to tell you directly what it means, I can evaluate the way that data is used to support or refute claims. I can recognize logical fallacies and distinguish them from instances of valid reasoning. Moreover, this is the kind of thing that a non-scientist who is good at critical thinking (whether a journalist or a member of the public consuming a news story) could evaluate as well.

One way to judge scientific credibility (or lack thereof) is to scope out the logical structure of the arguments a scientist is putting up for consideration. It is possible to judge whether arguments have the right kind of relationship to the empirical data without wallowing in that data oneself. Credible scientists can lay out:

  • Here’s my hypothesis.
  • Here’s what you’d expect to observe if the hypothesis is true. Here, on the other hand, is what you’d expect to observe if the hypothesis is false.
  • Here’s what we actually observed (and here are the steps we took to control the other variables).
  • Here’s what we can say (and with what degree of certainty) about the hypothesis in the light of these results.
  • Here’s the next study we’d like to do to be even more sure.

And, not only will the logical connections between the data and what is inferred from them look plausible to the science writer who is hip to the scientific method, but they ought to look plausible to other scientists — even to scientists who might prefer different hypotheses, or different experimental approaches. If what makes something good science is its epistemology — the process by which data are used to generate and/or support knowledge claims — then even scientists who may disagree with those knowledge claims should still be able to recognize the patterns of reasoning involved as properly scientific. This suggests a couple more things we might ask credible scientists to display:

  • Here are the results of which we’re aware (published and unpublished) that might undermine our findings.
  • Here’s how we have taken their criticisms (or implied criticisms) seriously in evaluating our own results.

If the patterns of reasoning are properly scientific, why wouldn’t all the scientists agree about the knowledge claims themselves? Perhaps they’re taking different sets of data into account, or they disagree about certain of the assumptions made in framing the question. The important thing to notice here is that scientists can disagree with each other about experimental results and scientific conclusions without thinking that the other guy is a bad scientist. The hope is that, in the fullness of time, more data and dialogue will resolve the disagreements. But good, smart, honest scientists can disagree.

This is not to say that there aren’t folks in lab coats whose thinking is sloppy. Indeed, catching sloppy thinking is the kind of thing you’d hope a good general understanding of science would help someone (like a scientific colleague, or a science journalist) to do. At that point, of course, it’s good to have backup — other scientists who can give you their read on the pattern of reasoning, for example. And, to the extent that a scientist — especially one talking “on the record” about the science (whether to a reporter or to other scientists or to scientifically literate members of the public) — displays sloppy thinking, that would tend to undermine his or her credibility.

There are other kinds of evaluation you can probably make of a scientist’s credibility without being an expert in his or her field. Examining a scientific paper to see if the sources cited make the claims that they are purported to make by the paper citing them is one way to assess credibility. Determining whether a scientist might be biased by an employer or a funding source may be harder. But there, I suspect many of the scientists themselves are aware of these concerns and will go the extra mile to establish their credibility by taking the possibility that they are seeing what they want to see very seriously and testing their hypotheses fairly stringently so they can answer possible objections.

It’s harder still to get a good read on the credibility of scientists who present evidence and interpretations with the right sort of logical structure but who have, in fact, fabricated or falsified that evidence. Being wary of results that seem too good to be true is probably a good strategy here. Also, once a scientist is caught in such misconduct, it’s entirely appropriate not to trust another word that comes from his or her mouth.

One of the things fans of science have tended to like is that it’s a route to knowledge that is, at least potentially, open to any of us. It draws on empirical data we can get at through our senses and on our powers of rational thinking. As it happens, the empirical data have gotten pretty complicated, and there’s usually a good bit of technology between the thing in the world we’re trying to observe and the sense organs we’re using to observe it. However, those powers of rational thinking are still at the center of how the scientific knowledge gets built. Those powers need careful cultivation, but to at least a first approximation they may be enough to help us tell the people doing good science from the cranks.

Janet D. Stemwedel About the Author: Janet D. Stemwedel is an Associate Professor of Philosophy at San José State University. Her explorations of ethics, scientific knowledge-building, and how they are intertwined are informed by her misspent scientific youth as a physical chemist. Follow on Twitter @docfreeride.

The views expressed are those of the author and are not necessarily those of Scientific American.

Rights & Permissions

Comments 20 Comments

Add Comment
  1. 1. lamorpa 3:24 pm 09/30/2011

    Sounds very much like a religion when you look at it from the ‘outside’.

    Link to this
  2. 2. jendavison 3:38 pm 09/30/2011

    So, you’re saying that the scientific method is a good tool to use in evaluating science. Nice!

    Link to this
  3. 3. ttfweb 4:32 pm 09/30/2011

    I have worked with scientist for a number of years, and the questions I now ask right from the beginning is
    - What is this person’s vested interest in a given outcome or finding?
    - Is this person open to critical review?
    - What are the underlying assumptions of the research?

    My general observation is that if someone has spent a good part of their career establishing a certain scientific premise, they struggle with being objective about contrary findings that may render them obsolete. They are people – I think anyone would be challenged here.

    I believe most scientist are working in good faith to further our body of knowledge, but it’s lunacy to not take into account they will interpret results (or lack of results) through their own personalities and agendas.

    Link to this
  4. 4. zsingerb 4:45 pm 09/30/2011

    I took logic in college and IAMORPA is right. You could find logical processes in evaluating religious “experiments” as well as evaluating science. This just means they were logical about coming to a wrong conclusion.

    Link to this
  5. 5. lamorpa 4:59 pm 09/30/2011

    The people I have known who have the most subjective judgement are those who have convinced themselves they are the most objective.

    Link to this
  6. 6. GregPfister 6:27 pm 09/30/2011

    “…this might reduce science journalism to a mere matter of publishing press releases, or of reporting on scientists’ social events, sense of style, and the like.”

    Yup, sounds like most science journalism to me. Regurgitation + social events, which are about the only things they understand, or think their readers understand.

    How much of this is lack of use of some scientific “methods” (seldom taught to journalists) vs. laziness, vs. assumptions about audience, vs. fear of lawsuits? I don’t know. Probably all of them.

    Link to this
  7. 7. Postulator 8:35 pm 09/30/2011

    Unfortunately, science seems increasingly to rely on groupthink in various disciplines. Once dissenters are stomped upon for the mere act of dissent, non-experts are likely to start asking questions.

    A couple of examples (including the obvious):

    1. Climate change science. “I’m a climate scientist, you’re not, shut up”. Guess what guys, you’re reliant on input from a range of other scientific schools. You also rely on data from within your own area of expertise, that you don’t always thoroughly review. The world isn’t black and white
    2. Einsteinian physics. Someone announces “It moved faster than light!” The rest of the community immediately responds “No it didn’t” before even taking the opportunity to check the facts. Get used to the idea that Einstein wasn’t absolutely right, and his theories will be overturned just as surely as he displaced Newton.

    One of the most important things science journalists can do (or more correctly, science publications) is to name peer reviewers. There must be some responsibility attached to peer review, and at the moment there is none.

    Another thing that should be done as part of scientific training is to teach inquiry. Not enough scientists seem prepared to challenge the groupthink. It’s unhealthy from the perspective of “how to win grants”, it’s unhealthy from a career perspective, but it’s incredibly important to the advancement of science.

    Link to this
  8. 8. priddseren 12:19 am 10/1/2011

    I think considering the past reputation of scientists or the groups of scientists affect their credibility as well. Take the so called Climatologists who insist human caused global warming exists, is human caused and the world is headed to total destruction and genocide as a result of this global warming. I myself am not a scientist but I am in a field that requires considerable analytical skills. I can easily see the climatologists are not following basic scientific principles. Their hypothesis is stated. Every experiment they do and every possible result is proof of their belief. None of their research is done with what the article states as here are the expectations if the hypothesis is true and here is what happens if it is false. As far as they are concerned it is true and any data or evidence to the contrary is dismissed. A good example is the recent speculation of deep ocean heating to explain why the global warmists computer model predictions of temperature increases for the last decade were not close to actual reality as we have observed. The possibility their theory is wrong or flawed, the concept that computer models of a little understood atmosphere might actually not work or their assumptions of average measures could possibly be off or not relevant. Examples like the so called “normal average global temperature” and the what the CO2 saturation should be might not be relevant. Then add in their complete dependence on computer models for their so called evidence and we have a group of discredited scientists. Yes it is possible the heat was there and is in fact in the deep ocean. It is also just as possible the heat never existed and all. These “scientists” are changing the model to ensure their theory continues and are matching it to reality. When they should toss the models and start using observed reality as the model and test for both ocean warming and no heat at all.  The result is anyone who claims to be a climatologist or claims to work in this is discredited by association.  Other examples of reasons why scientists can’t be trusted are when a group of them in a certain disciplines such as archeology or anthropology insist what they know is absolute. Examples would be the Clovis first theory in America. Regardless of all the evidence that exists demonstrating humans were in fact here before Clovis as well as multiple Asian and European contact during this time are all dismissed. By nature these disciplines can’t make definitive claims on any part of the past because the information available is limited, incomplete and not necessarily interpreted correctly. This does not mean every reasonable theory or accepted truth should be discarded every time a new theory appears but when evidence exists to the contrary of a theory it should be investigated.  Scientists who have maintained their reputations and credibility probably do not get as much scrutiny as the scientists in fields where it is very obvious bias is there. The bias might be their own beliefs and pet theories, getting grant money, advancing their own reputations but the end result is questioning their integrity.  Had today’s scientists performed actual science where we can confidently assume the experiments were done with the expectation of obtaining results that prove or disprove a theory, then their credibility would not be questioned so much.

    Link to this
  9. 9. priddseren 12:31 am 10/1/2011

    Another problem not mentioned really is politicians. When it is politicians who declare CO2 a pollutant with the backing of grant purchased scienists, it is hard to consider science credible. CO2 is the core molecule for carbon life on this planet. There is no possible way this molecule is a pollutant yet the politicians voted it that way at the behest of “scientists”.

    Is it really difficult to understand why so many non-scientists are willing to question the claims made by scientists? Sure many legitimate scientists are being painted by the same brush as the purchased scientists but it should be expected. Since most science is paid for by government and politicians are the least trusted people in the country.

    Link to this
  10. 10. priddseren 12:35 am 10/1/2011

    I agree with Postulator here, Maybe the nutrino is not moving faster than light but when an observation in the real world occurs to the contrary, only experiments to prove the results false and why or show the observation as true are legitimate. Declarations that Einstein is some sort of omnipotent being who declared all laws of physics and is never wrong, are simply not valid or relevant.

    Link to this
  11. 11. blindboy 3:07 am 10/1/2011

    Science journalists, if the concept has any meaning, should have a thorough grounding in basic science across whatever fields they intend to write about. For any area that involves epidemiology they should also have some serious qualifications in statistics. The worst science reporting, apart from that produced by the amateur climatologists so well represented here, almost always involves influences on human health, well being or performance. Particular care needs to be taken when reporting these areas as people are likely to change their behavior. Clearly distinguishing between correlation and causation is a good starting point!

    Link to this
  12. 12. witzelcheck 11:39 am 10/1/2011

    More than journalists, the government using biased scientific claims is in more desperate need of regulation. Right now, they tell us what to eat, and America’s health problems are on the rise. Myself and a growing number out there are discovering a different look on nutrition that keeps us out of the doctor’s office (Google Paleo/Primal), yet the government still uses the phrase “science suggests” and tells us to eat tons of grain and dairy. Yet Obesity, blood pressure, and chronic illnesses have SPIKED recently. When will THIS be fixed? I think it is the larger issue…

    Or maybe this IS the approach to fix it. After all, if we do stop a good amount of this journaling, the government will have nothing to fall back on when trying to support their own claims. I don’t know, maybe this will help that issue in its own way.

    When I did my final research paper for a class in this area, one of the only publications I could find that “suggested” whole grains were good was a huge study funded by General Mills (connect dots, follow money, etc.). This above article only touches the tip of the iceberg. When industries or even the government FUND the studies, even more questions should be directed towards the integrity of the scientists that are publishing their conclusions.

    Link to this
  13. 13. johnwerneken 11:40 am 10/1/2011

    Great article. Informative, accurate, useful. Not so the comments. Why does religeon or climate have to get brought up all the time? What one choses to believe about religeous choice or actions to address climate are not scientific choices at all lol…and not very important questions either, not compared to peace or to poverty…

    Link to this
  14. 14. michaell84 5:38 pm 10/1/2011

    Postulator, priddseren.

    Are you worried about purported scientist groupthink as it relates to any other conclusions? Or only when it threatens your own beliefs?

    As a scientist, the statement “not enough scientists seem prepared to challenge the groupthink” seems a long ways off (I invite you to produce additional evidence and arguments supporting this claim, if you have some.) I think if you replace “groupthink” with “consensus based on available evidence”, I think you’re part way there.

    Overturning scientific consensus is no small task, and requires a very strong case to be built (although it does happen from time to time.) Contrary to your assertion, the reward in doing so is tremendous. Great fame accompanies those who make big new discoveries which overturn longstanding results.

    So when we have a group claiming that they’ve observed particles moving faster than the speed of light (for example) we have two possibilities: they’ve made a mistake, or the numerous scientists working in this area before them have all made a mistake. Where do you place your bets?

    Link to this
  15. 15. LarryW 11:43 pm 10/1/2011

    I agree with John Werneken. Almost all the comments about scientists and science show profound scientific illiteracy. It is these type of people who will never, and have no capability of understanding science and evidence and how theories, inferences, and observations (experiment) are used to either confirm or reject or modify theories.

    A great example of such illiteracy is the criticism of scientists who out of hand, do not believe the faster-than-the-speed-of-light by neutrinos results. This position is both correct and absolutely the appropriate. For 100 years, Einsteins special theory of relativity has been supported by experiments, observations, to scientists initial surprise. The overturning of Newton’s theories and acceptance of Einsteins occurred over many decades. Why? Because, Newton’s theories had very accurately modeled how nature actually behaves. You don’t throw out a working model of nature based upon even several confirmations of a new theory.

    For decades, and as technology allowed, new inferences based on Einstein’s theory were generated and tested. These inferences were not contemplated by Einstein. After 100 years of confirmation, you don’t throw out Einstein’s now confirmed theory because of a single experiment. The experiment needs to be reviewed, it needs to be replicated.

    I’m certainly in no way competent to evaluate the results of this experiment, and I have no business having an opinion of these results — and I don’t. Neither does Werneken, but he does anyway.

    The key issue which is not addressed in the original article is the need of those who are not competent nor expert in the particular field to not have an opinion about anything they have no knowledge about. Instead, you might want to monitor all the experts in the field. In this particular case, I have read general comments of such experts and got a general idea of what the experts are thinking — but I can guarantee that I, and no one without the expertise, will possibly truly and deeply understand what is being said, nor the possible implications to Einstein’s theory if the result of this experiment is confirmed.

    So I’ve heard two explanations or modifications — which I think I can relate but not understand. There are others:

    1) Light travels a longer distance than the neutrinos because light travels in a geodesic (a straight line on a curved surface — space-time) not in a straight line, whereas neutrinos do travel in a straight line.
    2) Modify Einstein’s Special theory changing speed of light to speed of neutrinos as the constant.

    What are the implications of the above ideas or others if the neutrino experiment is confirmed? I don’t know, and neither does anyone else at this point. But it’s going to be the experts who run the experiments and develop the theories, and not any non-expert.

    Link to this
  16. 16. LarryW 11:48 pm 10/1/2011

    Correction. I misrepresented the position of Werneken when discussing the neutrino experiment. I should have referenced commenter Postulator.

    My apologies.

    Link to this
  17. 17. jbay 2:48 am 10/2/2011

    “Examining a scientific paper to see if the sources cited make the claims that they are purported to make by the paper citing them is one way to assess credibility.”

    This is absolutely true. Unfortunately, unless our non-scientist is lucky enough to be associated with a university, reading through the citations in just a single article will cost hundreds of dollars. Academic journals charge upwards of $30 just for temporary access to a single article, and a given paper may have a few dozen citations. The research is often funded by the taxpayer.

    Until science is affordable to the general public, we can’t expect anybody to educate themselves about science, to look critically at data, or to investigate the claims made by scientists.

    Link to this
  18. 18. blindboy 3:38 am 10/2/2011

    I absolutely agree with LarryW’s comment about opinions. The right to hold an opinion does not extend to all opinions having equal value. The best position for most of us on many scientific questions is to state the position he described as clearly as possible. “I don’t know enough about it to have an opinion.”

    Link to this
  19. 19. Postman1 5:19 pm 10/3/2011

    Blindboy, LarryW, Everyone has some opinion on everything, whether informed or not and whether voiced or not. The problem as I see it, is that many of today’s scientists and science journalists is the
    1) inability to put their theories into layman’s terms, and
    2) failure to accurately predict results of their theories, and
    3) failure to reveal beforehand, how their theory can be falsified.
    In case some scientists don’t see the importance behind an informed public, I would point out that, via politicians, the public controls the purse strings.

    Link to this
  20. 20. Postman1 6:02 pm 10/3/2011

    My second sentence in comment #19 should read “The problem, as I see it, with many of today’s scientists and science journalists, is their…”
    My 2 1/2 year old grandson is visiting. Need I say more?

    Link to this

Add a Comment
You must sign in or register as a member to submit a comment.

More from Scientific American

Email this Article