The Baloney Detection Kit: How to draw boundaries between science and pseudoscience
From The Borderlands of Science: Where Sense Meets Nonsense (2001, Oxford University Press) by Michael Shermer

When exploring the borderlands of science we often face a "boundary problem" -- where to draw the line between science and pseudoscience? The boundary is the line of demarcation between geographies of knowledge, the border defining countries of claims. The problem with this geo-political analogy is that where rivers and mountain ranges, and deserts and seas help geographers and politicians demarcate (however artificially) the boundaries between countries, knowledge sets are fuzzier and the lines between them are blurry. It is not always clear where to draw the line. In deference to Carl Sagan, who coined the phrase "Baloney Detection Kit," I suggest ten questions to ask when encountering any claim, that can help us solve the boundary problem:

1. How reliable is the source of the claim? As Daniel Kevles showed so effectively in his book The Baltimore Affair, in investigating possible scientific fraud there is a boundary problem in detecting a fraudulent signal within the background noise of mistakes and sloppiness that is a normal part of the scientific process. The investigation of research notes in a laboratory affiliated with Nobel laureate David Baltimore by an independent committee established by Congress to investigate potential fraud, revealed a surprising number of mistakes. But science is messier than most people realize. Baltimore was exonerated when it became clear that there was no purposeful data manipulation.
   
   
2. Does this source often make similar claims? Pseudoscientists have a habit of going well beyond the facts, so when individuals makes numerous extraordinary claims they may be more than just iconoclasts. This is a matter of quantitative scaling, since some great thinkers often go beyond the data in their creative speculations. Cornell's Thomas Gold is notorious for his radical ideas, but he has been right often enough that other scientists listen to what he has to say. Gold proposes, for example, that oil is not a fossil fuel at all, but the by-product of a deep hot biosphere. Hardly any earth scientists I have spoken with take this thesis seriously, yet they do not consider Gold a crank. What we are looking for here is a pattern of fringe thinking that consistently ignores or distorts data.
   
   
3. Have the claims been verified by another source? Typically pseudoscientists will make statements that are unverified, or verified by a source within their own belief circle. We must ask who is checking the claims, and even who is checking the checkers? The biggest problem with the cold fusion debacle, for example, was not that Stanley Pons and Martin Fleischman were wrong; it was that they announced their spectacular discovery before it was verified by other laboratories (at a press conference no less), and, worse, when cold fusion was not replicated they continued to cling to their claim.
   
   
4. How does the claim fit with what we know about how the world works? An extraordinary claim must be placed into a larger context to see how it fits. When people claim that the pyramids and the sphinx were built over 10,000 years ago by an advanced race of humans, they are not presenting any context for that earlier civilization. Where are the rest of the artifacts of those people? Where are their works of art, their weapons, their clothing, their tools, their trash? This is simply not how archaeology works.
   
   
5. Has anyone gone out of the way to disprove the claim, or has only confirmatory evidence been sought? This is the confirmation bias, or the tendency to seek confirmatory evidence and reject or ignore disconfirmatory evidence. The confirmation bias is powerful and pervasive and is almost impossible for any of us to avoid. It is why the methods of science that emphasize checking and rechecking, verification and replication, and especially attempts to falsify a claim, are so critical.
   
   
6. Does the preponderance of evidence converge to the claimant's conclusion, or a different one? The theory of evolution, for example, is proven through a convergence of evidence from a number of independent lines of inquiry. No one fossil, no one piece of biological or paleontological evidence has "evolution" written on it; instead there is a convergence of evidence from tens of thousands of evidentiary bits that adds up to a story of the evolution of life. Creationists conveniently ignore this convergence, focusing instead on trivial anomalies or currently unexplained phenomena in the history of life.
   
   
7. Is the claimant employing the accepted rules of reason and tools of research, or have these been abandoned in favor of others that lead to the desired conclusion? UFOlogists suffer this fallacy in their continued focus on a handful of unexplained atmospheric anomalies and visual misperceptions by eyewitnesses, while conveniently ignoring the fact that the vast majority (90 to 95 percent) of UFO citings are fully explicable with prosaic answers.
   
   
8. Has the claimant provided a different explanation for the observed phenomena, or is it strictly a process of denying the existing explanation? This is a classic debate strategy -- criticize your opponent and never affirm what you believe in order to avoid criticism. But this stratagem is unacceptable in science. Big Bang skeptics, for example, ignore the convergence of evidence of this cosmological model, focus on the few flaws in the accepted model, and have yet to offer a viable cosmological alternative that carriers a preponderance of evidence in favor of it.
   
   
9. If the claimant has proffered a new explanation, does it account for as many phenomena as the old explanation? The HIV-AIDS skeptics argue that lifestyle, not HIV, causes AIDS. Yet, to make this argument they must ignore the convergence of evidence in support of HIV as the causal vector in AIDS, and simultaneously ignore such blatant evidence as the significant correlation between the rise in AIDS among hemophiliacs shortly after HIV was inadvertently introduced into the blood supply. On top of this, their alternative theory does not explain nearly as much of the data as the HIV theory.
   
   
10. Do the claimants' personal beliefs and biases drive the conclusions, or vice versa? All scientists hold social, political, and ideological beliefs that could potentially slant their interpretations of the data, but how do those biases and beliefs affect their research? At some point, usually during the peer-review system, such biases and beliefs are rooted out, or the paper or book is rejected for publication. This is why one should not work in an intellectual vacuum. If you don't catch the biases in your research, someone else will.

###

Michael Shermer is the founding Publisher of Skeptic magazine (www.skeptic.com) and the author of How We Believe and The Borderlands of Science.

###

[ View Next Perspective ]

[ View Previous Perspective ]