Friday, May 11, 2012

Introduction to a Layman's Standards of Evidence


Introduction to a Layman's Standards of Evidence


One of the difficulties of discussing issues of science and theology is that few people have a grasp of the standards of evidence. Even science aficionados have a high rate of ignorance.

The official standards of evidence themselves don't help. If one delves into the depths of the interwebs to research this topic, one often unveils high level mathematical constructs. There's few simple set of rules that the average human could understand.

I endeavor to solve this problem, by proposing a layman's standard. Of course, different branches of science and categories have their own different standards, however, a generic one couldn't hurt.

Call it a rule of thumb - a guide.  Read and understand, if nothing else. For each standard, I will try my best to explain why it's important.

Quality of Evidence and the Scientific Case

Science is a 1000 piece jigsaw puzzle.  Each puzzle piece is a piece of evidence. Some pieces are more useful than others - you know of what I speak. The edge pieces are the first one goes for. Then those with hard lines and patterns that can easily be paired up. Over time, the stupid single-color pieces are eventually filled in.

In short, some pieces of evidence are more valuable than others.

To make a case for a claim scientifically, one needs to provide sufficient evidence (so the picture can be revealed). The more extraordinary the claim, the more evidence is needed. 

Are we all set with the puzzle analogy? Good, we're switching to another one.

Evidence is like money, and demonstrating a claim is the cost.  The better the quality of the evidence, the more "money" it's worth, and the easier it'll be to make your case. In order to demonstrate your claim, you might need $100 of evidence money, but that required amount depends on the claim.
  • If you claim you have a pencil, that claim may cost $0.05 evidence dollars.
  • If you claim you have a dog, that claim may cost $1 evidence dollars.
  • If you claim you have a car, that claim may cost $100 evidence dollars.
  • If you claim you have a spaceship, that claim may cost $100,000 evidence dollars.
  • If you claim you have an interstellar spaceship from a parallel dimension, that claim may cost $100,000,000,000,000 evidence dollars.
The more extraordinary the claim, the more of a burden you have to convince others - the more it'll take for them to buy it. What makes a claim extraordinary, you ask? Just think about how frequently it's happened in the past. Lots of people own pencils. Not everyone owns a car. Very very few people own a spaceship. We haven't yet had an interstellar spaceship from a parallel dimension.

In short, the more unprecedented the phenomenon, the harder it is to "prove".

Standards of Evidence

The next question is, "So, interweb dude, what determines the worth of a piece of evidence?"

I'll explain.

The following attributes are necessary to produce high quality evidence. I would argue that few of the attributes could be failed without the evidence being essentially useless.

Let us begin.

Logical Connection

The evidence must have some logical connection to the claim. It must hypothetically support the case. For example, a bear footprint would support the claim that a bear is nearby, whereas, a trampoline wouldn't.

That one is fairly straight forward. Most people don't have an issue with this attribute.

Testability / Repeatability

The evidence must be testable and repeatable - mostly to help ensure that it wasn't just a fluke. A lot of data gathering has flawed and noisy data, so a statistical analysis of a large sample set is typically required.

Something that can be repeated helps cut through the noise, not to mention confirming that the asserted evidence exists at all.


The evidence must be presentable, as in, the data should be deliverable to someone else to check. One could merely claim to have the evidence at home, but forgot to bring it in.  In order to be evaluated and confirmed, the evidence must be presentable.

Simply telling a person about the evidence is insufficient, unless it's a reference to where the evidence can be attained - though it must be actually be attainable.


The evidence must not be mind-based, but based objectively in the real world somehow.

I am not solipsistic, however, the mind/brain suffers from a wide range of psychological issues, from cognitive biases, memory errors, perceptual errors, priming, etc. That's not to say that we're incapable of functioning, but it does cast a shadow on the credibility of claims that we make. People are wrong a lot.

How do we fix this? As much as we can, we try to rely on objective evidence - that which exists outside of the mind. My car isn't a figment of my imagination - it's a real thing - something we can all examine and agree upon. The existence of my car isn't subject to memory errors or confirmation bias, for instance. It exists regardless of what I think about it.

Ultimately, we are capable of handling subjectivity, reducing its error to a manageable amount, but the more we can keep the evidence objective, the higher quality the evidence will be.

Here's some examples of subjective claims that we can't do anything with:
  • I know there's a God because I felt his presence.
  • I can tell that this bacteria is designed, therefore it is, therefore God.
  • God talks to me in my mind.
Evidence needs to be more presentable and available than this. It's difficult to distinguish these examples from delusion, hallucination or lying - which means the evidence is practically worthless to anyone else.


The evidence must be falsifiable. Meaning, there must be a hypothetical way that the evidence can be disproven.

This one is a bit difficult to explain why it's important.

Time management - we have a lot of data to get through. Like a human resources department going through job applicants - they need an effective and quick way of sorting potential candidates into "rejected" and "investigate further". If the candidate is "unfalsifiable", it can't be placed into "rejected" even if it can't really be placed into "investigate further". 

Some evidence can appear to support a claim, while being false, but without falsifiability, one can't tell. In fact, many crackpot artists specifically create claims so they can't be falsified - because these claims can also persist. Can you really fully discount evidence that hasn't been disproven?

Here's some examples of unfalsifiable claims:
  • The power of prayer - tends to be indistinguishable from prayer not working, and whenever we try to test it, proponents of prayer will tell us that it can't be.
  • God talks to me in my mind - like most subjective evidence, we can't falsify it. How would we? For all we know, God is actually speaking to a person, circumventing all known forms of communication or interaction.
As a basic rule of thumb, one good way to tell whether something is unfalsifiable is to ask the question, "Could we tell the difference between the claim being true and the claim being false?". 

For the power of prayer, for instance, the resulting events with and without prayer are often indistinguishable. People frequently pray for things that would be likely to come true anyway. When the prayer doesn't work, that's because God works in mysterious ways.

Each possible outcome is designed so that prayer always wins. The prayer model literally cannot fail. In addition, you'd get the same results if you prayed to something other than God, like a jug of milk. I've embedded that video below because I think it makes the case fairly well, if you're interested.

Falsifiability is fairly important to establishing good quality evidence, because it's capacity to be disqualified is a strength in the bigger picture of science.

Principle of Exclusion

The principle of exclusion is sort of a meta analysis of the evidence. In short, if the evidence implicates many possibilities, the evidence isn't very strong in your favor. If that's the case, you'll need additional evidence to exclude other possibilities.

As an example, let's say someone stole your mailbox, and you are examining the crime scene to figure out who did it.

If you find an average shoe print in the ground, that piece of evidence implicates potentially thousands and thousands of people. That's not very useful.

Alternatively, someone has a photo of the person's vehicle who stole the mailbox. The photo shows a red Chevy Silverado with white stripes, Mitt Romney election bumper sticker, and a Maine liscence plate. This very much narrows the possibilities, and is much more useful.

The pinnacle of this principle would be if the possibilities are narrowed down to one, within reason. Let's say that photo captures not only the face of the person, but the licence plate number. This is very useful evidence.

I say "within reason", because it's possible the photo is a forgery, and someone is trying to frame this person, but that starts to get a bit outlandish. We have to stay practical about this analysis.


Just like an employer has standards to qualify/disqualify potential employees in order to more effectively find the best available candidate, we have standards in which we determine whether a piece of evidence is valuable.

As one would save up for a car, one has to collect enough evidence to demonstrate a claim beyond a reasonable doubt. The more expensive the car, the more evidence is required. It's easier to present 200 $100 bills than it would be to present 2,000,000 pennies, so one tries to get fewer better pieces, than more of lesser.

Unfortunately, $20,000 bills aren't very common.

Potential Rebuttals and Issues

Firstly, these standards are operating on the assumption that there is actually a real world. Whether we live in the real world or The Matrix, we do seems to be experiencing "some kind" of reality where these standards appear to work fairly well - and that's ultimately why the standards are as they are - because they work.

Secondly, I can already hear the objections of theists. "Your standards necessarily exclude any evidence that could support a god, which is convenient if you don't want to believe in him."

It's a question of goals. As an overweight scientific skeptic atheist, my goal isn't to believe in any pre-selected thing. My goal is to investigate reality, figure out as much of it as possible, and ensure that I have as few false beliefs as possible.

That requires a tool - a good tool that has proven to be effective and efficient. I've described a part of that tool. I'm trying to answer the question, "What's true?", and I choose to use the best tool available for investigating reality - science.

I'm not trying to exclude God. I'm trying to have a good effective process for figuring out what is actually true. I didn't make up these God claims, and I'm sorry that theists have chosen to believe something that is indefensible.

"But what if God is true and your so-called science can't figure it out?", you ask. Then we're out of luck. The point is that there isn't an alternative method of knowing that works. God is basically unknowable because of this.

Show me an alternative method of epistemology, and show me that it works. Then we can have a conversation.