Thursday, January 3, 2013

You're doing it backwards

Thoughtful Chipmunk
Here I'm going to try to describe how many people are "doing it backwards" when starting with a conclusion and then looking for evidence to support it. It'll be the start to a video script, probably.

I experimented with using my iPad + stylus to draw visual aides, but all I can say is that iPad is no Centiq. I've tried drawing on paper and scanning it, drawing on whiteboards and photographing it, or just drawing directly in Photoshop with shapes and a mouse. Nothing seems to be ideal.

Anyway, here's my explanation ...

Visual Aide - Starting with conclusion or starting with evidence
Two basic directions exist for supporting a claim - following the evidence and cherry picking evidence.

In terms of accuracy, following the evidence is far more reliable than cherry picking. To understand why this is the case, we'll need to look at some basic mechanics of evidence.

Visual Aide - Implicated possibilities of a piece of evidence
Each piece of evidence can implicate several possibilities. Sometimes, these implicated possibilities aren't currently known. Torn open trash bags that were outside could implicate a bear. It could also implicate a raccoon, or even a identity thief trying to find personal information on documents you've thrown out.

Take special note of the fact that a possibility can be implicated, even if it turns out the possibility wasn't ultimately responsible for generating the evidence.

This may seem like a trivial point, but it's one of the most Important mechanics of what can derail an investigation.

If one is starting with a conclusion that one is trying to scour reality for supporting evidence, it's possible to build a case for the claim that is constructed entirely from falsely implicated evidence, as mentioned before. On the surface, the case would appear to be significantly supported by positive evidence, and it would take a considerable amount of critical thinking and review to reveal that the case is based on false premises.

If the structure of cherry picked cases isn't sounding familiar yet, we do have a name of one kind of application of the approach - "conspiracy theories". It's common to find people making bizarre claims and backing them up with a large amount of disparate facts.

Further, multiple different and even opposing claims can be supported by large amounts of the same evidence. Realizing this, it should be abundantly clear that this approach is incredibly dubious in regards to accuracy.

Let's now look at the approach of following the evidence to a conclusion.

Visual Aide - one possibility is right and the others are wrongThe "follower", who isn't starting with a conclusion, finds a piece of evidence. Like before, this evidence can implicate several possibilities. In a sense, it's like a fork in the road. When one comes to this fork, it would be silly to arbitrarily choose one over the others without good cause.

What does the follower do? He or she gathers further evidence to cross reference and isolate which of the possibilities is most likely true - like intersecting Venn diagrams to narrow down the possibilities.

Since the goal is to figure out what is actually true, the evidence follower invests a significant amount of time on error mitigation and validation/confirmation.

These behaviors are significantly rare when cherry picking evidence, and often, the person making the case is unaware of refutations, contradictions or inconsistencies - those things which tend to be better revealed by evaluating multiple lines of evidence that one does when following the evidence, even if the evidence evaluated didn't appear to originally support an argument.

Ultimately, it's not impossible to successfully and correctly make a case by retroactively accumulating evidence that supports it, but it is precariously more difficult than starting over from scratch, and honestly trying to evaluate the clues that exist in an accurate and comprehensive manner.

No comments:

Post a Comment