competitive strategy, quality improvement, statistical methods, evaluation research, philosophy of science, critical thinking
Print Friendly and PDF

Anecdotal Evidence

This post was written by John on September 29, 2005
Posted Under: Statistical Thinking
pf button both Anecdotal Evidence

As I had mentioned in an earlier post, no one knew the real extent of looting and violence in New Orleans. All we had as a basis was what is called ‘anecdotal evidence’; that is, stories. We are now beginning to see a more accurate picture as actual data is being compiled. Fear Exceeded Crime’s Reality in New Orleans is the headline of a New York Times article that begins this exploration. Contrary to what was widely reported at the time, there were no registered complaints of murders or rapes and much of the store looting was caused by people trying to survive by taking food, water and other necessities because none was available elsewhere.

From a ‘statistical thinking’ point of view it raises a couple of interesting questions. What is the value of anecdotal evidence? To what extent can it be relied on to support any hypothesis? What are the likely traps one can fall into in using stories as evidence of facts and what is the cost of falling into those traps?

The testimony of eye witnesses is notoriously unreliable. This linked article addresses this subject. Also most of us are familiar with the parlor game where a story is repeated from person to person and by the fourth or fifth person, bears little resemblance to the original. Memory is unreliable. These are the threads from which the fabric of anecdotal evidence is weaved.

Memories themselves can often be misinterpreted. The 2000 movie Memento is a fascinating tale fueled by this (and other) paradox(es). The star tattoos ‘facts’ onto his leg so he won’t forget them (He’s unable to form new memories), but we see as the movie unfolds that these are not facts at all. They are stories that he is told that he believes to be true. He is relying on ‘face value’ for the reliability of his interpretation. This is not enough; at least for the purpose of science.

Data can not be interpreted apart from it’s context. Usually to understand a given figure it must be seen in light of the variation of the system from which it comes. With a sample of one (a person’s story), there is no variation and thus understanding (in the scientific sense) is almost impossible. That not does imply that these stories have no meaning, but rather that their scientific usefulness is quite limited.

It is interesting to note in this vein that much if not most news reporting consists of stories. They are interesting, entertaining and sometimes even titillating, but from the point of view of establishing useful hypotheses about the way of things, they are very limited. More samples are needed.

Sometimes additional sampling is not possible. That’s why things that only occur sporadically are hard to predict (eartchquakes for example). We can construct hypotheses about the causes for the onset of earthquakes and attempt to use these ideas to predict earthquake occurrences, but our opportunities to test these hypotheses are infrequent.

As to cost, it would be hard to estimate what the damage – monetarily and socially – of these over estimates was. Police were re-deployed who might have saved lives had they been able to continue with search and rescue operations. City officials were castigated for failing to maintain order. The results of the exageration even played into the politics of blame that followed the disaster.

Failure to understand the nature of knowledge produced by anecdotal evidence in this instance was a serious shortcoming.

Add a Comment

required, use real name
required, will not be published
optional, your blog address

Previose Post: