More on Samples of One

In the previous entry I discussed the hazards of samples of one. Later I began reading news articles from a variety of sources and also some blogs and they often involved an individual account of one kind or other. This type of evidence is call ‘anecdotal evidence in research circles and it almost invariably involves the use of a sample of one. That is, it is typically one person’s account experience in the subject area of the article. Newspaper accounts routine use this perspective.

But since, as we have discussed, samples of one can be extremely misleading, shouldn’t newspaper accounts using them be disregarded? Well, to be the real statistically minded person I am, I must say, “It depends”.

The question itself poses a larger issue, however and when considered in full, much of what is reported in news accounts may be distorted or contain (or lead to) totally unwarranted conclusions. Reporters, based on my sample of two, are reluctant to change. The first person account has literary merit and is a good way to lead into an article from the point of view of grabbing the readers’ attention.

The next time you read a news article on the internet, in the paper or in a magazine note the extent to which the conclusions of the article or its summary are based on stories of individuals or small groups. Ask yourself if a different group or person had been selected, the extent to which the conclusions might be different. Sure people get mugged in big cities and their stories are compelling, but the fact is, that even in the most dangerous cities, most people don’t get mugged.

In the same way that no two widgets produced by the widget machine are the same, no two experiences are the same among individuals. Different people experiencing the same event can have vastly different reactions to it. In fact, they are not even experiencing the same event. Of what use then is an eyewitness account? Such person accounts hinder generalization because they cannot, in themselves, show the range of experience that many individuals do have. In that way they are misleading.

Another problem with relying on first person accounts as evidence upon which to base a conclusion is that they are very unreliable. They are based not only on interpretation of an event at some moment in time, but as they are related to the reporter they are a story that is a memory of that event. Like all stories these accounts change over time and may eventually stray far from the actual event as it unfolded.

Advertising often uses anecdotal evidence. After all, that’s really what a testimonial is. Assuming that the spokesperson has even used the product (which is sometimes pretty clearly not the case), it is still a sample of one person’s opinion. Interesting, but no basis at all for generalization.

Please don’t misunderstand. Opinions are great. But one person’s opinion is only a good indicator of what happened if there was no variation and, if we know nothing else, we know that life is nothing but variation.
For example a new reporter writes the story of a family’s experience as they struggle through economic hard times. The story may be compelling. Indeed a good writer’s job is to make it so. But, what does the story of that family tell use about what families across America are experiencing in their economic struggles? Not very much.

The breadth of experience of citizens as they struggle economically is enormous. No two familial situations will be the same and the range of experiences can be huge. The family cited in the news account only gives us one data point. To truly understand the economic picture in the country we need dozens of data points, perhaps hundreds. Maybe they should be stratified by region or industry and so on.

Samples of one are not always labeled as such, but they are everywhere. The message of statistical thinking is to be extremely leery of drawing conclusions or making generalizations based on such a small amount of evidence.