Tests of Hypothesis

When discussing hypothesis testing, I have, from time to time, used the example of the criminal justice system to act as a metaphor for the logic and philosophical issues involved. It remains a good way to pose the logical dilemmas and the types of errors involved and to also discuss the idea of a system generally and how to improve it.There is, in the American system of justice, a presumption of innocence. That is, the burden of proof that a crime was committed rests with the government (the prosecutor). In a research situation, the burden of proof rests with the advocate of the research hypothesis (e.g. a researcher may assert that this drug will work to cure this disease)Using our criminal justice example, a first hypothesis (in statistics usually called a null (no difference) hypothesis) would state that the person who has been apprehended is innocent. This hypothesis can only be rejected if sufficient evidence of guilt is produced. In the research example, the null hypothesis (as it is called) would be that the drug does not work to cure the disease.The second (alternate) hypothesis is that the individual is guilty. The government takes this position, that’s why they arrested the individual. The question is can the prosecutor produce enough evidence of guilt to establish guilt. The burden, as we have said, rests with the prosecutor. In the drug example, the second hypothesis is sometimes called the research hypothesis and in our drug case, it would be that the drug does cure the disease.There are two mistakes we can make. We can let a guilty person go free. Or, alternatively we can call a person guilty who is, in fact, innocent. Or, we can say the drug works when it doesn’t, or we can fail to detect that the drug does work and conclude that it doesn’t.Usually in statistics those are called Type I and Type II errors. Or, ‘Errors of the First Kind and Errors of the Second Kind. To be clear, Errors of the First Kind are to mistakenly reject the no difference (null) hypothesis. Errors of the Second Kind would be to failing to reject the no difference hypothesis when there was actually a difference (the person is guilty)So, in the criminal justice case, we have a trial. In the research case, we do a research study.The innocence or guilt of the person at trial is not known. We do not know for certain if the drug works. That is the key. If we knew for certain, we wouldn’t need a trial.And because of that uncertainty we will make those mistakes.Statistics has been called a tool for “…making decisions in the light of uncertainty…” If we had a sure fire way to know the innocence or guilt of the individual, we wouldn’t need the trial or study. But we are uncertain.So we set a standard of proof. A pre-selected point or criterion that, when met, will be sufficient to say we will make a decide this way or that. In the U. S. criminal justice system, that standard is ‘…beyond a reasonable doubt’. In other words, the null hypothesis (the presumption of innocence) has to be rejected beyond a reasonable doubt.It is important to realize that this standard is arbitrary. There is nothing about it derived from theory that makes one standard more valid than another. Obviously the selection of the standard will affect the frequency with which one makes the two mistakes.It is popular to try to avoid making the two mistakes. Cries of outrage are heard whenever a person thought to be guilty is freed and we find (particularly since the advent of DNA testing) that non-guilty people are sometimes convicted in spite of their evident innocence.It may be no consolation to the victims in these cases, but from a system point of view it is important to understand that these mistakes are inevitable. They are a function of the uncertainty. Only in the most extreme circumstances can either of these mistakes be eliminated and that is by committing the other mistake as often as possible.If society never wants to convict an innocent person, don’t convict anyone. But the maximum number of guilty people will go un-convicted. If society never wants to let a guilty person go free, then convict everyone. But the maximum number of innocent people will be convicted.Outside of those extreme cases, the mistakes are unavoidable. They each will be committed; one once in a while and the other once in a while.Thus the aim of any study framework should be to try to achieve a balance.One more note before moving on. If we fail to reject the no difference hypothesis, it does not necessarily follow that the alternate hypothesis is true. Just because we don’t convict the individual in court does not mean that he or she is innocent. It means that we could not meet the burden of proof. Because the standards are arbitrary they can be set in such a way as to make meeting the burden of proof easier or more difficult.Part II – How it works in statistics