Hypothesis testing and arguments

Lately ideas related to hypothesis testing have been showing up a lot on the blogs I read, both in general--what exactly is it, what is the right way to do it, how do we teach it, how should we frame the conclusions, etc.--and in specific contexts. A common caveat is that failing to reject a null hypothesis is not the same as demonstrating that the null is true. Can't argue with that, but when facing this kind of question--what conclusion can we or should we draw if we do or do not reject the null--I think that it is essential to recognize that the statistical test is being employed as a rhetorical device and that it should not be viewed as independent of other elements of the argument. Hypothesis testing, like any bit of statistical inference (or any bit of science, for that matter), can in principle be executed and presented in a value-free manner, but in practice it virtually never is. Once you start talking about what we would like to know, you're imposing values on the process. Any argument about what we should do based on the statistical results will necessarily be value-laden. If there is some decision to be made based on the results of a hypothesis test in which the null was not rejected, one has to decide how failing to reject the null bears upon that decision, and there isn't going to be a single, objective answer to that.

Here's an example. I get a lot of headaches, and I recently began taking a medication for something else that has headaches as a potential side effect. I seem to have been having headaches more frequently than usual, but it's hard to tell whether that is because of the new medication, especially since my headache frequency has exhibited a fair amount of variance over time. I would bet that, given all of the data that I could gather about my headaches and any potentially related variables, I would not be able to reject the null hypothesis that the new medication has not increased my headache frequency. On the other hand, I bet I would also not be able to reject the null hypothesis that the new medication has increased my headache frequency. Rejecting a null means having data that are sufficiently unlikely given the null, which in this case means observing a headache frequency that is high enough (to reject the first null) or low enough (to reject the second). I could easily observe a frequency that is somewhere in between.

Failing to reject a null hypothesis can be viewed in different ways, depending on where the null came from in the first place. If the null is something that we have good reason to believe is true, and we fail to reject that null, then we are going to maintain the prior belief (which isn't quite the same as taking the statistical results as proof that the null is true, but we end up in the same place). If, on the other hand, we would like to be able to reject the null--for example, if we have a new blood-pressure medication, and the null equates to "this medication does not lower blood pressure"--then we won't be so sure that the null is true, but we won't be so sure it's false either. The conclusion will be something like "there is no evidence that this medication significantly lowers blood pressure." One might still take one's chances, though, depending on the alternatives available. The statistical statement itself is more or less objective, but the broader conclusion isn't.

Is my medication giving me headaches? I wish I knew.