Hypothesis testing and arguments

Lately ideas related to hypothesis testing have been showing up a lot on the blogs I read, both in general--what exactly is it, what is the right way to do it, how do we teach it, how should we frame the conclusions, etc.--and in specific contexts. A common caveat is that failing to reject a null hypothesis is not the same as demonstrating that the null is true. Can't argue with that, but when facing this kind of question--what conclusion can we or should we draw if we do or do not reject the null--I think that it is essential to recognize that the statistical test is being employed as a rhetorical device and that it should not be viewed as independent of other elements of the argument. Hypothesis testing, like any bit of statistical inference (or any bit of science, for that matter), can in principle be executed and presented in a value-free manner, but in practice it virtually never is. Once you start talking about what we would like to know, you're imposing values on the process. Any argument about what we should do based on the statistical results will necessarily be value-laden. If there is some decision to be made based on the results of a hypothesis test in which the null was not rejected, one has to decide how failing to reject the null bears upon that decision, and there isn't going to be a single, objective answer to that.

Torture

Here's a hypothetical:

Let's say that the police have a man in custody who has kidnapped a child and hidden the child in a location that he refuses to disclose. Let's also assume that inflicting physical pain on this man is the only way to get him to disclose the location, that there would be no permanent damage caused by this infliction of pain, and that there is no other way that the police will be able to find the child. Is it then okay to torture the man for the sake of the child?

More about beliefs vs. facts

This article refers to research that makes some progress on the question of how to correct false beliefs. At the same time, reading it renews the concern I discuss here and here. If the article is correct about the reason for the persistence of false beliefs, we should expect that some people hold correct beliefs for the wrong reasons. I.e. if beliefs can be based on something other than facts, then some beliefs will be only coincidentally consistent with facts. These coincidental cases can lead to an unjustified confidence in one's whole process of belief formation. I'm concerned that someone reading the article might reason as follows:

• Global warming is a problem.
• I believe that it is a problem.
• Therefore, I am on the side of truth.
• Therefore, my beliefs about other things are correct also.

Quite the meta-problem.

Knowledge and the real unicorn

My daughter often talks about alicorns and sometimes pretends to be an alicorn. That's a cross between a pegasus and a unicorn, i.e. a unicorn with wings. So why can't she just be a unicorn with wings? Because unicorns don't have wings, of course. Not real unicorns (according to My Little Pony, that is, and perhaps other sources as well).

Someone who is asked how to kill a vampire will probably say a stake through the heart, and perhaps will mention other methods; but will most likely not qualify the statement with anything like "typically" or "according to the majority of popular sources." When it comes to some of the finer points of vampire mythology, like whether they can survive in sunlight, there is significant disagreement among popular sources (The Hunger: yes; Buffy the Vampire Slayer: no; Twilight: yes, but they prefer to avoid direct sunlight because it makes them glitter conspicuously). It wouldn't be unheard of for vampire geeks to argue about such things without acknowledging that they are really arguing about which mythology is the dominant one, or which mythology each person prefers. Like when I was talking to someone who claimed that "Elvish" and not "Elfish" is the correct way to refer to the language of elves. ("Says who?" I said. "You realize elves aren't real, right?")

Consider, by way of contrast, Sherlock Holmes. He is no more real than unicorns, vampires, or elves, but if you want to know his address, there is a definitive answer. Or at least there is an answer that virtually everyone can agree on, because virtually everyone would agree that Arthur Conan Doyle is the definitive source. This even though there may very well be fan fiction out there in which Holmes lives at 221A Baker Street, or 1600 Pennsylvania Avenue, or wherever. In the same vein, one can confidently claim that J.R.R. Tolkien is the definitive source for information about hobbits. He did invent them, after all. There is much more room for argument about elves (and orcs, and all kinds of other stuff), not least because there won't be universal agreement about what "definitive" even means. (Tolkien also capitalizes the names of these races, but I'm not on board with that. Languages, on the other hand, even made-up ones, should be capitalized, imo.)

Here comes the tortured analogy. Nowadays, knowledge is viewed more like the mythology of vampires or elves than that of Holmes or hobbits. I think that's fair to say. There is good reason to view results of scientific research as authoritative (which is not to say infallible), whereas arguing over whether one non-scientific source is more authoritative than another tends to boil down to declaring allegiance to something--an ideological group, a way of thinking, a set of values, etc. Which is something like claiming that one depiction of vampires (or elves or unicorns) is "right" and another "wrong." Far from a perfect analogy, but what the hell.

God and null hypotheses

Comedian Tim Minchin performs a song, "Thank You God," prefatory to which he tells a story about meeting a fan of his who is a Christian. When asked why he does not believe in God, Minchin says this is part of a more general policy of only believing things for which he has evidence. What he does not acknowledge is that there also is no evidence that God does not exist. Don't get me wrong: I do enjoy the man's comedy. However, if we're being scientific about this, we start with one of two null hypotheses--God exists, or God does not exist--and we cannot reject either. One might argue that one of those hypotheses is the more reasonable starting point, but it is incorrect to claim that the lack of evidence is an unconditional indication that God does not exist (although, to be fair, I'm not completely sure Minchin was trying to make that claim; maybe he was standing up for agnosticism rather than atheism).

Scientists of all kinds avoid any confusion over the conclusions of research by stating the conclusions carefully. You'll often hear the statement, "There is no evidence that [blank]," which basically means that if we start with the assumption [not blank], we don't have enough evidence to reject that assumption with any reasonable degree of certainty. That could be the case after hundreds of peer-reviewed studies have attempted to demonstrate [blank], or it may be that no one has even tried. Either way, "There is no evidence that [blank]" is not at all the same claim as "There is evidence that [not blank]" and should not be treated as such. Case in point: when my firstborn was an infant, I heard from more than one source that there is no evidence that allowing a baby to "cry it out" causes any lasting psychological damage, but I wasn't about to let my son cry for hours at a stretch.

I think it is common for null hypotheses to go unnoticed or unacknowledged. There's always some kind of default belief. Consider the vaccination scare, still active in some quarters. I don't think that childhood vaccinations cause autism, and that conclusion is predicated on the belief that we shouldn't think that vaccines cause autism unless we have some good reason to do so. One could take the opposite belief as the default, once the possibility of a connection is raised: that vaccinations do cause autism, or some other kind of harm, unless we have strong reason to believe otherwise. I'm not sure if that belief is thoroughly unreasonable, but it is certainly subject to criticism. Such as: given all of the potential causes and potential effects in the universe, any pair chosen at random are extremely unlikely to be related, and therefore the reasonable default belief is that there is no causal relationship between any pair of randomly chosen events. And why are we focusing on autism in particular? Why not other disorders or diseases, or even positive effects, for that matter? Why not take the default belief to be that childhood vaccines cause pattern baldness later in life? And so forth. Apart from the reasonableness of the null itself, adopting the wrong null is costly if there is insufficient evidence to reject the null. If you avoid vaccines because you believe they cause autism, you risk disease that vaccines can prevent (and we have very good evidence for that).

Back to Tim Minchin: it hardly makes sense not to believe anything without evidence. A more defensible statement is that one will not deviate from some kind of baseline beliefs without sufficient evidence. Then the origin of these baseline beliefs becomes the issue. These baseline beliefs are often unstated and may even be subconscious. I doubt that the anti-vaccination crowd is thinking things through in the way I describe above. Opposition to vaccinations may arise from a fundamental belief in the ability of the human body to heal itself under some kind of natural conditions, or from a distrust of the medical establishment's ability to improve health in a substantive way. I suspect that anyone considering the vaccination question has some kind of predisposition toward one side of the debate or the other, and that this predisposition often goes unarticulated.

I can imagine all kinds of baseline beliefs that fuel conflict over social and political issues. For example:

• People are basically good (or bad)
• Government agencies are generally corrupt (or trustworthy)
• Life on Earth is generally getting better (or worse)

One can find indications of any of these things but nothing like definitive proof. If we think about beliefs scientifically, the way to get evidence for something is to assume the opposite and then demonstrate that the data are inconsistent with that assumption. There is no further scientific guidance in forming the null hypothesis: it's just the thing you are trying to disprove. Of course there will always be some reason why one chooses to look for evidence of one particular thing, but this cannot be reduced to a purely logical exercise. I would like to see more explicit acknowledgement of all sorts of baseline beliefs: e.g., the anti-vaxxer says, "I oppose vaccinations because I think medical science is full of it," or the research scientist says, "I am trying to prove this result because it would be neat if it were true."

Tim Minchin's disbelief in God is a matter of faith. Nothing wrong with that, and he can try to argue that this belief is somehow better or more reasonable than the opposite belief. But he can't claim that he is on the side of objective truth.

Do we really need to regulate that?

Why is it illegal to smoke on domestic flights in the U.S.? Whatever benefit there is of prohibiting smoking on a flight accrues to those who are actually on the plane. If this benefit is large enough, then it is in the interest of the airline itself to prohibit smoking, because the benefit to passengers translates directly into greater willingness to pay for airfare and greater profits for the airline. Is it somehow easier to enforce a smoking ban if the force of law is behind it? Did the government pass the law to insulate airlines from the ire of those passengers who would prefer to smoke in flight? Maybe, but I would also note that some airlines had their own smoking ban before the law took effect in 1998. Is the purpose of the law to level the playing field, because there is some negative consequence of allowing airlines to compete through their policies regarding smoking? Again, maybe. I wouldn't suggest that anyone should be able to smoke on airplanes, either as an economist or as an air traveler; but it is clearly incorrect to assume that the only way to achieve this outcome is by passing a law. This seems to be a fairly common belief: that if the government doesn't make something happen, it won't happen.

Siri, tell me what I want to hear

Web sites use increasingly sophisticated algorithms to tailor content to the individual user: Netflix and Amazon deliver personalized recommendations, news aggregators filter content, and Google search results depend not just on the search terms themselves but also on whatever personal characteristics or details of one's historical usage are available. I imagine a near future in which I'll get into my driverless car around dinner time and be driven to my favorite Mexican restaurant without even having to say anything. If the car has enough predictive ability and access to enough data, I won't have to specify that I'm in the mood for Mexican, or even that I'm hungry. The car would never be able to get it exactly right every time, but it's easy to imagine it being close enough that the user wouldn't bother quibbling about it.

So, could this possibly be a bad thing? Eli Pariser, author of The Filter Bubble (and presenter of a TED talk), thinks so. His concern is that all this personalization of content has an isolating effect. My starting point is an idealized view of filtering as a way of culling some subset of information from all of the information available. The user can sift through the information the old-fashioned way and decide what to consume--e.g., glance over the headlines and decide what articles to read--or allow the filter to do the sifting. If the filter arrives at the same subset of information that the user would have ultimately consumed anyway, then the filter is unambiguously beneficial insofar as it eliminates the cost to the user of sifting through the information.

The question is then in what way or to what extent the actual scenario diverges from this ideal. I can think of three possibilities:

Just the facts, ma'am

An addendum to this post about the problems when uncertainty is disregarded. In addition to, or instead of, expressing uncertainty about something, one can simply focus on the part that is certain. E.g. if someone asks who George Zimmerman is, instead of describing him as "that guy who murdered Trayvon Martin," one could describe him as "that guy accused/acquitted of murdering Trayvon Martin." Factual and indisputable. It works for less concrete statements too, like "Volvos have a reputation for safety," instead of "Volvos are really safe." Can't hurt to state only the parts we can all agree on as facts, and to debate the parts that are debatable.

Circumspection and other virtues

In a previous post, I lamented the problems arising from the "well-intentioned propagation of misinformation." We would all be better off if everyone could just be less confident.

Too much and not enough

I've been thinking about informational issues for several years. Part of the motivation for this arose during the vaccination scare, which was at a fever pitch around the time my son was born.