Fisher did not receive the criticisms from Neyman and Pearson favorably. He labeled their methods as “childish” and “absurdly academic.” Fisher was especially opposed to the notion of making a choice between two hypotheses, preferring instead the calculation of the “significance” of the available evidence—a concept he had put forward. While a decision is considered conclusive, Fisher’s significance tests were designed to provide only a provisional opinion that could be revisited later. Nevertheless, Fisher’s advocacy for open-minded scientific inquiry was somewhat compromised by his persistent stipulation that researchers adopt a 5 percent cutoff for a “significant” p-value, asserting that he would “ignore entirely all results which fail to reach this level.”
Over time, the acrimony gave way to years of ambiguity, as educational resources increasingly blended Fisher’s null hypothesis testing with the decision-based approach of Neyman and Pearson. What was once a nuanced debate concerning the interpretation of evidence and discussions about statistical reasoning and experimental design, evolved into a set of rigid rules that students were expected to follow.
The mainstream scientific community eventually depended on simplistic p-value thresholds and binary decisions regarding hypotheses. In this environment, experimental effects were considered either present or absent, and medicines were viewed as either effective or ineffective. It wasn’t until the 1980s that significant medical journals began to break free of these entrenched practices.
Interestingly, much of this change can be traced back to an idea proposed by Neyman in the early 1930s. During the Great Depression, with economic challenges mounting, there was an increasing demand for statistical insights into population dynamics, albeit with limited resources at the government’s disposal for such studies. Politicians wanted results quickly, often within weeks or months, limiting the feasibility of comprehensive research. Consequently, statisticians resorted to sampling smaller subsets of populations, providing a chance to develop new statistical methods. For instance, if the goal was to estimate the proportion of the population with children, and a random sample of 100 adults revealed none as parents, the next step would be to determine what this implies about the broader population. Definitive conclusions couldn’t be drawn, as a different sample might show otherwise. Neyman introduced an innovation to address this, calculating a “confidence interval” within which the true population value is expected to lie a certain percentage of the time.
The concept of confidence intervals can be challenging, as they require interpreting tangible data by envisioning numerous hypothetical samples. Similar to type I and type II errors, Neyman’s confidence intervals tackle an essential query, though they often confuse students and researchers. Despite these challenges, confidence intervals provide valuable insight into the uncertainty of a study. It is tempting, particularly in media and politics, to focus on a singular average value for its apparent certainty and precision, which may lead to misguided conclusions. Thus, in public-facing epidemiological analysis, confidence intervals are reported to circumvent undue emphasis on specific values.
Since the 1980s, medical journals have increased the focus on confidence intervals rather than solely on binary conclusions. Nonetheless, changing these habits proves difficult. The relationship between confidence intervals and p-values is not straightforward. For instance, if a null hypothesis posits that a treatment has no effect, and the estimated 95 percent confidence interval for the effect does not include zero, the p-value will be less than 5 percent, prompting the rejection of the null hypothesis based on Fisher’s approach. Consequently, medical publications may be less interested in the intervals themselves and more concerned with the inclusion or exclusion of specific values. Although the field endeavors to move beyond Fisher’s influence, his arbitrary 5 percent threshold remains impactful.
