12 P-Value Misconceptions

< Back to search results

Steven Goodman reviews twelve misconceptions about the meaning of p-values.

• Format Texts
• Language/s English
• Target Audience Further education
• EBM Stage 3 - Appraising evidence
• Duration >15 mins
• Difficulty Advanced

Key Concepts addressed

• 1-2f Consider all of the relevant fair comparisons
• 2-3f Confidence intervals should be reported
• 2-3h Don’t confuse “no evidence” with “no effect”

Details

The P value is a measure of statistical significance that appears in virtually all medical research papers. Its interpretation is made extraordinarily difficult because it is not part of any formal system of statistical inference.

As a result, the P value’s inferential meaning is widely and often wildly misconstrued, a fact that has been pointed out in innumerable papers and books appearing since at least the 1940s.

This commentary reviews a dozen of these common misinterpretations and explains why each is wrong. It also reviews the possible consequences of these improper understandings or representations of its meaning.

Finally, it contrasts the P value with its Bayesian counterpart, the Bayes’ factor, which has virtually all of the desirable properties of an evidential measure that the P value lacks, most notably interpretability.

The most serious consequence of this array of P-value misconceptions is the false belief that the probability of a conclusion being in error can be calculated from the data in a single experiment without reference to external evidence or the plausibility of the underlying mechanism.

Read more…