science & techanalysisWikipedia

Testing many hypotheses guarantees some false results look statistically significant

the gist

P-hacking, also called data dredging, is the practice of running many statistical tests on one dataset and reporting only the ones that come out significant. At a 5% threshold, about 5% of random hypotheses will look significant by chance alone, so a large enough search almost always finds spurious correlations. The term was coined in a 2014 paper by the three researchers behind the Data Colada blog, who were uncovering such problems in social science. It is an example of ignoring the multiple comparisons problem: subgroups are compared without disclosing how many comparisons were run. Done unknowingly or deliberately, it can make false findings appear credible and undermine scientific integrity.

how it works

A p-value estimates how likely a result would be by chance alone. If a researcher tries twenty comparisons, about one will cross the 5% line by luck, so reporting only that one makes noise look like a discovery.

so what

P-hacking is one reason many published findings failed to replicate, fuelling the reproducibility debates in psychology and medicine. Pre-registering studies and correcting for multiple comparisons are now standard defences.

From Wikipedia, the free encyclopedia · text CC BY-SA 4.0

Illustrative photo · Tyler Vigen, CC BY 4.0, via Wikimedia Commons