science & techanalysisWikipedia
Testing many hypotheses guarantees some false results look statistically significant
P-hacking, also called data dredging, is the practice of running many statistical tests on one dataset and reporting only the ones that come out significant. At a 5% threshold, about 5% of random hypotheses will look significant by chance alone, so a large enough search almost always finds spurious correlations. The term was coined in a 2014 paper by the three researchers behind the Data Colada blog, who were uncovering such problems in social science. It is an example of ignoring the multiple comparisons problem: subgroups are compared without disclosing how many comparisons were run. Done unknowingly or deliberately, it can make false findings appear credible and undermine scientific integrity.
A p-value estimates how likely a result would be by chance alone. If a researcher tries twenty comparisons, about one will cross the 5% line by luck, so reporting only that one makes noise look like a discovery.
P-hacking is one reason many published findings failed to replicate, fuelling the reproducibility debates in psychology and medicine. Pre-registering studies and correcting for multiple comparisons are now standard defences.
From Wikipedia, the free encyclopedia · text CC BY-SA 4.0
Illustrative photo · Tyler Vigen, CC BY 4.0, via Wikimedia Commons
