How can we recognize accurate scientific results when things like p-hacking are so common?
First thing to consider is that research can never provide absolute certainty. Any result you find is only preliminary. You can have various degrees of preliminary: a single study with an acceptable but not brilliant design can provide interesting clues or first steps for a larger project, but not much more. It is right and productive that these studies are published, but don't treat them as truth.
Second thing to consider is that the problem is not necessarily with the articles themselves, but with the lack of replication. So, if you find a result that interest you, try to find replications. If none exist, then you need to be more careful.
The p-curve (see p-curve.com) has been proposed as a way of identifying whether a set of results is likely to have been p-hacked. The term and technique was coined and/or developed by Uri Simonsohn and colleagues, who blog about replication and p-curving at Data Colada.
- "P-Curve Won’t Do Your Laundry, But Will Identify Replicable Findings"
- "Ambitious P-Hacking and P-Curve 4.0"
- "P-curve vs. Excessive Significance Tests"
The technique has been applied by several research groups besides Simonsohn et al, ie Head et al 2015
There have been critics of p-curves (including John "most published finding are wrong" Ionnadis ), as well as other techniques such as excessive significance tests and the distribution of published p-values.
Some references:
- Simonsohn, Simmons, Nelson (2014) "P-Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results" Perspectives on Psychological Science V9(6) p.666-681
- Simonsohn, Nelson, Simmons, (2014) "P-curve: A Key to the File Drawer," Journal of Experimental Psychology: General, V143(2), p.534-547
- Simonsohn, Simmons, Nelson (2015) "Better p-curves" Journal of Experimental Psychology: General.
- Ioannidis and Bruns. 2016. p-Curve and p-Hacking in Observational Research. PLoS ONE.
- Head et al 2015 The Extent and Consequences of P-Hacking in Science. PLoS ONE