In this project I investigate the use and misuse of p values in papers published in five (high-ranked) journals in experimental psychology. I use a data set of over 135 thousand p values from more than five thousand papers. I inspect (1) the way in which the p values are reported and (2) their distribution. The main findings are following: first, the authors choose the mode of reporting their results in an arbitrary and strategic way, that is, to make them seem more statistically significant than they really are (which is well known to improve the chances for publication). Specifically, they frequently report p values “just above” significance thresholds directly, whereas other values are reported by means of inequalities (e.g. “p<.1″), they round the p values down more eagerly than up and adjust critical values to the data at hand. Further, about 8% of reported p values are inconsistent with their underlying statistics (e.g. F or t) and most of these mistakes go in the direction of “greater significance”. Finally, it appears that there are “too many” “just significant” values, suggesting use of data or model manipulation techniques to bring the p value to the preferred side of the threshold.
PhD Lunch Seminars Amsterdam
- Speaker(s)
- Michal Krawczyk (University of Amsterdam)
- Date
- 2008-11-11
- Location
- Amsterdam