‘Spell-checker for statistics’ reduces errors in the psychology literature

A free-to-use tool designed to detect statistical errors is significantly reducing the number of mistakes creeping into papers, suggests a study published this January¹.

First described in a 2015 publication², statcheck is an online tool that can identify errors in the P values — a controversial but popular statistical measure — reported in scientific papers. The tool, developed by Michèle Nuijten, a methodologist at Tilburg University in the Netherlands, has been applied to tens of thousands of psychology studies during peer review and by authors themselves, and according to a 2017 study, it is right more than 95% of the time³.

Several journals have now implemented statcheck — which Nuijten calls a spell-checker for statistics — in their peer-review processes. In the new study, which was published on the PsyArXiv preprint server and has not yet been peer-reviewed¹, Nuijten set out to quantify the tool’s impact by comparing the rate of errors in P values in two psychology journals before and after they implemented statcheck, and in two journals that have never used the tool.

Error reduction

The team looked at more than 7,000 papers published since 2003, and found that papers published in Psychological Science and Journal of Experimental and Social Psychology had 4.5 percentage points fewer statistical errors after they started using statcheck. By contrast, the Journal of Experimental Psychology: General and Journal of Personality and Social Psychology, which don’t use the tool, had a reduction of only around one percentage point.

NatureTech hub

That means the journals using statcheck had a more than fourfold reduction in errors in reported P values compared with those that have not implemented it, Nuijten says.

Nuijten attributes the reduced error rate in papers in the journals that don’t use statcheck to an increased awareness of issues around statistical significance over the study period, as well as to researchers using statcheck themselves before they submit their manuscripts.

The software is available as a web application to which researchers can upload their manuscripts. It has been accessed more than 100,000 times since its launch in September 2016. The tool is also available as a plug-in for Microsoft Word to help researchers identify errors during the writing process.

Patricia Bauer, a psychologist at Emory University in Atlanta, Georgia, and editor-in-chief of Psychological Science, says, “statcheck is a free, easy to use, easy to implement step that authors can take and that the journal can require.” The journal requires authors to submit a clean statcheck report with no reported errors as a prerequisite to publication.

But Anna-Lena Schubert, a psychologist at the Johannes Gutenberg University of Mainz in Germany and secretary of the German Psychological Society, worries that it might not be possible to obtain a clean statcheck report if the tool has made mistakes. In those cases, she says, authors should have the opportunity to submit a rebuttal.

Still, Schubert concludes, “given its relatively low error rate, it would be very beneficial for psychology journals to adopt statcheck as a screening tool to increase the rigour and robustness of psychology literature”.