Downwards curve: many researchers see reviewing as a burden that lowers their productivity.Credit: Getty
In November 2022, health economist Chris Sampson found himself in desperate need of a hero. As associate editor for Frontiers in Health Services, he’d been trying to get a paper reviewed since April. He’d sent out about 150 invitations to potential reviewers and received four reviews, but only one of sufficient quality to be useful. Sampson, who works at the Office of Health Economics, a research and consultancy company based in London, needed two more good reviews, so he tweeted: “I need a #peerreview hero … Heroes, DM me.”
Sampson’s plight is a problem faced by editors the world over in the face of uncontrolled growth in the number of journals and papers. The tally of articles indexed by the citation database Web of Science tripled from about one million in 1990 to nearly three million in 2016, according to the website Publons1, which tracks peer-review contributions and is now part Web of Science under the analytics company Clarivate. But the size and composition of the reviewer pool has not kept pace, say editors, because journals often favour well-known scientists from countries with established science infrastructures, rather than early-career researchers or scientists from emerging science nations.
Balazs Aczel is a psychologist at Eötvös Loránd University in Budapest who studies the processes of science. Using a data set covering more than 87,000 scholarly journals, Aczel and his colleagues estimated that researchers globally, in aggregate, spent the equivalent of more than 15,000 years on peer review in 2020 alone2. And many scientists are declining to review more frequently. On Clarivate’s ScholarOne, a manuscript-tracking platform that helps to organize reviews for more than 8,000 academic journals, the average rate at which scientists accept a review dropped from 37.5% in 2020 to 32.3% in 2022.
Balazs Aczel found that global researchers collectively spent more than 15,000 years on peer review in 2020.Credit: Gabor Z. Horvath
Pandemic burnout seems to have exacerbated the problem. A poll of Nature readers last November (the results of which will be reported later this month) found that about one-third had reduced their reviewing activity since March 2020. Senior and mid-career researchers, who perform the bulk of peer reviews, were most likely to cut back.
“It is unrewarded work,” says Donna Yates, a criminologist at Maastricht University in the Netherlands. “One gets nothing from peer review, other than being taken away from work that helps your research or teaching.” In addition, the scant evidence available suggests that peer review adds only marginally to publications’ quality, says Olavo Amaral, who studies the reproducibility of science at the Federal University of Rio de Janeiro in Brazil.
Many scientists are increasingly frustrated with journals — Nature among them — that benefit from the unpaid work of reviewing while charging high fees to publish in them or read their content. A spokesperson for Springer Nature says: “We’re always looking to find new and better ways of recognizing peer reviewers for their valuable and essential work.” They also pointed to a 2017 survey of more than 1,200 Nature reviewers, in which 87% of respondents said they considered reviewing to be their academic duty, 77% viewed it as safeguarding the quality of published research and 71% expected no reward or recognition for reviewing (see go.nature.com/3dnzpgr). But now, amid post-pandemic challenges, many reviewers find that duty overly burdensome.
Yates and others suggest that cash payments would solve the problem, but others say such a system would be unethical and unsustainable. A better solution might be to share the load more widely — with early-career scientists, for instance, or those in less well resourced countries, who are not yet heavily involved — or even to let computer algorithms bear some of the burden.
“I think the notion that we have to review every paper might be a bit utopic,” says Amaral. “I think the system itself might be untenable.”
Expectation and reward
The current peer-review system can be traced back to the eighteenth century, when the United Kingdom’s Royal Societies distributed reports to expert members. The goal was to veto anything that might damage publishers’ reputations. A more formal system, using anonymous experts as publishing gatekeepers, emerged in Britain and United States during the nineteenth century, when papers supplanted pamphlets and lectures as the main means of disseminating scientific results. An influx of public funding for US science in the mid-twentieth century created further pressure for quality control as the volume of published papers increased. Thus, peer review arose in English-speaking nations when science was an elite pursuit — unlike today’s global enterprise, which also includes the preprint option to get information out without gatekeeping by reviewers or journals.
Olavo Amaral says that journals can create better systems for controlling research quality.Credit: Patrícia Bado
Journals typically require two or three reviewers per paper, and scientists do four to five reviews a year on average1, although some perform many more, according to Publons. They’re generally expected to fold reviewing into their academic workloads. It’s rarely an explicit duty or rewarded with money or credit, although it can be rewarding: scientists not only get to see the latest research before anyone else, but also gain insight into the review process that can help with their own submissions.
Jesse Cook, a graduate student in clinical psychology at the University of Wisconsin–Madison, considers reviewing an important act of altruism for the scientific community. In 2019, a couple of years into his PhD studies on sleep, he began consulting for companies — who asked him how much he’d like to be paid. He now makes between US$150 and $200 an hour for that work — throwing into sharp relief the fact that journals rely on similar levels of expertise in their peer-review processes, but pay nothing.
The COVID-19 pandemic made him consider his time more carefully. “I think we all had a perspective adjustment,” says Cook. “And obviously family time became a priority for many.” Cook still does about one review per month — but he also declines more often than he used to, preserving time for his family and dissertation research. Others are also being more selective in the invitations they accept. “Reviewing for free for a for-profit company is kind of a scam,” says Amaral, who prefers to review for non-profit journals.
Rebeccah Lijek, a molecular biologist and peer-review scholar at Mount Holyoke College in South Hadley, Massachusetts, directs her reviewing energies to preprints, posted on servers such as bioRxiv and open to comments from anyone. She loves being able to consider the science just on its merits, without judging whether a study is original enough to deserve publication by a specific journal. She finds the process more collegial and has made new connections and friends through it. “It’s just a little bit more fun,” says Lijek.
Rebeccah Lijek reviews mostly preprints.Credit: Mount Holyoke College staff photography
Preprints can help to spread scientific findings quickly, as happened during the COVID-19 pandemic. But they don’t replace the publications that scientists need to populate their CVs. And the publication process is getting slower, says Detlef Weigel, a plant geneticist at the Max Planck Institute for Biology in Tübingen, Germany. Forty years ago, reviews came back in about three weeks, he recalls — now, he doesn’t expect a reply for six weeks at the earliest.
By the time papers come out, studies might be months or even years old, which can have serious policy implications, says Sandersan Onie, a psychology researcher at the Black Dog Institute, a non-profit mental-health research facility in Sydney, Australia. For example, he cites a study that he’s submitting on suicide rates in Indonesia. “We need to get this information out to help shape policy,” says Onie, who is also president of the Indonesian Association for Suicide Prevention in Jakarta.
Sandersan Onie says that delays in peer review can be a matter of life and death when it comes to psychology studies.Credit: Aphrodite Delaguiado
The never-ending publication cycle creates more work for researchers, says Firdaus Hafidz, a public-health researcher at Gadjah Mada University in Yogyakarta, Indonesia, who is still trying to publish work from his PhD, earned in 2018. If a review takes months or years, he adds, he might have to update the literature review or redo analyses.
The glacial pace of peer review can cause some young researchers to leave academia altogether, says Pradeep Kumar, a biomaterials scientist at the University of the Witwatersrand in Johannesburg, South Africa. To move from the master’s degree programme to PhD candidacy there, students must publish a paper. When delays occur, students often head off to work in industry for the interim — but some never return. Kumar has lost three students, all women, that way. “Publication delayed is publication denied,” he says.
In search of incentives
To combat delays, the centuries-old peer-review system needs a redesign, says Onie. “If you want people to do something, it just needs to be incentivized.”
Some journals do pay reviewers or waive their article fees or subscription costs. For example, submitting three or more reviews per year for a Nature Research journal entitles reviewers to a free, online subscription to the journal they choose from Nature Research’s offerings. The Springer Nature spokesperson says that the publisher’s reviewers appreciate recognition in the form of these subscriptions, and having their names included in the published papers.
Other journals give reviewers public credit by posting the reviews or crediting scientists’ profiles on Clarivate’s Web of Science Reviewer Recognition Service. But Yates says that such online credit is worthless. She reviews three papers for every one she submits, but would like to be paid for her efforts, as she is for applying her criminology expertise as a consultant in government projects, or for giving expert testimony in courts. “Peer review should be flat-out paid,” she says.
Why early-career researchers should step up to the peer-review plate
Other scientists argue that paying reviewers would be unaffordable for many journals unless they raised subscription fees. Aczel and his colleagues estimated the monetary value of the time that scientists in three nations devote to reviewing, and came up with upwards of US$1.5 billion a year in the United States, more than $600 million in China, and nearly $400 million in the United Kingdom2. Non-profit journals might not be able to compete for reviewers if commercial rivals paid. And researchers eager for an easy pay cheque might churn out lower-quality reviews.
Pablo Manavella, a plant molecular biologist at the Coastal Agrobiotechnology Institute in Santa Fe, Argentina, worries that journals would just pass the extra costs of reviewer fees onto authors — something that those in lower-income nations could ill afford. Extra fees, in the form of article-processing charges, would come out of Manavella’s scant government grants and be funnelled to reviewers, who are more likely to reside in wealthier nations. “I don’t think that this is right,” he says.
Nor is a cheque what most reviewers desire. In a 2019 report about grant peer review by Publons, the authors asked scientists what would motivate them to take on reviews. Cash ranked sixth, behind such incentives as more-explicit recognition by employers and online records of review efforts3.
“A lot of people respond quite well to having some kind of credibility or prestige,” such as being listed on an editorial board in exchange for review, says Sampson. “I think that’s a much more generalizable and fairer basis for rewarding peer review.”
Expanding evaluation
But none of these incentives gives researchers what they truly need — time. To get thoughtful reviews, without sucking reviewers dry, journals might need a far-reaching overhaul.
One simpler solution is to enlarge the pool of reviewers, seeking out a more diverse group. “We’re not using enough early-career researchers,” says Bernd Pulverer, head of scientific publications at EMBO Press in Heidelberg, Germany, and formerly chief editor of Nature Cell Biology. “We’re using too many men, too many white people.” EMBO is tracking reviewers by gender and country and inviting more early-career researchers, Pulverer says.
To find reviewers, editors often turn to software. But these computer-generated lists aren’t as diverse as they could be, says Regine Paul, a political scientist at the University of Bergen in Norway and co-editor of Critical Policy Studies. Her system gives her 30 suggestions for each paper. “Among those there will usually be one or two scholars not from the United States or Europe or Australia,” she laments. She does her best to find a wider range of reviewers herself, but doesn’t always have enough time.
Computer-generated lists of reviewers aren’t diverse enough, says Regine Paul.Credit: Eivind Senneset
Onie notes that some potential reviewers in lower-income nations might decline to review because they lack confidence. In Indonesia, he says, “we have professors who have never published a paper because academia is so new.” A handful of programmes now offer training, and even certification, in peer review for early-career or inexperienced researchers.
Seeking reviews from members of under-represented groups could add to the ‘minority tax’ that they face already through being routinely asked to spend unpaid time on issues such as diversity, equity and inclusion. Moreover, in some nations in the global south, reviewing could eat into a researcher’s income or laboratory funding, because some governments and universities tie funding to publications. Kumar says that Witwatersrand, for example, grants investigators a proportion of government subsidies to run their labs, and that proportion is based on their publishing rate. Hafidz and Onie, both based in Indonesia, say they can receive cash payments of hundreds or even thousands of US dollars for publishing in a top-tier journal. But reviews don’t garner any bonuses, so any time spent reviewing, rather than on getting papers published, could be money down the drain.
Another solution could be to reduce the amount of reviewing that journals require. Some journals are screening papers more heavily before sending them out for review, and are rejecting papers or asking for improvements before inviting reviewers to weigh in, says Josh Dahl, senior director of product management at Clarivate. Other researchers say that some types of paper, such as those introducing databases or ideas without new data, could go through with less extensive review.
To fix peer review, break it into stages
Computers could help, too. Journals already rely on algorithms to identify plagiarism or altered images. And although tools such as artificial intelligence aren’t poised to take over peer review just yet, software could check statistical analyses or run a checklist on basic methods4 — functions that, according to data collected by Amaral, human reviewers perform only modestly well anyway5.
Journals could also recycle reviews, so that not every resubmission would require three new reviewers. Aczel estimated that this would save 28 million reviewer-hours per year2. For example, scientists whose studies are rejected by one Springer Nature or PLOS journal can resubmit to other journals in the same family, with the same reviews attached. At the end of January, the journal eLife instituted a model whereby all peer-reviewed papers are published on its platform as ‘reviewed preprints’ alongside an assessment from the journal and public reviews. Authors have the option to respond, and can then choose whether to revise, complete publication with eLife or send the study and reviews to another publication.
Other organizations offer reviews before authors commit to a particular journal. Peer Community in, a non-profit initiative, offers peer review, after which authors can publish freely in one of the organization’s partner journals or submit their study, with accompanying reviews, to other publications. Another programme, Review Commons, provides authors with reviews, which they can then include in submissions to several affiliate journals, including those from EMBO Press. The programme, launched at the end of 2019, is “working very well”, Pulverer says. “The information is sufficient for editors to make informed decisions.” Reviewers are eager to participate, because they’re able to focus on the science in the study, rather than its suitability for a specific journal.
Serge Horbach, a postdoc in the sociology of science at Aarhus University in Denmark, thinks that science is heading for a ‘publish–review–curate’ model. Instead of acting as gatekeepers, journals would come in at the end, to archive the version of record and index papers in databases. “The journal gives it a stamp of approval and says, now we accept this as certified knowledge,” Horbach explains.
Amaral, who sees the current peer-review system as insupportable, says it doesn’t have to stay that way. “I think we can rebuild better quality control, in better ways.” Doing so might keep reviewers from cracking under the pressure of the ever-growing scientific enterprise.
