NIH plans grant-review overhaul to reduce bias

The US National Institutes of Health (NIH) has released a tentative plan to change how its research grant applications are scored, with the aim of reducing bias and lowering the burden on reviewers. Under the new system, reviewers would no longer rate researchers’ expertise or their institutions’ access to resources, and there would be fewer scoring criteria overall.

The NIH’s Center for Scientific Review (CSR), which organizes the peer-review groups that evaluate more than 90% of the research grants awarded by the agency, announced these proposed changes at a meeting on 8 December attended by Lawrence Tabak, acting NIH director, and a panel of his advisers. The revamp has not yet been finalized, and any changes would not be implemented until 2024 at earliest.

NIH issues a seismic mandate: share data publicly

Critics have long contended that the NIH perpetuates reputational bias in its review process, which has led to funding disparities among grant recipients. According to a 2022 NIH analysis, the 10% of institutions that receive the most money from the agency get about 65% of its overall funding for research projects, and the bottom half receives less than 5%. These proportions have remained persistent for years.

Many institutions — such as minority-serving universities — produce strong research, says Noni Byrnes, the director of the CSR. But these data show that because of reputational bias, these institutions have not been on a level playing field in the review process, she adds. “And ultimately, what we want is the science to work.”

Researchers who spoke to Nature say that changes to the peer-review system are long overdue, and that the proposals are a good start. But some think the revisions don’t go far enough to correct these entrenched problems.

Criteria overhaul

Reviewers currently score NIH research proposals according to five criteria: significance, investigator(s), innovation, approach and environment (where the research will be carried out). These criteria are defined by US legislation, so the NIH cannot modify them without approval from lawmakers, but it can change the way they’re interpreted or scored. The new system doesn’t throw out the old criteria, but groups them into three categories: the importance of the research, its feasibility and rigour, and the expertise and resources of the researcher and their institution.

Four lessons from the pandemic to reboot the NIH

Byrnes says that the last category, which won’t be scored under the proposal, is frequently misinterpreted. Reviewers sometimes score applicants and their institutions without considering them in the context of the proposed research — the original intention of the category. This has led to higher scores for prestigious institutions and individuals. Under the proposal, rather than score this category, reviewers would choose whether they think researcher expertise or institutional resources are adequate or not. If they select the latter, they can leave specific feedback about deficiencies in a text box on the review form. This will “prevent reviewers from waxing poetic about a really famous investigator that tilts the evaluation of the science”, Byrnes says.

“For a big funder like NIH, these changes are quite bold,” says Sandra Bendiscioli, a research policy specialist at the European life-sciences organization EMBO in Heidelberg, Germany, who has published about bias in peer review.

The root of the problem

Some advisers attending the 8 December meeting pushed back on the plan, suggesting that researchers and institutional resources are crucial factors in determining the merit of research projects. “I do think there’s some value in some objective score to assess the investigator,” said Shelley Berger, an epigeneticist at the University of Pennsylvania in Philadelphia. Without a score, Berger added, it could be difficult to understand the reviewer’s thinking and how they factored researcher expertise into their decision. Speaking at the meeting, Byrnes countered that reviewers would still have the option to leave comments about their concerns, which could be reflected in the overall impact score.

Although some critics appreciate that the agency is trying to eliminate reputational bias, they say the proposed changes do not address the root of funding disparities at the NIH. Omolola Eniola-Adefeso, a biomedical engineer at the University of Michigan, Ann Arbor, who co-authored a 2021 article calling on the NIH to fund more Black scientists¹, tells Nature that, to have a tangible impact, more diversity is needed among reviewers. She cites a landmark analysis published more than a decade ago², which found that Black researchers were significantly less likely to receive NIH research funding compared with white researchers, even when factors such as scientific credentials and employer were taken into account. This means efforts to lessen the impact of reputational bias might not translate to more equitable funding, she says.

In addition to considering the race and gender of reviewers, ensuring that panels have representation from under-represented institutions could have a big impact, says Enrique Neblett, a public-health researcher at the University of Michigan. Given the persistence of funding disparities, the NIH needs to take bolder action, he says.

This isn’t the first time the NIH has tried to address reputational bias. In 2017, the agency released a plan to put a cap on the number of grants that a researcher could hold at any one time, but fierce criticism from the biomedical research community led the agency to scrap the changes. Byrnes tells Nature she is optimistic that the new proposal will not face the same kind of backlash. The 2017 plan was made during “a different time, with a whole different set of people in leadership at NIH”, she says.

Bendiscioli applauds the agency for searching for innovative ways to reduce bias in its funding and emphasized the potential global ripple effects of the change, given that the NIH is the world’s largest public funder of biomedical research. “When a big funder like NIH changes things, other funders often want to follow.”

The CSR is accepting public feedback about the revisions until 10 March 2023. It will also release proposed changes to the criteria for scoring clinical-trial and training grants in the coming months.