Enhancing Peer Review
Starting in 2007, NIH conducted a year-long, formal self-assessment of its peer review system. This assessment aimed to maintain the hallmarks of objectivity, fairness, and maximum competition that form its foundation, while accommodating the growing breadth, complexity, and interdisciplinary nature of modern research. The assessment involved recommendations from external and internal working groups, feedback from advocacy groups and regional town hall meetings, and consultation with professional societies. The final report, issued in March 2008, outlined broad challenges, and recommended transformative enhancements of the NIH peer review system. Subsequently, NIH convened internal committees to outline strategies and timelines to achieve implementation goals in four broad priority areas:
- Engage the best reviewers
- Improve the quality and transparency of review
- Ensure balanced and fair reviews
- Engage in continuous review of peer review
Figure 1: Timeline for Enhancement of the NIH Peer Review Process
In spring 2008, NIH engaged in a detailed, intense, and rapid planning process (see Figure 1) to implement and launch the enhancements. The first changes—adjustments to recognize early stage investigators—were launched in less than a year. The changes began rolling out quickly thereafter and were accompanied by extensive training sessions and communication efforts. Remarkably, the advent of ARRA funding sped rather than slowed implementation. NIH used the new shorter application form for ARRA research grant applications in advance of the scheduled NIH-wide implementation of this enhancement. Other planned enhancements launched on their original timelines.
The peer review enhancement process entailed numerous policy announcements (see Table 1).
Table 1: Enhancing NIH Peer Review: Selected Policy Announcements
||NIH Announces New Scoring Procedures for Evaluation of Research Applications Received for Potential FY 2010 Funding
||NIH Announces Enhanced Review Criteria for Evaluation of Research Applications Received for Potential FY 2010 Funding
|NOT-OD-09-003 and NOT-OD-09-016
||New NIH Policy on Resubmission (Amended) Applications
||Revised New and Early Stage Investigator Policies
Following are highlights of the enhancements made within each priority area.
Engage the Best Reviewers
- New members of scientific review groups were given additional flexibility regarding their tour of duty. They now can expand their period of service preparing for and attending fewer meetings per year over a longer period of time. NIH expects that this option for flexibility will make it easier for reviewers to serve on scientific review groups.
- The Scientific Review Officers who staff SRGs and SEPs now have guidance on best practices for recruiting reviewers.
- NIH is conducting pilot tests of the use of high-bandwidth technological support for review meetings (such as virtual participation via videoconference) to provide reviewers with alternatives to in-person meetings, which require considerable time investments for travel.
- NIH implemented a policy for continuous submission of certain applications from appointed members of chartered NIH advisory groups and frequent temporary members (SRGs and Advisory Councils). Under the continuous submission policy, eligible applicants can submit their R01, R21, and R34 applications continuously (without regard to deadlines). The applications are reviewed by a SRG or SEP no later than 120 days after receipt and then are referred to the appropriate Advisory Council for the final level of review at its next meeting. This benefit is provided as part of the NIH continuing commitment to recognize outstanding peer review service. The first use of the continuous submission policy, in February 2008, was so successful that, in July 2009, it was extended to ad hoc members of advisory groups.
Improve the Quality and Transparency of Review
- NIH began using enhanced review criteria to evaluate research grant applications submitted for potential FY 2009 funding. The enhanced review criteria emphasize the potential impact of the work proposed and de-emphasize details of the experimental design with the intention of improving the quality of review. The enhanced review criteria form the basis for ongoing efforts to align the application format with the review criteria, which will greatly facilitate the transparency of the review process.
- NIH implemented a new 1-9 scoring system, in lieu of the current 41-point scale. Moreover, instead of giving the application just one score, each assigned reviewer also gives a numerical score for each of the now enhanced review criteria. For most applications, the criteria are significance, investigator(s), innovation, approach, and environment. Additional review criteria may be added for applications submitted in response to RFAs and certain Program Announcements. The nine-point scale is designed to provide an optimum range for making reliable and meaningful distinctions among applications.
- Reviewers are using structured templates to compose their critiques of the applications they review. The template focuses the review on the application’s strengths and weaknesses relative to each criterion and fosters more concise and clear communication of the reviewer’s assessment.
- Applications have been shortened and restructured. Applications submitted on and after January 25, 2010, are organized to align with the structure and content of the enhanced review criteria. This helps ensure that review and applicant expectations coincide for a more efficient and transparent process. At the same time, NIH shortened the page limits for certain sections of applications. This both reduces burden and focuses applicants and reviewers on the essentials of proposed research plans.
Ensure Balanced and Fair Reviews across Scientific Fields and Career Stages, and Reduce Administrative Burden
- To ensure that the largest number of high-quality and meritorious applications receive funding earlier and to improve system efficiency, NIH decreased the number of allowed grant application resubmissions (amendments) from two to one.
- Where possible, NIH is clustering New Investigator and Early Stage Investigator22 applications during review, and the same approach was extended to clinical research applications.
- The standard review criteria used by reviewers to evaluate applications for research grants and cooperative agreements were enhanced (see Improve Quality and Transparency of Review above) to include consideration of the investigator’s career stage.
Continuous Assessment of Peer Review
- Ongoing evaluation is critical to the health of the NIH peer review system and assuring that the system embodies the core values of competence, fairness, timeliness, and integrity. To achieve this end, NIH operationalized a dynamic effort to assess the cumulative outcomes of the changes being brought about by the peer review enhancements. This is part of a larger effort to develop appropriate measures and indicators for future monitoring efforts.
Launching RePORT: A Central Portal for Information on NIH Research Activities
NIH is committed to promoting a high level of public accountability for its investment of public funds. As part of that effort, NIH strives to provide extensive, detailed, and accurate information on its research funding in a user-friendly format. To that end, the Research Portfolio Online Reporting Tool (RePORT) was created by OER. RePORT serves as the central repository for all NIH external reports and as a public access point for comprehensive information, data, and analyses of NIH research activities. This includes information on NIH expenditures and the results of NIH-supported research, as well as a section on reports specific to recent issues of interest, such as the Recovery Act. To facilitate and encourage public use of RePORT, a tutorial introducing the major features of RePORT is presented on the site.
The RePORT home page provides links to frequently requested information and to major sections of the site, including:
- The NIH Data Book, which provides basic summary statistics on extramural grants and contract awards, grant applications, the organizations NIH supports, the scientific workforce, and trainees and fellows supported through NIH programs. NIH Data Book charts and tables are generated and updated automatically from a database of NIH statistics and can be exported to PowerPoint or printed in a printer-friendly format.
- NIH Strategic Plans, a site that provides links to strategic plans including IC, NIH-wide, topical, and HHS and inter-agency plans, with information on plans in the process of being updated.
- Categorical Spending, which provides the link to and information about the NIH Research, Condition, and Disease Categorization (RCDC) system. (See section immediately below for more information.)
- RePORT Expenditures and Results (RePORTER), NIH’s new and improved searchable database of funded research projects. (See section below—RePORTER: Expanded Information on Scientific Projects—for more information.)
- The Reports page, which provides access to a searchable database of reports. Each report has been categorized by topic, IC, the portfolio being reported on, the budget mechanisms and activities through which the programs included in the report are funded, and the years covered by the report. There are several drop-down menus that can be used to narrow the search further, which reduces the database containing hundreds of reports to a small set that matches the selected criteria.
- Other information, including this report—The Biennial Report of the Director, National Institutes of Health.
Research, Condition, and Disease Categorization (RCDC) System
In mid-January 2009, NIH launched a new process for providing detailed funding information, by fiscal year, for 215 major research categories, as part of its extensive efforts to keep the American people informed about how their tax dollars are used to support biomedical and behavioral research. The process, known as Research, Condition, and Disease Categorization (RCDC), uses a computerized approach to mine the descriptive text associated with NIH research projects and match it to standardized parameters to categorize the NIH research projects. The public can access the resulting categorical spending reports on the RePORT website.
NIH developed RCDC because it needed a more consistent system for reporting on its research spending and saw that advances in computer technology for data and text mining would enable the agency to modernize its systems. About the same time, the National Academies, an organization that provides scientific advice to the Federal government, issued two reports recommending a change in the way NIH categorizes its research portfolio. Subsequently, the U.S. Congress, through the NIH Reform Act of 2006,23 mandated that NIH build a tool to categorize the agency’s research.
Hundreds of NIH technical and scientific experts helped create the RCDC categorization methods and identify key terms and concepts. RCDC provides increased consistency of reporting, and in turn, enhances NIH’s capacity for portfolio analysis and strategic planning. RCDC also provides improved transparency through the RePORTER database, and improves NIH's accountability for its spending and ability to respond to public inquiries.
The 215 categories reported through the RCDC process are the same categories that historically have been requested by and reported to Congress and the public at the end of each fiscal year. Some of the research funding amounts that the RCDC system reports may differ from NIH reports issued in the past. That is because the RCDC process applies a uniform definition, for each category, across all NIH's research projects. Individual research projects can be included in multiple categories, so the sum of all research/disease categories does not add up to 100 percent of NIH-funded research for a given fiscal year. The annual estimates reflect amounts that change as a result of science, actual research projects funded, and the NIH budget. Despite the changes in categorizing NIH research using the RCDC system, NIH's methods for budgeting and spending tax dollars remain the same.
RePORTER: Expanded Information on Scientific Projects
For many years, one of the most common ways for the public to find information on NIH research programs was to search for projects in NIH’s Computer Retrieval of Information on Scientific Projects (CRISP) system. Now a new system that provides much more detailed information about projects is on-line. The new system, accessed through the RePORT website, is called RePORTER (RePORT Expenditures and Results). Like its predecessor CRISP, RePORTER allows users to locate and view NIH awards using their own search criteria. However, RePORTER also gives users access to budget award information, research results, and other research outcomes such as patents and publications. RePORTER includes data from 1985 through to the present—including projects funded through ARRA—and project lists can be sorted and downloaded to Excel. New features will continue to be added to RePORTER in several releases throughout FY 2010.