Peer review for funding decisions

Once the proposal is submitted, funding agencies will first conduct an initial evaluation (triage). For example, before the review process, the DFG (German Research Foundation) Head Office checks the proposals to ensure that all formal requirements have been met. If not, the applicant has the opportunity to supply missing information. NSFC’s (China) eligibility checks are carried out according to relevant qualification requirements for different programs. Applicants are notified of the results of these eligibility checks, specifying whether the applications are accepted for review or rejected without review. In this way, some proposals may be filtered out in this process.

Although peer review has well-known flaws (discussed further below), the majority of applicants and funders agree that peer review, especially external peer review, is the best method for allocating research funds (Meadmore et al., 2020). Funding agencies must ensure that all the important aspects of the proposal fall within the expertise of the selected reviewers. Reviewers must be recognized as experts in their fields and be capable of providing an objective appraisal of the proposal. In addition, they must carefully avoid conflicts of interest arising from collaboration or competition, teacherstudent relations, reciprocal reviews (see also Section 8.3.1 for recent development), etc. It goes without saying that an expert group usually includes colleagues from all over the world.

Some funders prefer three reviewers per application to come to a majority view among the initial reviewers (Meadmore et al., 2020). In most cases, funding agencies adopt remote reviews, panel reviews (see the next step), or site visits for different proposals. Scientific research fund review practices can vary significantly due to differences in social contexts, institutional structures, and stages of development. This diversity is evident among U.S. funding agencies, each of which tends to align its review approach with its specific mission. For example, the Department of Energy (DoE) considers the laboratory environment, the Department of Defense (DoD) emphasizes technical aspects, and the National Institutes of Health (NIH) prioritizes public health and patientrelated outcomes. Such varied evaluation methods contribute to more balanced and appropriate project assessments.

The next step after the review is to evaluate the reviewer’s statement about the submitted proposal. Generally, the corresponding division director and a committee (panel) make the final decision.

The role of the panel is to make a substantiated decision based on the reports of the reviewers or a ranking of projects, leading to a yes: funded or no: not funded decision. This is generally done in face-to-face meetings or through environmentally friendlier virtual meetings.

The selection of panel members, perhaps even more important than that of experts, is a key element in organizing a successful review procedure. The first point one has to decide is whether the panel is brought together ad hoc or whether members are assigned for a fixed time. In most funding agencies, members of such panels are specialists in a broad field chosen for several overlapping years. This overlap ensures continuation. This panel must be gender-balanced; it should also have a reasonable age balance, so that it is not dominated by older, even retired, scientists. Although the expert group should include colleagues from all over the world, for the panel, that is another question. Including foreign colleagues may increase the costs and may lead to linguistic problems. While incorporating international colleagues into funding panels can enhance expertise and bring diverse perspectives, it is essential to weigh these benefits against potential drawbacks, especially in the social sciences and humanities. Careful selection of panel members, with attention to cultural competence and contextual knowledge, is crucial to ensure fair and effective evaluations (Baccini & Re, 2025). Other questions related to the composition of panels include: must this panel always include a statistician, a computer scientist, a social scientist, a philosopher of science, and other non-specialists (Luo et al., 2021), besides specialists in the panel’s field? This is a legitimate question as many project proposals include statistics, (big) data handling, and discuss social implications.

If panel members do not agree, is there a person who is responsible for making the final decision? Is that person a member of the panel? It may happen that for a difficult decision, such as one about which there exists controversy, or involving a huge sum of money, the final answer comes from “higher up.” In such cases, there should be a majority and a minority report so that the final decision maker(s) can make a well-founded decision. In some funding agencies, the president represents the funding organization, whose task is to confirm the panel’s decision or make the final decision whenever the panel cannot conclude. Yet, in, e.g. the National Natural Science Foundation of China (NSFC), the president and other officials do not have the right to intervene with the panel’s experts. The NSFC states that it is a democracy of scientists.

After the decision, funding agencies inform applicants and other stakeholders about the final decision as soon as possible. Some funding agencies have also established peer review appeal systems, such as the U.S. National Science Foundation (NSF) and the National Institutes of Health (NIH) in the USA to provide investigators and applicant organizations the opportunity to seek reconsideration of the initial review results.

Cole et al. (1977) reported the findings of the first phase: by 75 extended interviews with NSF staff and analyzing 1,200 proposals drawn from ten NSF programs, they found that ratings were strongly related to actual funding decisions. In this way, they concluded that although highly stratified, relying on peer review ratings, the scientific enterprise is exceedingly equitable. However, in the second phase, the result of the COSPUP experiment showed 24 percent and 30 percent of reversal rates with NSF’s funding decisions, even though the reversals shifted from the top 25 positions to the bottom 25 positions; around the cutting point, the reversal rate reached 60 percent in one program. The results indicated that whether or not a proposal is funded depends, in a large proportion of cases, upon which reviewers happened to be selected for it. Cole et al. (1981) reported experimental results and concluded that substantial disagreement among eligible reviewers may reverse funding decisions.

Two or more honest reviewers coming to divergent opinions is called the inter-rater problem (Hug & Ochsner, 2022). The inter-rater problem is found to exist ubiquitously in the peer review system and has detrimental effects on funding decisions, as it violates the consistency criterion for peer review ratings. If these divergent opinions correspond to certain recognizable groups, the inter-rater effect may lead to an in-group versus out-group effect. A similar problem occurs when calibrating scores by different reviewers, such as when “your 3 is my 9” (Wang & Shah, 2019). The detrimental effects of the inter-rater problem tend to make peer review unreliable.

Marsh and Ball (1981) proposed to measure the reliability and validity of a review as the correlation between two independent assessors’ ratings of the same submissions across a large number of different submissions, while Mutz et al. (2015) performed a highly technical test in the framework of funding decisions. They set out to answer the following three questions related to the reliability, validity, and fairness of the funding procedure. Is the funding procedure reliable? Specifically, they investigated the extent of agreement among reviewers on the quality of grant applications and whether the proposed research project should be funded. Second, they investigated if the procedure was valid; that is, do ratings correlate with scientific performance measures applied after funding? Finally, they wondered if the procedure was fair. This means: are there external factors that affect the decision-making process, but that have nothing to do with the quality of the grant proposal? Because researchers have, for most applications, only data ex-post, namely for those that are actually funded, they applied a novel method where non-funded proposals are considered as missing data, and, statistically, taken into account in this way. They applied their methods to real data from the FWF (Austrian Science Fund). From their experiment, they came, at least in this particular case, to some recommendations for testing peer review, such as increasing the number of ex-post reviewers from one to two, and these must be different persons than those that acted as ex-ante reviewers; final decisions should be made in different meetings, each related to one large discipline (not in one big meeting for all fields). As reliability, validity, and fairness turned out to be closely related, they recommend that in future studies, these aspects should, as far as possible, be examined together. Finally, they stated that one should make a distinction between potential bias and real bias. Potential bias refers to the ex-ante evaluation of project submissions, for example, potential bias between male and female applicants, while real bias (dealing with fairness) takes the actual performance, ex-post, into account.

In a study in cooperation with the Australian Research Council, Frijters and Torgler (2019) critically evaluated peer reviews of grant applications and the potential biases associated with applicants, assessors, and their interactions. They found that peer reviews lacked reliability. They observed a major systematic bias and even unreliability, and invalid ratings by assessors nominated by the applicants themselves. To explore how reviewers assign ratings to the applications they review and whether there is consistency in the reviewers’ evaluation of the same application in the U.S. National Institutes of Health (NIH), Pier et al. (2018) replicated all aspects of the NIH peerreview process. They found low levels of agreement among reviewers in evaluations of the same grant applications, not only in terms of the preliminary rating that they assigned, but also in terms of the number of strengths and weaknesses that they identified. Yang (2003) sent a set of ten proposals to three sets of reviewers after the first round of review and found that the peer-review ratings of the proposal ranked in the middle changed significantly, and the only funded proposal was evaluated as mediocre in one set of reviewers. This result indicated that the inter-rater problem also existed in the NSFC review system.

Existing research on the success of peer review panels has focused on understanding whether there is a correlation between good peer-review scores and successful research outcomes, but all in all, these studies yielded mixed results. Li and Agha (2015), for instance, studied whether NIH reviewers yielded high value-added reviews, meaning that differences in scores given to funded grant applications predict differences in subsequent research output, controlling for previous accomplishments of applicants. Their research indicated that this was indeed the case. Specifically, they found that a one-standard-deviation worse peer review score among awarded grants was associated with 15% fewer citations, 7% fewer publications, and 14% fewer patents, proving the effectiveness of the NIH peer-review system. However, some parts of their findings have been questioned by Fang et al. (2016), who, using a subset of the data used by Li and Agha, pointed out that their analysis confirmed that although peer review has some ability to discriminate between the quality of proposals, this does not extend to the critical range of percentile scores.

Testing peer review for funding should be part of the evaluation of a country’s research system. However, this would naturally lead to international comparisons, which are notoriously difficult to perform as inputs and outputs are difficult to measure.

Assuming that members of the panel are professional scientists and/or administrators of scientific institutes (not specialists working in industry), should they be paid extra or sit on a panel just part of their job? Could they, perhaps, be “paid” by points that can be used for promotion? Or, in the case of university professors, by reduced teaching obligations?

A difficult question related to peer review is whether reviewers should be paid, especially for proposals that need a lot of expertise and time to evaluate. In mathematics – a field in which it can take weeks of study before anyone can fully understand and appreciate the contents of a proposal– some colleagues think that fair compensation would be appropriate (Fried, 2007).

A related question is whether reviewers, instead of being paid, are recognized as contributors. This question is related to the expected (or required) role of a scientist at a university, a company, a national or international research institute, or other public or private research-related enterprises, such as a think tank or quality news magazine. The second Publons report (Hayes & Hardcastle, 2019) contains the results of a large survey of scientists and funders. It sketches the framework in which peer review takes place and points out the continued relevance of the peer review process. One of the conclusions is that researchers are dissatisfied with the lack of recognition of their review work. They believed that greater recognition by reviewers would improve the process. Although cash payments may seem an attractive driver of reviewer participation, this report found that cash payments are not a significant means of reviewer motivation.

Hochberg et al. (2009) describe the so-called tragedy of reviewer commons. This expression refers to the possible misuse of colleagues as reviewers. If everyone were to misuse the time and efforts of colleagues (in the role of reviewers), no one would want to review anymore. This is especially true when reviewers do not receive any credit or monetary rewards for their work. Problems in this respect include authors targeting inappropriate journals or funding organizations, reviewer fatigue, individuals not bearing their share of the overall refereeing responsibility resulting in overburdened colleagues who do take responsibility, and rejected authors not taking reviewers’ comments into account when resubmitting their manuscript or project.

Before discussing the lack of transparency, we first state what is meant by the expression “transparency in peer review for funding.” According to Meadmore (2020):

Transparency in peer review for research funding refers to the level of openness that the peer review processes provide for recommending research to be funded, who makes the decisions, how and when the decisions are made and the extent to which this is made clear to all involved.

This definition includes several aspects, mostly related to the openness of the different steps involved in the review procedure. Such openness would open up the possibility for outsiders to study the quality of peer review, starting with simple data collection about the time taken to review and the time taken to come to a consensus decision.

We note that the topics of transparency and openness, and their relationship, crop up on different occasions in this article.

Interdisciplinary projects often have specific problems. Besides the lack of institutional appreciation, a general bias in the evaluation as evaluation standards are often discipline-based (De Rijcke et al., 2016; Mallard et al., 2009). Bromham et al. (2016) show that funding often has a bias against interdisciplinarity. If the review panels consist of disciplinary experts, this result is no surprise. Indeed, as far as we know, there does not exist a worldwide database in which suitable interdisciplinary experts can be found for assessment purposes.

What happens if the proposals are multi-disciplinary? In Hayes and Hardcastle (2019), an anonymous survey respondent wrote:

Nobody ever wants to pitch in to review interdisciplinary grant proposals because everyone thinks it’s outside of their area. That suffocates all attempts to think outside the box.

During the period 2019-2024, based on the “attributes” of scientific problems, the NSFC experimented with allocating its funding to the four categories of scientific problems, where interdisciplinary was one of the categories. However, as both the number of proposals and funded projects in the interdisciplinary category were significantly lower than those in the other categories (Yang et al., 2024), the NSFC canceled this category in 2024.

Although solving global problems such as climate change and saving biodiversity needs input from all over the globe, international collaborations are, by their nature, more difficult to realize than national collaborations. Moreover, it is a fact that many funding agencies are national, measuring their “results” on the influence on local economies and their own population.

Funding agencies and researchers often call for more studies on the structure and organization of peer review in order to ensure that funding decisions are made more scientifically (De Vrieze, 2017). However, such research requires access to data, which is typically kept confidential, making systematic investigation nearly impossible. Squazzoni et al. (2020) argued that, as custodians of the scholarly record, publishers, independent journals, funders, and learned societies have a responsibility to engage in data-sharing and to support the development of infrastructures that strengthen the peer review system. They propose the establishment of a broad data-sharing framework to enable systematic research in this area.

To foster institutional collaboration in sharing data on diverse review formats, a group of scholars, publishing professionals, and funders launched a European Union–funded project called PEERE, supported by COST (the European Cooperation in Science and Technology). In 2017, PEERE released a protocol for sharing peer review data that carefully addresses ethical considerations, responsible data management, protection, and privacy (Squazzoni et al., 2020).

In addition, the Transparency and Openness Promotion (TOP) Guidelines highlight the importance of institutional commitments to facilitate meta-research on the effectiveness and integrity of peer review practices (Lee & Moher, 2017).

In addition to playing a summative role, peer review must also play a formative role. It has become unthinkable that the answer to a submitted proposal is simple yes/funded or no/not funded, even when wrapped in a polite letter. The peer review process must be fair, and the act of submitting a proposal should be a learning opportunity for all involved.

An early career researcher (Parrilla-Gutierrez, 2021) complained about the lack of feedback on his applications. Rejections of articles submitted to a journal or of projects submitted as a grant proposal are part of a scientist’s life. Yet, given the large amount of time invested in a grant proposal, and the many favors one has to ask for from other academics and sometimes even from commercial suppliers, to write letters of recommendation and reduce prices to fit within an appropriate budget, it is rather disrespectful as this colleague wrote if a letter of rejection does not contain review reports with helpful suggestions for improvements.

Although articles have been written with titles such as “Secrets to writing a winning grant” (Sohn, 2020), that kind of information is nonexistent in our eyes. However, using feedback from colleagues, family and friends (representing the world outside academia), and funding agencies to improve the proposals is surely a way to increase the probability of winning a grant.

Interactive peer review is a valuable form of feedback. Even when submitters and reviewers remain anonymous to each other, it is beneficial to allow submitters to clarify specific points in response to reviewer questions. These clarifications do not constitute changes to the proposal but are added before the panel’s deliberations. If time permits, panel members should also have the opportunity to request clarification.

Already applied by some agencies, such as the Swiss National Science Foundation, is the use of a (partial) random selection procedure (Chawla, 2021). Indeed, when experts consider two or more proposals of equal value, then resorting to a lottery makes sure that no bias creeps in because it is particularly at this point that this might occur. It often happens that between proposals that definitely should be funded and those that certainly should not, there is a grey zone (Bendiscioli & Garfinkel, 2021). Especially for proposals in this zone, randomization might be a good solution. Hence, combining a lottery system with some aspects of peer review, leading to a modified lottery system, may well be part of a solution to solve the problems related to the contemporary peer review system for funding allocation (De Peuter & Conix, 2022). Bendiscioli and Garfinkel (2021) noted that this brings transparency to the selection process of otherwise equally deserving proposals. Solving the “grey zone dilemma” by a lottery is an acknowledgment of the limits of precision achievable by classical peer review (Barlösius et al., 2023). However, a lottery considers projects as independent units, while in reality, scientific projects are parts of an interconnected system of conceptual, experimental, and technical practices (Bedessem, 2020). For this reason, Bedessem (2020) criticizes the lottery system.

Currently, it is a common problem for funding agencies to spend more time reviewing grants while success rates are dropping to demoralizing lows. This contributes to an inefficient science system in which a lot of energy is wasted. In response, a controversial and innovative suggestion was made by Bollen et al. (2014). These colleagues proposed a crowd-based, decentralized funding model, using the wisdom of the entire scientific community. In their proposal, each scientist receives a fixed and equal amount of money from a national funding agency. However, each scientist is required to pass on a fixed amount (say 50%) of their previous year’s money to other scientists whom they think would make the best use of the money. The details of this procedure can be found in the original article. If this proposal is accepted, it would drastically reduce the costs of funding agencies. Note that here, the term “peers” refers to the whole community of scientists. However, this procedure could easily become a popularity contest, and as such, could favor more visible fields, scientists, or teams with well-known successes in the past. Ideally, one should vote for projects, not for people.

The crowd-based funding model is an example of a decentralized funding model. Discussing decentralized models, Bedessem (2020) poses the question, “Who should have the right to vote?” The chosen solution reflects a perspective on participation in science. Would it be a balanced group consisting of scientists, direct stakeholders, citizens, or their chosen representatives? This choice is an important political and ethical issue.

Finding suitable reviewers is often a difficult and time-consuming task; recall that many potential reviewers decline (Gallo et al., 2020). A radical solution for this, proposed by Merrifield and Saari (2009) and discussed in (Guthrie, 2019), is to require submitters to review several other proposals with timely delivery, mandatory for their own application to be considered. As the review tasks are distributed among the applicants, this mechanism is called distributed peer review (DPR).

Distributed peer review has obvious advantages for funders. It reduces the burden to identify reviewers, and timely review delivery is guaranteed. Moreover, more reviewers than usual are available, so extreme opinions have less influence. In this way, DPR is more suitable for selecting proposals that can reach a consensus across the community. The main disadvantage is that submitters must perform more work than they would otherwise.

As applicants and reviewers act in the same review system in the DPR, reciprocal reviews are unavoidable. Because reciprocal reviews can exchange benefit and hence are strictly forbidden in a typical review system, mechanisms are needed to ensure that no gaming takes place in the system, a point that is discussed in Guthrie (2019) and Pearson (2025). Moreover, researchers interact overtly and covertly, must be a factor of consideration. Pearson (2025) and Brainard (2025) report that distributed peer review is now being tested by the Volkswagen Foundation and UK Research and Innovation (UKRI), and that the gains seem huge in terms of speed and efficiency.

Wang et al. (2011) published a proposal for a so-called heuristic review mechanism. This mechanism is not meant to deal with all types of funding. It is specifically constructed for pioneering research, that is, non-conventional, highly exploratory, possibly groundbreaking, and relatively high-risk basic research. It has the potential to be transformative. The proposed method provides an answer to the claim that peer reviews tend to be conservative in China, but also in the USA (Packalen & Bhattacharya, 2020). Topics are determined by the NSFC. A leader and reviewers for a given topic are selected. Among these are multi-disciplinary scientists and experts who have comprehensive science perspectives. The leader must help to create an open-minded communication atmosphere. The key step is the organization of a forum in which candidates present their innovative ideas, even if they do not yet have data to support them. The review panel can directly ask questions, leading to the information it deems necessary for selection purposes. A panel discussion will include brainstorming, in-depth discussions, and debates. After these interactions, reviewers and the leader discuss, vote, and jointly identify outstanding candidates. Finally, the selected candidates were supposed to write and submit a full research plan with almost certainty to be approved by the NSFC.

A somewhat similar idea has been implemented in Norway, informally named a “research sandpit,” and more formally Idélab’ (idea lab). It turned out that this model could indeed be useful to generate multi-disciplinary research as part of a multifaceted approach to funding scientific research (Maxwell & Benneworth, 2018).

The National Science Foundation of China (NSFC) is working on the profound reform of the review system for funding applications, aiming at a more efficient and fairer evaluation system. In this framework, three major tasks must be accomplished: identifying funding categories, improving evaluation mechanisms, and optimizing the layout of research areas for the funding system (Tang et al., 2021). To fulfill its reform aim, the NSFC is trying to optimize its review procedure to make interactions between the funder, the applicants, and the reviewers possible. In this new review procedure, the applicants can evaluate whether the scientific comments of reviewers are helpful or not and state the reason why the scientific comments are or are not helpful. For the moment, the evaluations from the applicants do not influence the funding decision, but they will help the funders to evaluate the reviewers’ responsibility and credibility. Based on these evaluations, funders expect to be able to select responsible and credible reviewers.

Liu and Rousseau (2023) proposed an approach in which it is allowed that a reviewer of a proposal joins the team (those scientists who submitted the proposal). This possibility acts as an incentive where all parties (submitters, reviewers, and funders) and science itself may benefit. This level of interaction between the reviewer and the proposer can become so close that the reviewer is allowed to join the original team for the benefit of science.