References
- Ayoubi, C., Pezzoni, M., & Visentin, F. (2021). Does it pay to do novel science? The selectivity patterns in science funding. Science and Public Policy, 48(5), 635-648.
- Azoulay, P., Zivin, J. S. G., & Manso, G. (2011). Incentives and creativity: evidence from the academic life sciences. RAND Journal of Economics, 42(3), 527-554.
- Baccini, A., & Re, C. (2025). Is the panel fair? Evaluating panel compositions through network analysis. The case of research assessments in Italy. Scientometrics, 130(3), 2093-2135. https://doi.org/10.1007/s11192-025-05255-9
- Barlösius, E., Paruschke, L., & Philipps, A. (2023). Peer review’s irremediable flaws: Scientists’ perspectives on grant evaluation in Germany. Research Evaluation, 32(4), 623-634.
- Bedessem, B. (2020). Should we fund research randomly? An epistemological criticism of the lottery model as an alternative to peer review for the funding of science. Research Evaluation, 29(2), 150-157.
- Bendiscioli, S. (2019). The troubles with peer review for allocating research funding. EMBO Reports, 20: e49472.
- Bendiscioli, S., Firpo, T., Bravo-Biosca, A., Czibor, E., Garfinkel, M., Stafford, T., Wilsdon, J., & Woods, H. B. (2021). The experimental research funder’s handbook (RoRI Working Paper No.6). https://doi.org/10.6084/m9.figshare.19459328.v1
- Bendiscioli, S., & Garfinkel, M. (2021). Dealing with the limits of peer review with innovative approaches to allocating research funding. https://www.embo.org/documents/science_policy/peer_review_report.pdf
- Bollen, J., Crandall, D., Junk, D., Ding, Y., & Börner, K. (2014). From funding agencies to scientific agency. EMBO Reports, 15(2),131-333.
- Bornmann, L., Wallon, G., & Ledin, A. (2008). Does the committee peer review select the best applicants for funding? An investigation of the selection process for two European molecular biology organization programmes. PLoS ONE, 3(10), e3480.
- Brainard, J. (2025). Should grant applicants judge competitors’ proposals? Science, 389(6756), 120-120.
- Bromham, L., Dinnage, R., & Hua, X. (2016). Interdisciplinary research has consistently lower funding success. Nature, 534, 684-687.
- Chawla, D. S. (2021, May 6). Swiss funder draws lots to make grant decisions. Nature Career News. https://doi.org/10.1038/d41586-021-01232-3
- Chen, L. (2021). Non-consensus research and the improvement of original innovation ability in university. Science & Technology Progress and Policy, 38(12), 19-24.
- Clements, J. C. (2020). Don’t be a prig in peer review. Nature, 585, 472-472.
- Cole, S., Cole, J. R., & Simon, G. A. (1981). Chance and consensus in peer-review. Science, 214, 881-886.
- Cole, S., Rubin, L., & Cole, J. R. (1977). Peer review and the support of science. Scientific American, 237(4), 34-41.
- Coveney, J., Herbert, D. I., Hill, K., Mow, K. E., Graves, N., & Barnett, A. (2017). “Are you siding with a personality or the grant proposal?” Observations on how peer review panels function. Research Integrity and Peer Review, 2, art. 19.
- Csiszar, A. (2016). Troubled from the start. Nature, 532, 306-308.
- De George, R.T., & Woodward, F. (1994). Ethics and manuscript reviewing. Journal of Scholarly Publishing, 25(3), 133-145.
- De Peuter, S., & Conix, S. (2022). The modified lottery: Formalizing the intrinsic randomness of research funding. Accountability in Research, 29(5), 324-345.
- De Rijcke, S., Wouters, P. F., Rushforth, A. D., Franssen, T. P., & Hammarfelt, B. (2016). Evaluation practices and effects of indicator use – A literature review. Research Evaluation, 25(2), 161-169.
- De Vrieze, J. (2017). Funders groan under growing review burden. Science, 357(6349), 343-343.
- DORA. (2012). San Francisco declaration on research assessment. DORA-ASCB. http://www.ascb.org/dora
- Fang, F. C., Bowen, A., & Casadevall, A. (2016). NIH peer review percentile scores are poorly predictive of grant productivity. eLife, 5, e13323.
- Fasio, L. (2025). What you can do when your research funding is axed? Nature, 641(8064), 824-824.
- Feliciani, T., Morreau, M., Luo, J., Lucas, P., & Shankar, K. (2022). Designing grant-review panels for better funding decisions: lessons from an empirically calibrated simulation model. Research Policy, 51, art. 104467.
- Fried, M. (2007). Should journals compensate referees? Notices of the AMS, May, p. 589.
- Frijters, P., & Torgler, B. (2019). Improving the peer review process: a proposed market system. Scientometrics, 119(2), 1285-1288.
- Gallo, S. A., Thompson, L. A., Schmaling, K. B., & Glisson, S. R. (2020). The participation and motivations of grant peer reviewers: A comprehensive survey. Science and Engineering Ethics, 26, 761-782. https://doi.org/10.1007/s11948-019-00123-1
- Guthrie, S. (2019). Innovating in the Research Funding Process: Peer Review Alternatives and Adaptations. AcademyHealth.
- Guthrie, S., Ghiga, I., & Wooding, S. (2018). What do we know about grant peer review in the health sciences? F1000Research, 6, 1335. https://doi.org/10.12688/f1000research.11917.2
- Hayes, M., & Hardcastle, J. (2019). Grant Review in Focus. London: Publons.
- HHMI. (2024). About HHMI Nobel Laureates. https://www.hhmi.org/about/nobel-laureates
- Hochberg, M. E., Chase, J. M., Gotelli, N. J., Hastings, A., & Naeem, S. (2009). The tragedy of the reviewer commons. Ecology Letters, 12(1), 2-4.
- Hojat, M., Gonella, J. S., & Caelleigh, A. S. (2003). Impartial Judgment by the “Gatekeepers” of Science: Fallibility and Accountability in the Peer Review Process. Advances in Health Sciences Education, 8(1), 75-96.
- Hoppe, T. A., Litovitz, A., Willis, K. A., Meseroll, R. A., Perkins, M. J., Hutchins, B. I., Davis, A. F., Lauer, M. S., Valantine, H. A., Anderson, J. M., & Santangelo, G. M. (2019). Topic choice contributes to the lower rate of NIH awards to African-American/black scientists. Science Advances, 5, eaaw7238.
- Horbach, S. P. J. M. (2021). No time for that now! Qualitative changes in manuscript peer review during the Covid-19 pandemic. Research Evaluation, 30(3), 231-239.
- Hug, S. E., & Ochsner, M. (2022). Do peers share the same criteria for assessing grant applications? Research Evaluation, 31(1), 104-117. https://doi.org/10.1093/reseval/rvab034
- Jacobson, S. H. (2017). NSF: Peer review peer-review processes. ORMS Today, 44(6), 12-13.
- Johnson, M. H., Cohen, J., & Grudzinskas, G. (2012). The uses and abuses of bibliometrics. Reproductive Biomedicine Online, 24(5), 485-486.
- Kolev, J., Fuentes-Medel, Y., & Murray, F. (2020). Gender differences in scientific communication and their impact on grant funding decisions. AEA Papers and Proceedings, 110, 245-249.
- Kummeling, H., Kluijtmans, M., & Miedema, F. (2024). The University in Transition. Utrecht: Utrecht University.
- Lee, C. J., & Moher, D. (2017). Promote scientific integrity via journal peer review data. Science, 357(6348), 256-257.
- Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2-17.
- Li, D. (2017). Expertise versus bias in evaluation: Evidence from the NIH. American Economic Journal: Applied Economics, 9(2), 60-92.
- Li, D., & Agha, L. (2015). Big names or big ideas: Do peer-review panels select the best science proposals? Science, 348(6233), 434-438.
- Li, X., Zhao, Y., Hao, H., Zheng, Z., & Yang, L. (2024). Policy exploration of original innovation in basic research: From non-consensus innovation to original exploratory program. Bulletin of National Science Foundation of China, 38(2), 320-327.
- Liu, Y. X., & Rousseau, R. (2023). A proposal for the peer review procedure for funding decisions. Scientometrics, 128(1), 861-865. https://doi.org/10.1007/s11192-022-04538-9
- Luo, J. W., Ma, L., & Shankar, K. (2021). Does the inclusion of non-academic reviewers make any difference for grant impact panels? Science and Public Policy, 48(6), 763-775. https://doi.org/10.1093/scipol/scab046
- Ma, L., Luo, J. W., Feliciani, T., & Shankar, K. (2020). How to evaluate ex-ante impact of funding proposals? An analysis of reviewers’ comments on impact statements. Research Evaluation, 29(4), 431-440.
- Mallard, G., Lamont, M., & Guetzkow, J. (2009). Fairness as appropriateness: Negotiating epistemological differences in peer review. Science, Technology & Human Values, 34(5), 573-606.
- Marhoffer, E. A., Ein-Alshaeba, S., Grimshaw, A. A., Holleck, J. L., Rudikoff, B., Bastian, L., & Gunderson, C. G. (2024). Gender disparity in full professor rank among academic physicians: A systematic review and metaanalysis. Academic Medicine, 99(7), 801-809. https://doi.org/10.1097/ACM.0000000000005695
- Marsh, H. W., & Ball, S. (1981). Interjudgmental reliability of reviews for the Journal of Educational Psychology. Journal of Educational Psychology, 73(6), 872-880.
- Maxwell, K., & Benneworth, P. (2018). The construction of new scientific norms for solving Grand Challenges. Palgrave Communications, 4(1), 52.
- Meadmore, K. (2020, Oct. 29). Do we need transparency for trust in peer review for research funding? NIHR Blog. https://www.nihr.ac.uk/blog/do-we-need-transparency-for-trust-in-peer-review-for-research-funding/25674
- Meadmore, K., Fackrell, K., Recio-Saucedo, A., Bull, A., Fraser, S. D. S., & Blatch-Jones, A. (2020). Decisionmaking approaches used by UK and international health funding organisations for allocating research funds: A survey of current practice. PLoS One, 15(11), e0239757.
- Merrifield, M. R., & Saari, D. G. (2009). Telescope time without tears: A distributed approach to peer review. Astronomy and Geophysics, 50(4), 16-20.
- Merton, R. K. (1968). The Matthew effect in science. Science, 159, 56-63.
- Moussian, B. (2016). Taking peer review seriously. EMBO Reports, 17(5), 617-617.
- Mutz, R., Bornmann, L., & Daniel, H.-D. (2015). Testing for the fairness and predictive validity of research funding decisions: A multilevel multiple imputation for missing data approach using ex-ante and ex-post peer review evaluation data from the Austrian Science Fund. Journal of the Association for Information Science and Technology, 66(11), 2321-2330.
- Nature. (2021). Research evaluation must change after the pandemic. Nature, 591(7948), 7.
- Nicholas, N., Watkinson, A., Jamali, H. R., Herman, E., Tenopir, C., Volentine, R., Allard, S., & Levine, K. (2015). Peer review: Still king in the digital age. Learned Publishing, 28(1), 15-21.
- NSFC. (1986). Application Preparation and Submission. https://www.nsfc.gov.cn/english/site_1/funding/E1/2022/01-12/25901.html
- Oxley, K., & Gulbrandsen, M. (2024). Variability and negligence: Grant peer review panels evaluating impact ex ante. Science and Public Policy, scae081. https://doi.org/10.1093/scipol/scae081
- Packalen, M., & Bhattacharya, J. (2020). NIH funding and the pursuit of edge science. Proceedings of the National Academy of Sciences of the United States of America, 117(22), 12011-12016.
- Parrilla-Gutierrez, J. M. (2021, Feb. 17). Dear grant agencies: Tell me where I went wrong. Nature Career Column. https://doi.org/10.1038/d41586-021-00444-x
- Pearson, H. (2025). How to speed up peer review: Applicants mark one another. Nature, 643(8071), 313-314.
- Peters, D., & Ceci, S. (1982). Peer-review practices of psychological journals: The fate of submitted articles, submitted again. Behavioral Brain Science, 5, 187-255.
- Pier, E. L., Brauer, M., Filut, A., Kaatz, A., Raclaw, J., Nathan, M. J., … & Carnes, M. (2018). Low agreement among reviewers evaluating the same NIH grant applications. Proceedings of the National Academy of Sciences of the United States of America, 115(12), 2952-2957.
- Price, S., & Flach, P. A. (2017). Computational support for academic peer review: A perspective from artificial intelligence. Communications of the ACM, 60(3), 70-79.
- Ramos, A., & Sarrico, C. S. (2016). Past performance does not guarantee future results: Lessons from the evaluation of research units in Portugal. Research Evaluation, 25(1), 94-106.
- Rosenthal, R., & Jacobson, L. (1992). Pygmalion in the classroom: Teacher expectation and pupils’ intellectual development (Newly expanded ed.). Crown House Publishing.
- Rousseau, R. (2021). Do COVID-related articles pass through the fast lane? The case of the International Journal of Infectious Diseases. In W. Glänzel, S. Heeffer, P. Chi, & R. Rousseau (Eds). Proceedings of the 18th International Conference on Scientometrics & Informetrics (ISSI 2021) (pp. 1533-1534). International Society for Scientometrics and Informetrics.
- Rousseau, R., Egghe, L., & Guns, R. (2018). Becoming metric-wise: A bibliometric guide for researchers. Chandos Publishing (Elsevier).
- Rousseau, R., Guns, G., Jakaria Rahman, A. I. M., & Engels, T. C. E. (2017). Measuring cognitive distance between publication portfolios. Journal of Informetrics, 11(2), 583-594.
- Rousseau, R., & Rousseau, S. (2021). Bibliometric techniques and their use in business and economics research. Journal of Economic Surveys, 35(95), 1428-1451.
- Severin, A., & Chataway, J. (2021). Purposes of peer review: A qualitative study of stakeholder expectations and perceptions. Learned Publishing, 34, 144-155.
- Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178-182.
- Sohn, E. (2020). Secrets to writing a winning grant. Nature, 577, 133-135.
- Squazzoni, F., Ahrweiler, P., Barros, T., Bianchi, F., Birukou, A., Blom, H. J. J., … & Willis, M. (2020). Unlock ways to share data on peer review. Nature, 578(7796), 512-514.
- Squazzoni, F., & Gandelli, C. (2012). Saint Matthew strikes again: An agent-based model of peer review and the scientific community structure. Journal of Informetrics, 6(2), 265-275.
- State Council of the People’s Republic of China. (2024). Regulations of the National Natural Science Foundation of China. https://www.nsfc.gov.cn/publish/portal0/tab471/info93942.htm
- Strauss, D. H., White, S. H., & Bierer, B. E. (2021). Justice, diversity, and research ethics review. Science, 371(6535), 1209-1211.
- Tang, S., Li, W., Liu, Q., & Liu, Y. (2021). RCC: NSFC’s new review mechanism: Its missions and challenges. In W. Glänzel, S. Heeffer, P.-S. Chi, & R. Rousseau (Eds.), Proceedings of ISSI2021 (pp. 1543-1544). KU Leuven University Press.
- Thorngate, W., & Chowdhury, W. (2014). By the numbers: Track record, awed reviews, journal space, and the fate of talented authors. In B. Kamiński & G. Koloch (Eds.), Advances in Social Simulation (pp. 177-188). Springer-Verlag.
- Triggle, C. R., & Triggle, D. J. (2007). What is the future of peer review? Why is there fraud in science? Is plagiarism out of control? Why do scientists do bad things? Is it all a case of: “All that is necessary for the triumph of evil is that good men do nothing?”. Vascular Health and Risk Management, 3(1), 39-46.
- Turner, S., Bull, A., Chinnery, C., Hinks, J., Mcardle, N., Moran, R., Payne, H., Woodford Guegan, E., Worswick, L., & Wyatt, J. C. (2018). Evaluation of stakeholder views on peer review of NIHR applications for funding: A qualitative study. BMJ Open, 8, e022548. Doi:10.1136/bmjopen-2018-022548
- UNESCO. (2024). UNESCO Call to Action: Closing the gender gap in science. https://unesdoc.unesco.org/ark:/48223/pf0000388641
- Veugelers, R., Wang, J., & Stephan, P. (2025). Do funding agencies select and enable novel research: evidence from ERC. Economics of Innovation and New Technology. https://doi.org/10.1080/10438599.2025.2486344
- Wang, J., & Shah, N. B. (2019). Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings. In Proceedings of the 18th International Conference on Autonomous Agents and Multi-Agent Systems (pp. 864-872).
- Wang, J., Veugelers, R., & Stephan, P. E. (2017). Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy, 46(8), 1416-1436. https://doi.org/10.1016/j.respol.2017.06.006.
- Wang, Y., Li, X., & Zheng, Y. (2011). The interactive heuristic review mechanism: A new method of assessing pioneering research projects of the National Natural Science Foundation of China. Research Evaluation, 20(4), 267-274.
- Ware, M. (2008). Peer review: Benefits, perceptions and alternatives (Summary Papers 4). Publishing Research Consortium.
- Wilson, R., & Lancaster, J. (2006). ‘Referee factor’ would reward a vital contribution. Nature, 441(7095), 812-812.
- Woolston, C. (2021). UK funder’s data point to uneven playing field. Nature, 591(7851), 683-684.
- Yang, H., Hao, H., Zhao, Y., Guo, X., Zheng, Z., Yang, L., & Wang, Y. (2024). Overview of proposals received by National Natural Science Foundation of China in 2023 during the intensive reception period. Bulletin of the National Science Foundation of China, 38(1), 2-7.
- Yang, L. (2003). Second peer review of foundation project—a case study. Bulletin of National Natural Science Foundation of China, 17(1), 50-53.
- Zhao, P., Sun, C., & Hao, H. (2025). The NSFC will experimentally implement key non-consensus projects. National Natural Science Foundation of China. https://www.nsfc.gov.cn/publish/portal0/tab440/info95140.htm
- Zhao, Y., Hao, H., Li, X., Zheng, Z., & Yang, L. (2023). Inspiration from the NSFC’s funding of original basic research: A case study on the pilot implementation of the original exploration program. China Soft Science Magazine, 10, 9-20.
- Zhou, Z., & Zhao, W. (2019). Funding system reform for excellence in science: An interview with Jinghai Li, President of NSFC. National Science Review, 6, 177-181.
- Ziman, J. (1968). Public Knowledge: An Essay Concerning the Social Dimension of Science. Cambridge University Press.