Evaluating grant proposals: lessons from using metrics as screening device

Guba, Katerina; Zheleznov, Alexey; Chechik, Elena

doi:10.2478/jdis-2023-0010

Abstract

Purpose

This study examines the effects of using publication-based metrics for the initial screening in the application process for a project leader. The key questions are whether formal policy affects the allocation of funds to researchers with a better publication record and how the previous academic performance of principal investigators is related to future project results.

Design/methodology/approach

We compared two competitions, before and after the policy raised the publication threshold for the principal investigators. We analyzed 9,167 papers published by 332 winners in physics and the social sciences and humanities (SSH), and 11,253 publications resulting from each funded project.

Findings

We found that among physicists, even in the first period, grants tended to be allocated to prolific authors publishing in high-quality journals. In contrast, the SSH project grantees had been less prolific in publishing internationally in both periods; however, in the second period, the selection of grant recipients yielded better results regarding awarding grants to more productive authors in terms of the quantity and quality of publications. There was no evidence that this better selection of grant recipients resulted in better publication records during grant realization.

Originality

This study contributes to the discussion of formal policies that rely on metrics for the evaluation of grant proposals. The Russian case shows that such policy may have a profound effect on changing the supply side of applicants, especially in disciplines that are less suitable for metric-based evaluations. In spite of the criticism given to metrics, they might be a useful additional instrument in academic systems where professional expertise is corrupted and prevents allocation of funds to prolific researchers.

References

Aagaard K., Kladakis A., & Nielsen M. W. (2020). Concentration or dispersal of research funding? Quantitative Science Studies, 1(1): 117–149., from https://doi.org/10.1162/qss_a_00002.
Search in Google Scholar Back to article
Abramo G., Cicero T., & D’Angelo C. A. (2013). Individual research performance: A proposal for comparing apples to oranges. Journal of Informetrics, 7(2), 528–539, from https://doi.org/10.1016/j.joi.2013.01.013.
Search in Google Scholar Back to article
Auranen O., & Nieminen M. (2010). University research funding and publication performance—An international comparison. Research Policy, 39(6), 822–834, from https://doi.org/10.1016/j.respol.2010.03.003.
Search in Google Scholar Back to article
Azoulay P., & Li D. (2020). Scientific grant funding. In Innovation and Public Policy. University of Chicago Press.
Search in Google Scholar Back to article
Batista P. D., Campiteli M. G., & Kinouchi O. (2006). Is it possible to compare researchers with different scientific interests? Scientometrics, 68(1), 179–189, from https://doi.org/10.1007/s11192-006-0090-4.
Search in Google Scholar Back to article
Batygin G. S. (2001) The invisible border: Grant support and restructuring the scientific community in Russia. Intellectual News, 9(1), 70–74, from https://doi.org/10.1080/15615324.2001.10426712
Search in Google Scholar Back to article
Beckert J. (2019). Shall I publish this auf Deutsch or in English? Sociologica, 13(1), 3–7, from https://doi.org/10.6092/issn.1971-8853/9378.
Search in Google Scholar Back to article
Bertoni M., Brunello G., Checchi D., & Rocco L. (2021). Where do I stand? Assessing researchers’ beliefs about their productivity. Journal of Economic Behavior & Organization, 185, 61–80, from https://doi.org/10.1016/j.jebo.2021.02.025.
Search in Google Scholar Back to article
Bornmann L. (2011). Peer review and bibliometric: potentials and problems. In: Shin, J., Toutkoushian R., Teichler U. (eds) University Rankings. The Changing Academy – The Changing Academic Profession in International Comparative Perspective, vol 3. Springer, Dordrecht, 145-164, from https://doi.org/10.1007/978-94-007-1116-7_8
Search in Google Scholar Back to article
Campbell D., Picard-Aitken M., Côté G., Caruso J., Valentim R., Edmonds S., (2010). Bibliometrics as a performance measurement tool for research evaluation: The case of research funded by the National Cancer Institute of Canada. American Journal of Evaluation, 31(1), 66–83, from https://doi.org/10.1177/1098214009354774.
Search in Google Scholar Back to article
Chankseliani M., Lovakov A., & Pislyakov V. (2021). A big picture: bibliometric study of academic publications from post-Soviet countries. Scientometrics, 126(10), 8701-8730, from https://link.springer.com/article/10.1007/s11192-021-04124-5
Search in Google Scholar Back to article
Clark B. Y., & Llorens J. J. (2012). Investments in Scientific Research: Examining the funding threshold effects on scientific collaboration and variation by academic discipline. Policy Studies Journal, 40(4), 698–729, from https://doi.org/10.1111/j.1541-0072.2012.00470.x
Search in Google Scholar Back to article
Denisova-Schmidt E. V. (2023). Academic dishonesty at Russian universities: A historical overview. Universe of Russia, 32(1), 159-181, from https://doi.org/10.17323/1811-038X-2023-32-1-159-181
Search in Google Scholar Back to article
Donovan C., & Butler L. (2007). Testing novel quantitative indicators of research ‘quality’, esteem and ‘user engagement’: An economics pilot study. Research Evaluation, 16(4), 231–242, from https://doi.org/10.3152/095820207X257030
Search in Google Scholar Back to article
Fang F. C., Bowen A., & Casadevall A. (2016). NIH peer review percentile scores are poorly predictive of grant productivity. eLife, 5, e13323, from https://doi.org/10.7554/eLife.13323
Search in Google Scholar Back to article
Fedderke J. W., & Goldschmidt M. (2015). Does massive funding support of researchers work?: Evaluating the impact of the South African research chair funding initiative. Research Policy, 44(2), 467–482, from https://doi.org/10.1016/j.respol.2014.09.009
Search in Google Scholar Back to article
Gantman E. R., & Fernández Rodríguez, C. J. (2016). Literature segmentation in management and organization studies: The case of Spanish-speaking countries (2000–10). Research Evaluation, 25(4), 461–471, from https://doi.org/10.1093/reseval/rvv031
Search in Google Scholar Back to article
Gläser J. (2004). Why are the most influential books in Australian sociology not necessarily the most highly cited ones? Journal of Sociology, 40(3), 261–282, from https://doi.org/10.1177/1440783304046370
Search in Google Scholar Back to article
Gök A., Rigby J., & Shapira P. (2016). The impact of research funding on scientific outputs: Evidence from six smaller European countries. Journal of the Association for Information Science and Technology, 67(3), 715–730, from https://doi.org/10.1002/asi.23406
Search in Google Scholar Back to article
Graves N., Barnett A. G., & Clarke P. (2011). Funding grant proposals for scientific research: Retrospective analysis of scores by members of grant review panel. BMJ, 343, d4797, from https://doi.org/10.1136/bmj.d4797
Search in Google Scholar Back to article
Grimpe C. (2012). Extramural research grants and scientists’ funding strategies: Beggars cannot be choosers? Research Policy, 41(8), 1448–1460, from https://doi.org/10.1016/j.respol.2012.03.004.
Search in Google Scholar Back to article
Gush J., Jaffe A., Larsen V., & Laws A. (2018). The effect of public funding on research output: The New Zealand Marsden Fund. New Zealand Economic Papers, 52(2), 227–248, from https://doi.org/10.1080/00779954.2017.1325921.
Search in Google Scholar Back to article
Guskov A. E., Kosyakov D. V., & Selivanova I. V. (2018). Boosting research productivity in top Russian universities: The circumstances of breakthrough. Scientometrics, 117(2), 1053–1080, from https://doi.org/10.1007/s11192-018-2890-8.
Search in Google Scholar Back to article
Guthrie S., Ghiga I., & Wooding S. (2018). What do we know about grant peer review in the health sciences? F1000Research, 6(1335), from https://doi.org/10.12688/f1000research.11917.2
Search in Google Scholar Back to article
Győrffy B., Herman P., & Szabó I. (2020). Research funding: Past performance is a stronger predictor of future scientific output than reviewer scores. Journal of Informetrics, 14(3), 101050, from https://doi.org/10.1016/j.joi.2020.101050.
Search in Google Scholar Back to article
Győrffy B., Nagy A. M., Herman P., & Török, Á. (2018). Factors influencing the scientific performance of Momentum grant holders: An evaluation of the first 117 research groups. Scientometrics, 117(1), 409–426, from https://doi.org/10.1007/s11192-018-2852-1.
Search in Google Scholar Back to article
Hammarfelt B., & Rijcke S. de (2015). Accountability in context: Effects of research evaluation systems on publication practices, disciplinary norms, and individual working routines in the faculty of Arts at Uppsala University. Research Evaluation, 24(1), 63–77, from https://doi.org/10.1093/reseval/rvu029.
Search in Google Scholar Back to article
Hicks D., Tomizawa H., Saitoh Y., & Kobayashi S. (2004). Bibliometric techniques in the evaluation of federally funded research in the United States. Research Evaluation, 13(2), 76–86, from https://doi.org/10.3152/147154404781776446.
Search in Google Scholar Back to article
Hornbostel S., Böhmer S., Klingsporn B., Neufeld J., & Ins M. von (2009). Funding of young scientist and scientific excellence. Scientometrics, 79(1), 171–190, from https://doi.org/10.1007/s11192-009-0411-5.
Search in Google Scholar Back to article
Jacob B. A., & Lefgren L. (2011). The impact of research grant funding on scientific productivity. Journal of Public Economics, 95(9), 1168–1177, from https://doi.org/10.1177/0011392118807514
Search in Google Scholar Back to article
Južnič P., Pečlin S., Žaucer M., Mandelj T., Pušnik M., & Demšar F. (2010). Scientometric indicators: peer-review, bibliometric methods and conflict of interests. Scientometrics, 85(2), 429-441, from https://doi.org/10.1007/s11192-010-0230-8
Search in Google Scholar Back to article
Koch T., & Vanderstraeten R. (2019). Internationalizing a national scientific community? Changes in publication and citation practices in Chile, 1976–2015. Current Sociology, 67(5), 723–741, from https://doi.org/10.1177/0011392118807514.
Search in Google Scholar Back to article
Langfeldt L., Benner M., Sivertsen G., Kristiansen E. H., Aksnes D. W., Borlaug S. B., (2015). Excellence and growth dynamics: A comparative study of the Matthew effect. Science and Public Policy, 42(5), 661–675, from https://doi.org/10.1093/scipol/scu083
Search in Google Scholar Back to article
Langfeldt L., Reymert I., & Aksnes D. W. (2021). The role of metrics in peer assessments. Research Evaluation, 30(1), 112–126, from https://doi.org/10.1093/reseval/rvaa032.
Search in Google Scholar Back to article
Larivière V., & Sugimoto C. R. (2019). The Journal Impact Factor: A brief history, critique, and discussion of adverse effects. In W. Glänzel, H. F. Moed, U. Schmoch, & M. Thelwall (Eds.), Springer Handbooks. Springer Handbook of Science and Technology Indicators. Cham: Springer International Publishing, from https://doi.org/10.1007/978-3-030-02511-3_1
Search in Google Scholar Back to article
Laudel G. (2006). The art of getting funded: How scientists adapt to their funding conditions. Scienca and Public Policy (Science and Public Policy), 33(7), 489–504, from https://doi.org/10.3152/147154306781778777
Search in Google Scholar Back to article
Li D., & Agha L. (2015). Big names or big ideas: Do peer-review panels select the best science proposals? Science, 348(6233), 434–438, from https://doi.org/10.1126/science.aaa0185
Search in Google Scholar Back to article
López Piñeiro, C., & Hicks D. (2015). Reception of Spanish sociology by domestic and foreign audiences differs and has consequences for evaluation. Research Evaluation, 24(1), 78–89, from https://doi.org/10.1093/reseval/rvu030
Search in Google Scholar Back to article
Luukkonen T. (2012). Conservatism and risk-taking in peer review: Emerging ERC practices. Research Evaluation, 21(1), 48–60, from https://doi.org/10.1093/reseval/rvs001.
Search in Google Scholar Back to article
Maisano D. A., Mastrogiacomo L., & Franceschini F. (2020). Short-term effects of non-competitive funding to single academic researchers. Scientometrics, 123(3), 1261–1280, from https://doi.org/10.1007/s11192-020-03449-x.
Search in Google Scholar Back to article
Mali F., Pustovrh T., Platinovšek R., Kronegger L., & Ferligoj A. (2017). The effects of funding and co-authorship on research performance in a small scientific community. Science and Public Policy (Science and Public Policy), 44(4), 486–496, from https://doi.org/10.1093/scipol/scw076.
Search in Google Scholar Back to article
Marina T., & Sterligov I. (2021). Prevalence of potentially predatory publishing in Scopus on the country level. Scientometrics, 126(6), 5019-5077, from https://doi.org/10.1007/s11192-021-03899-x
Search in Google Scholar Back to article
Moed H. F., Markusova V., & Akoev M. (2018). Trends in Russian research output indexed in Scopus and Web of Science. Scientometrics, 116(2), 1153–1180, from https://doi.org/10.1007/s11192-018-2769-8.
Search in Google Scholar Back to article
Mongeon P., Brodeur C., Beaudry C., & Larivière V. (2016). Concentration of research funding leads to decreasing marginal returns. Research Evaluation, 25(4), 396–404, from https://doi.org/10.1093/reseval/rvw007.
Search in Google Scholar Back to article
Morillo F. (2019). Collaboration and impact of research in different disciplines with international funding (from the EU and other foreign sources). Scientometrics, 120(2), 807–823, from https://doi.org/10.1007/s11192-019-03150-8.
Search in Google Scholar Back to article
Mosbah-Natanson S., & Gingras Y. (2014). The globalization of social sciences? Evidence from a quantitative analysis of 30 years of production, collaboration and citations in the social sciences (1980–2009). Current Sociology, 62(5), 626–646, from https://doi.org/10.1177/0011392113498866.
Search in Google Scholar Back to article
Najman J. M., & Hewitt B. (2003). The validity of publication and citation counts for Sociology and other selected disciplines. Journal of Sociology, 39(1), 62–80, from https://doi.org/10.1177/144078330303900106.
Search in Google Scholar Back to article
Nederhof A. J. (2006). Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review. Scientometrics, 66(1), 81–100, from https://doi.org/10.1007/s11192-006-0007-2.
Search in Google Scholar Back to article
Park M., Leahey E., & Funk R. J. (2023). Papers and patents are becoming less disruptive over time. Nature, 613(7942), 138-144, from https://doi.org/10.1038/s41586-022-05543-x
Search in Google Scholar Back to article
Paudel P. K., Giri B., & Dhakal S. (2020). Is research in peril in Nepal? Publication trend and research quality from projects funded by the University Grants Commission-Nepal. Accountability in Research, 27(7), 444–456, from https://doi.org/10.1080/08989621.2020.1768374.
Search in Google Scholar Back to article
Sandström U., & Hällsten M. (2008). Persistent nepotism in peer-review. Scientometrics, 74(2), 175-189, from https://doi.org/10.1007/s11192-008-0211-3
Search in Google Scholar Back to article
Saygitov R. T. (2014). The impact of funding through the RF President’s Grants for young scientists (the field – medicine) on research productivity: A quasi-experimental study and a brief systematic review. PLOS ONE, 9(1), e86969, from https://doi.org/10.1371/journal.pone.0086969
Search in Google Scholar Back to article
Sokolov M. (2019). The sources of academic localism and globalism in Russian sociology: The choice of professional ideologies and occupational niches among social scientists. Current Sociology, 67(6), 818–837, from https://doi.org/10.1177/0011392118811392
Search in Google Scholar Back to article
Sokolov M. (2021). Can Russian Research Policy be Called Neoliberal? A Study in the Comparative Sociology of Quantification. Europe-Asia Studies, 73(6), 989–1009, from https://doi.org/10.1080/09668136.2021.1902945
Search in Google Scholar Back to article
Sterligov I. (2017). The monster ten you have never heard of: Top Russian scholarly megajournals. Higher Education in Russia and Beyond, 11, 11–13.
Search in Google Scholar Back to article
Tonta Y. (2018). Does monetary support increase the number of scientific papers? An interrupted time series analysis. Journal of Data and Information Science, 3(1), 19–39, from https://content.sciendo.com/view/journals/jdis/3/1/article-p19.xml.
Search in Google Scholar Back to article
van den Besselaar P. (2012). Selection committee membership: Service or self-service. Journal of Informetrics, 6(4), 580-585, from https://doi.org/10.1016/j.joi.2012.05.003
Search in Google Scholar Back to article
van Raan A. F. J. (1998). In matters of quantitative studies of science the fault of theorists is offering too little and asking too much. Scientometrics, 43(1), 129–139, from https://doi.org/10.1007/BF02458401.
Search in Google Scholar Back to article
Wang J., & Shapira P. (2015). Is there a relationship between research sponsorship and publication impact? An analysis of funding acknowledgments in nanotechnology papers. PLOS ONE, 10(2), e0117727, from https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0117727.
Search in Google Scholar Back to article
Wang L. L., Wang X. W., Piro F. N., & Philipsen N. (2020). The effect of competitive public funding on scientific output. Research Evaluation, 2020(September), 1–13, from https://doi.org/10.1093/reseval/rvaa023
Search in Google Scholar Back to article
Wuchty S., Jones B. F., & Uzzi B. (2007). The Increasing Dominance of Teams in Production of Knowledge. Science, 316(5827), 1036–1039, from https://doi.org/10.1126/science.1136099
Search in Google Scholar Back to article
Yan E., Wu C. J., & Song M. (2018). The funding factor: A cross-disciplinary examination of the association between research funding and citation impact. Scientometrics, 115(1), 369–384, from https://doi.org/10.1007/s11192-017-2583-8.
Search in Google Scholar Back to article
Zhao S. X., Lou W., Tan A. M., & Yu S. (2018). Do funded papers attract more usage? Scientometrics, 115(1), 153–168, from https://doi.org/10.1007/s11192-018-2662-5.
Search in Google Scholar Back to article