Have a personal or library account? Click to login
Evaluating grant proposals: lessons from using metrics as screening device Cover

Evaluating grant proposals: lessons from using metrics as screening device

Open Access
|Jun 2023

Figures & Tables

Figure 1.

Flow-chart of the data processing
Flow-chart of the data processing

Figure 2.

Comparison of principal investigators’ publications between two waves of grant competitions.
Comparison of principal investigators’ publications between two waves of grant competitions.

Figure 3.

Comparison of projects’ publication output between the two waves of grant competitions.
Comparison of projects’ publication output between the two waves of grant competitions.

Figure 4.

Correlation between scientometric parameters of the principal investigators at proposal submission and project publications.
Correlation between scientometric parameters of the principal investigators at proposal submission and project publications.

Project publication output by quartiles (only journal articles included)

PhysicsSSH
Journal quartilesFirst wave (%)Second wave (%)First wave (%)Second wave (%)
Q1630 (49.8)579 (59)42 (8.4)48 (13)
Q2298 (23.6)166 (16.9)93 (18.7)72 (19.5)
Q3225 (17.8)137 (14)110 (22.1)141 (38.2)
Q4112 (8.9)100 (10.2)253 (50.8)108 (29.3)
Total1265 (100)982 (100)498 (100)369 (100)

Descriptive statistics of principal investigators’ research performance and project publication output

PhysicsSSH
2014(N projects=115)2017(N projects=82)2014(N projects=94)2017(N projects=41)
Q1 papers (N) before
Mean (SD)22.3 (58.4)17.5 (22.5)0.319 (0.845)1.22 (2.72)
Median [Min, Max]11 [0, 422]10.5 [0, 114]0 [0, 5.00]0 [0, 15]
Q1 papers (%) before
Mean (SD)40.7 (24.1)48.4 (27.3)6.82 (18.4)16.2 (24.5)
Median [Min, Max]34 [0, 92.5]50.9 [0, 100]0 [0, 100]0 [0, 87.5]
Q1 papers (N) after
Mean (SD)5.48 (5.77)7.06 (5.44)0.447 (0.875)1.17 (1.99)
Median [Min, Max]4 [0, 38]6 [0, 31]0 [0, 4]0 [0, 8]
Q1 papers (%) after*
Mean (SD)46.6 (29.9)55.9 (27.7)9.29 (21.0)11.4 (18.3)
Median [Min, Max]50 [0, 100]55.9 [0, 100]0 [0, 100]0 [0, 80]

Principal investigators’ publications by journal quartiles (only journal articles included)

Journal quartilesPhysicsSocial science and humanities
First wave (%)Second wave (%)First wave (%)Second wave (%)
Q12563 (53.7)1437 (52.4)30 (12.3)50 (24.3)
Q2710 (14.9)452 (16.5)13 (5.4)30 (14.6)
Q3800 (16.7)600 (21.9)57 (23.4)65 (31.5)
Q4703 (14.7)253 (9.2)144 (59)61 (29.6)
Total4776 (100)2742 (100)244 (100)206 (100)

Descriptive statistics of principal investigators’ research performance in the five years before they received funding

PhysicsSSH
2014(N projects=115)2017(N projects=82)2014(N projects=94)2017(N projects=41)
All Scopus publications in the 5 years before submitting (N)
Mean (SD)41.5 (64.5)33.4 (31.1)2.6 (4.39)4.95 (4.68)
Median [Min, Max]25 [2, 466]25 [0, 153]1 [0, 29]4 [0, 27]
Journal papers indexed by Scopus (N)
Mean (SD)35.3 (61.8)31.0 (29.2)2.12 (4.03)4.8 (4.7)
Median [Min, Max]21 [1, 450]22 [0, 150]0 [0, 28]4 [0, 27]
Journal papers indexed by Scopus (%)
Mean (SD)82.2 (14.2)91 (17.3)39.3 (44.1)89.4 (27.1)
Median [Min, Max]84.6 [41.7, 100]96.3 [0, 100]0 [0, 100]100 [0, 100]
Conference papers indexed by Scopus (%)
Mean (SD)14.6 (13.5)4.96 (9.16)4.02 (16.7)0.488 (3.12)
Median [Min, Max]12.5 [0, 52.9]0 [0, 56.3]0 [0, 100]0 [0, 20]
Other Scopus papers (%)
Mean (SD)3.29 (6.02)1.59 (2.89)9.84 (22.8)2.79 (8.97)
Median [Min, Max]1.16 [0, 50]0 [0, 11.1]0 [0, 100]0 [0, 33.3]

Descriptive statistics on projects’ publication output between the two waves of grant competitions

PhysicsSSH
2014(N projects=115)2017(N projects=82)2014(N projects=94)2017(N projects=41)
All publications (N)
Mean (SD)24.5 (17.4)21.9 (11.2)46.2 (27.1)56.2 (26.7)
Median [Min, Max]18 [3, 94]19 [5, 67]46 [5, 111]49 [14, 111]
Journal papers indexed by Scopus (N)
Mean (SD)11 (7.83)12 (5.61)5.3 (4.46)9 (6.12)
Median [Min, Max]9 [0, 43]11 [3, 37]4 [0, 18]9 [0, 40]
Journal papers indexed by Scopus (%)*
Mean (SD)51.1 (24.8)61.5 (23.3)16.0 (17.4)19.8 (14.9)
Median [Min, Max]50 [0, 100]62.8 [9.09, 100]10.2 [0, 80]16.2 [0, 81.3]
Non-Russian Scopus journal papers (N)
Mean (SD)9.24 (6.79)10.5 (5.66)2.22 (2.75)3.63 (3.09)
Median [Min, Max]8 [0, 41]10 [1, 37]1 [0, 14]3.00 [0, 11]
DOI: https://doi.org/10.2478/jdis-2023-0010 | Journal eISSN: 2543-683X | Journal ISSN: 2096-157X
Language: English
Page range: 66 - 92
Submitted on: Dec 19, 2022
Accepted on: Apr 13, 2023
Published on: Jun 7, 2023
Published by: Chinese Academy of Sciences, National Science Library
In partnership with: Paradigm Publishing Services
Publication frequency: 4 issues per year

© 2023 Katerina Guba, Alexey Zheleznov, Elena Chechik, published by Chinese Academy of Sciences, National Science Library
This work is licensed under the Creative Commons Attribution 4.0 License.