Figure 1.
Familiarity with calls to reform existing research evaluation practices_
| Do you know…? (N = 263) | Yes, I know this | Yes, I have heard about this but do not know its content | No, I don’t know this | |||
|---|---|---|---|---|---|---|
| The DORA declaration | 27 | (10.3%) | 29 | (11.0%) | 207 | (78.7%) |
| The Leiden Manifesto | 25 | (9.5%) | 41 | (15.6%) | 197 | (74.9%) |
| Responsible metrics | 14 | (5.3%) | 42 | (16.0%) | 207 | (78.7%) |
| Metric Tide report | 7 | (2.7%) | 22 | (8.4%) | 234 | (89.0%) |
Distribution of participants by function and gender compared with the distribution of the population of Sapienza University on December 31, 2018 (“Total N” and “%”)_ (Adapted from Rousseau et al_, 2021)_
| Function | N. of resp. | % in sample | % sample compared to total N in category | Total N | % |
|---|---|---|---|---|---|
| Full professor | 60 | 19.17 | 8.90 | 674 | 20.39 |
| Associate professor | 121 | 38.66 | 10.42 | 1,161 | 35.12 |
| Full-time assistant professor | 69 | 22.04 | 6.26 | 1,102 | 33.33 |
| Contract professor (professori incaricati) | 8 | 2.56 | - | - | - |
| Temporary assistant professor L. 230/05 | 1 | 0.32 | - | - | - |
| Temporary assistant professor L 240/10 Tipo A | 23 | 7.35 | 10.85 | 212 | 6.41 |
| Temporary assistant professor 240/10 Tipo B | 16 | 5.11 | 10.19 | 157 | 4.75 |
| Other (retired) | 15 | 4.79 | - | - | - |
| Gender | |||||
| Male | 199 | 63.58 | 10.04 | 1,982 | 59.95 |
| Female | 113 | 36.10 | 8.53 | 1,324 | 40.05 |
| X | 1 | 0.32 | - | - |
Cronbach’s alpha for the components capturing metric-wiseness_
| Cronbach’s alpha | ||
|---|---|---|
| Component 1 | Technical knowledge of indicators | Not applicable (true - false statements) |
| Component 2 | Use of indicators | 0.3278 |
| Component 3 | Researchers’ intrinsic motivation | 0.3735 |
| Component 4 | External pressure | 0.6135 |
Component 3 statements were ordered in increasing levels of absolutely agreeing participants_
| N = 263 | Absolutely agree | Agree | Neutral | Disagree | Absolutely disagree |
|---|---|---|---|---|---|
| C3.S1 - If I do not have the expertise to solve a particular problem, I do not hesitate to ask a colleague to collaborate with me | 14 (5.3%) | 25 (9.5%) | 68 (25.9%) | 80 (30.4%) | 76 (28.9%) |
| C3.S2 - I select research problems inspired by my own curiosity | 29 (11.0%) | 49 (18.6%) | 48 (18.3%) | 51 (19.4%) | 86 (32.7%) |
| C3.S3 - I select topics for research based on their potential to advance science | 82 (31.2%) | 120 (45.6%) | 49 (18.6%) | 7 (2.7%) | 5 (1.9%) |
Component 2 was ordered by the increasing number of participants who agreed_
| N = 263 | Absolutely agree | Agree | Neutral | Disagree | Absolutely disagree |
|---|---|---|---|---|---|
| C2.S1 - Bibliometric indicators are equally useful in evaluating disciplinary and interdisciplinary research | 7 (2.7%) | 46 (17.5% | 79 (30.0%) | 87 (33.1%) | 44 (16.7%) |
| C2.S2 - Besides citation-based indicators, one must, in applied fields, also take patent-based and similar indicators into account when evaluating researchers | 22 (8.4%) | 94 (35.7%) | 108 (41.1%) | 22 (8.4%) | 17 (6.5%) |
| C2.S3 - The social influence of research must be taken into account in evaluating researchers | 34 (12.9%) | 81 (30.8%) | 83 (31.6%) | 45 (17.1%) | 20 (7.6%) |
| C2.S4 - Besides citation-based indicators, one must also take journal standing within a field into account | 41 (15.6%) | 130 (49.4%) | 64 (24.3%) | 18 (6.8%) | 10 (3.8%) |
| C2.S5 - A purely bureaucratic, automatic and quantitative approach to research evaluation is unbiased for an individual researcher | 62 (23.6%) | 39 (14.8%) | 42 (16.0%) | 47 (17.9%) | 73 (27.8%) |
| C2.S6 - The quality of a researcher should be measured in relative terms within a field rather than in absolute terms | 126 (47.9%) | 108 (41.1%) | 22 (8.4%) | 4 (1.5%) | 3 (1.1%) |
Component 4 statements were ordered in increasing levels of absolutely agreeing participants_
| N = 263 | Absolutely agree | Agree | Neutral | Disagree | Absolutely disagree |
|---|---|---|---|---|---|
| C4.S1 - My institution influences how I communicate the results | 10 | 52 | 87 | 70 | 44 |
| of my research | (3.8%) | (19.8%) | (33.1%) | (26.6%) | (16.7%) |
| C4.S2 - I feel ‘publish or perish’ pressure in carrying out my | 12 | 48 | 96 | 70 | 37 |
| research | (4.6%) | (18.3%) | (36.5%) | (26.6%) | (14.1%) |
| C4.S3 - I select topics for research based on their potential to get | 15 | 35 | 84 | 88 | 41 |
| published quickly | (5.7%) | (13.3%) | (31.9%) | (33.5%) | (15.6%) |
| C4.S4 - It is important to use social media (Twitter, blogs…) to | 27 | 104 | 88 | 36 | 8 |
| share the results of my research | (10.3%) | (39.5%) | (33.5%) | (13.7%) | (3.0%) |
| C4.S5 - It is important to use academic research networks | 30 | 90 | 65 | 59 | 19 |
| (Mendeley, ResearchGate.) to share the results of my research | (11.4%) | (34.2%) | (24.7%) | (22.4%) | (7.2%) |
| C4.S6 - My likelihood of being promoted depends only on the | 38 | 96 | 55 | 55 | 19 |
| number of articles published in journals indexed in WoS or Scopus | (14.4%) | (36.5%) | (20.9%) | (20.9%) | (7.2%) |
| C4.S7 - ANVUR influences my publication strategies | 41 (15.6%) | 58 (22.1%) | 51 (19.4%) | 49 (18.6%) | 64 (24.3%) |
| C4.S8 - Open Science (including publication, conservation and reuse of research data) is relevant for my research | 42 (16.0%) | 95 (36.1%) | 101 (38.4%) | 21 (8.0%) | 4 (1.5%) |
| C4.S9 - My likelihood of being promoted depends mainly on the number of articles of which I am first or corresponding author | 53 (20.2%) | 104 (39.5%) | 68 (25.9%) | 32 (12.2%) | 6 (2.3%) |
| C4.S10 - I feel completely free to publish my research in any way I want | 99 (37.6%) | 128 (48.7%) | 27 (10.3%) | 7 (2.7%) | 2 (0.8%) |
| C4.S11 - The Ministry of Education and Research (MIUR) influences my publication strategies | 107 (40.7%) | 86 (32.7%) | 32 (12.2%) | 29 (11.0%) | 9 (3.4%) |
| C4.S12 - My institution (Sapienza) influences my publication strategies | 144 (54.8%) | 92 (35.0%) | 19 (7.2%) | 6 (2.3%) | 2 (0.8%) |
The most important motivation for publishing (participants could select 3 out of 16 listed items)_
| N = 313 | N | % |
|---|---|---|
| To contribute to the scientific progress in your discipline | 177 | 57% |
| To share your research findings with the academic community | 146 | 47% |
| To improve your chances of receiving research funding | 95 | 30% |
| Your personal intrinsic motivation | 84 | 27% |
| To increase your chances to be promoted | 69 | 22% |
| To improve your standing among your peers | 55 | 18% |
| To help others (e.g. doctoral students, project collaborators…) | 54 | 17% |
| To increase your probability of finding a new position | 32 | 10% |
| To improve your standing in your current institution | 30 | 10% |
| To make your current position permanent | 26 | 8% |
| To increase the prestige and the resources allocated to your department | 23 | 7% |
| To improve the standing of your institution | 19 | 6% |
| To share your research findings with policymakers and practitioners | 18 | 6% |
| To fulfill project requirements | 16 | 5% |
| To fulfill administrative requirements | 16 | 5% |
| To get a monetary reward | 2 | 1% |
Keyword search analysis of four documents of recent research evaluation reform initiatives_
| Initiative document | Number of times the keyword “heterogeneity” or “diversity” or “variety” or “multiplicity” or “dissimilarity” is present in the document |
| DORA declaration (DORA, 2015) | Variety 1 time cited (“6. Greatly reduce emphasis on the journal impact factor as a promotional tool, ideally by ceasing to promote the impact factor or by presenting the metric in the context of a variety of journal-based metrics”.) |
| Leiden Manifesto (Hicks et al., 2015) | No keywords cited |
| Hong Kong Declaration (Moher et al., 2020) | Variety 2 times cited but with a generic meaning of many not related to evaluation principles (“Selective publishing of research with positive results (i.e. publication bias) distorts science’s evidence base and has been demonstrated in a variety of disciplines including economics, psychology, and clinical and preclinical health research”; “The Center for Open Science’s Transparency and Openness Promotion initiative provides information on data transparency standards for a wide variety of discipline journals”.) |
| European Commission (2021) Scoping document | Variety 2 times cited (“Career assessment should take into account the variety of activities of academics such as teaching, research, entrepreneurship, management or leadership”; “To achieve excellent and relevant higher education, support is also needed to stimulate pedagogical innovation, focused on the learners, with a variety of learning spaces and flexible, interdisciplinary paths”.) |
Technical knowledge of indicators (ranked according to the number of correct answers)_
| N = 263 | True, I am sure | True, I think | I do not know | False, I think | False, I am sure | Correct answers |
|---|---|---|---|---|---|---|
| C1.S1 - Bibliometric indicators can easily be compared across disciplines [FALSE] | 8 (3.0%) | 21 (8.0%) | 34 (12.9%) | 95 (36.1%) | 105 (39.9%) | 200 (76.0%) |
| C1.S2 - Open Access journals never have a Web of Science impact factor [FALSE] | 7 (2.7%) | 16 (6.1%) | 78 (29.7%) | 82 (31.2%) | 80 (30.4%) | 162 (61.6%) |
| C1.S3 - On average older researchers have higher h-indices [TRUE] | 54 (20.5%) | 96 (36.5%) | 52 (19.8%) | 49 (18.6%) | 12 (4.6%) | 150 (57.0%) |
| C1.S4 - Citations received in conference proceedings are always included in an article’s total number of received citations in the Web of Science [FALSE] | 12 (4.6%) | 35 (13.3%) | 117 (44.5%) | 68 (25.9%) | 31 (11.8%) | 99 (37.6%) |
Description of the dataset (Adapted from Rousseau et al_, 2021)_
| Faculty | Group | Sample | Total Sapienza | |||
|---|---|---|---|---|---|---|
| N | % in sample | % in category | N | % | ||
| Mathematics, Physics and Natural Sciences | Exact sciences | 59 | 18.85 | 14.15 | 417 | 12.61 |
| Architecture | 9 | 2.88 | 5.36 | 168 | 5.08 | |
| Civil and Industrial Engineering | 37 | 11.82 | 12.80 | 289 | 8.74 | |
| Information Engineering, Informatics and Statistics | Engineering & Technology | 38 | 12.14 | 17.12 | 222 | 6.72 |
| School of Aerospace Engineering | 2 | 0.64 | 20.00 | 10 | 0.30 | |
| Pharmacy and Medicine | 36 | 11.50 | 7.81 | 461 | 13.94 | |
| Medicine and Dentistry | Medical sciences | 32 | 10.22 | 5.50 | 584 | 17.66 |
| Medicine and Psychology | 29 | 9.27 | 8.76 | 331 | 10.01 | |
| Arts and Humanities | Humanities and Social sciences | 38 | 12.14 | 10.11 | 376 | 11.37 |
| Economics | 11 | 3.51 | 6.15 | 179 | 5.41 | |
| Law | 4 | 1.28 | 4.65 | 86 | 2.60 | |
| Political Science, Sociology and Communication Science | 18 | 5.75 | 9.84 | 183 | 5.54 | |
| TOTAL | 313 | 9.47 | 3,306 | |||