Have a personal or library account? Click to login
Evidence for studying interactions between science and policy: An exploration of scholarly and policy references in Overton-indexed policy documents Cover

Evidence for studying interactions between science and policy: An exploration of scholarly and policy references in Overton-indexed policy documents

Open Access
|Nov 2025

References

  1. Adie, E. (2020, April 23). Who are COVID-19 guidelines & policy documents citing? Overton Blog. https://blog.overton.io/blog/2020/04/23/who-are-the-covid-19-guidelines-policy-citing
  2. Alla, K., Hall, W. D., Whiteford, H. A., Head, B. W., & Meurk, C. S. (2017). How do we define the policy impact of public health research? A systematic review. Health Research Policy and Systems, 15(1), 84. https://doi. org/10.1186/s12961-017-0247-z
  3. Alonso-Alvarez, P., & Van Eck, N. J. (2024). Coverage and metadata availability of african publications in OpenAlex: A comparative analysis (No. arXiv:2409.01120). arXiv. http://arxiv.org/abs/2409.01120
  4. Atapour, H., Maddahi, R., & Zavaraqi, R. (2024). Policy citations of scientometric articles: An altmetric study. Scientometrics, 129(7), 4423–4436. https://doi.org/10.1007/s11192-024-05091-3
  5. Black, N. (2001). Evidence based policy: Proceed with care. BMJ, 323(7307), 275–279. https://doi.org/10.1136/bmj.323.7307.275
  6. Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for Information Science and Technology, 64(2), 217–233. https://doi.org/10.1002/asi.22803
  7. Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8(4), 895–903. https://doi.org/10.1016/j.joi.2014.09.005
  8. Bornmann, L., Haunschild, R., & Adams, J. (2019). Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF). Journal of Informetrics, 13(1), 325–340. https://doi.org/10.1016/j.joi.2019.01.008
  9. Bornmann, L., Haunschild, R., & Marx, W. (2016). Policy documents as sources for measuring societal impact: How often is climate change research mentioned in policy-related documents? Scientometrics, 109(3), 1477–1495. https://doi.org/10.1007/s11192-016-2115-y
  10. Costas, R., de Rijcke, S., & Marres, N. (2021). “Heterogeneous couplings”: Operationalizing network perspectives to study science-society interactions through social media metrics. Journal of the Association for Information Science and Technology, 72(5), 595–610. https://doi.org/10.1002/asi.24427
  11. Cristofoletti, E. C., Salles-Filho, Es., Hollanda, S., Juk, Y., Pinto, K. E. F., Toledo, C. G., Cabral, B. P., Tetzner, G. A., & Campgnolli, E. (2023, April 21). The use of research in policy documents: Exploring methodological potentialities. 27th International Conference on Science, Technology and Innovation Indicators (STI 2023). https://dapp.orvium.io/deposits/6443055271296903d638d0d4/view
  12. Dorta-González, P., Rodríguez-Caro, A., & Dorta-González, M. I. (2024). Societal and scientific impact of policy research: A large-scale empirical study of some explanatory factors using Altmetric and Overton. Journal of Informetrics, 18(3), 101530. https://doi.org/10.1016/j.joi.2024.101530
  13. Drongstrup, D., Malik, S., Aljohani, N. R., Alelyani, S., Safder, I., & Hassan, S.-U. (2020). Can social media usage of scientific literature predict journal indices of AJG, SNIP and JCR? An altmetric study of economics. Scientometrics, 125(2), 1541–1558. https://doi.org/10.1007/s11192-020-03613-3
  14. Fairclough, R., & Thelwall, M. (2015). More precise methods for national research citation impact comparisons. Journal of Informetrics, 9(4), 895–906. https://doi.org/10.1016/j.joi.2015.09.005
  15. Fang, Z., & Costas, R. (2020). Studying the accumulation velocity of altmetric data tracked by Altmetric.com. Scientometrics, 123(2), 1077–1101. https://doi.org/10.1007/s11192-020-03405-9
  16. Fang, Z., Costas, R., Tian, W., Wang, X., & Wouters, P. (2020). An extensive analysis of the presence of altmetric data for Web of Science publications across subject fields and research topics. Scientometrics, 124(3), 2519–2549. https://doi.org/10.1007/s11192-020-03564-9
  17. Gauch, S., & Blümel, C. (2016). Deliverable D5.1 – Altmetrics Status Quo OPENing UP new methods, indicators and tools for peer review, impact measurement and dissemination of research results (https://ec.europa.eu/research/participants/documents/downloadPublic?documentIds=080166e5ae8a151c&appId=P PGMS). https://ec.europa.eu/research/participants/documents/downloadPublic?documentIds=080166e5ae8a 151c&appId=PPGMS. https://ec.europa.eu/research/participants/documents/downloadPublic?documentIds= 080166e5ae8a151c&appId=PPGMS
  18. Haunschild, R., & Bornmann, L. (2017). How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data. Scientometrics, 110(3), 1209–1216. https://doi.org/10.1007/s11192-016-2237-2
  19. Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The leiden manifesto for research metrics. Nature, 520(7548), 429–431. https://doi.org/10.1038/520429a
  20. Hu, L., Huang, W., & Bu, Y. (2024). Interdisciplinary research attracts greater attention from policy documents: Evidence from COVID-19. Humanities and Social Sciences Communications, 11(1), 1–10. https://doi. org/10.1057/s41599-024-02915-8
  21. Jonker, H., & Vanlee, F. (2024). Linking science with media and policy: The case of academics in Flanders, Belgium. Quantitative Science Studies, 5(3), 556–572. https://doi.org/10.1162/qss_a_00311
  22. Kassab, O., Bornmann, L., & Haunschild, R. (2020). Can altmetrics reflect societal impact considerations?: Exploring the potential of altmetrics in the context of a sustainability science research center. Quantitative Science Studies, 1(2), 792–809. https://doi.org/10.1162/qss_a_00032
  23. Kolahi, J., & Khazaei, S. (2018). Altmetric analysis of contemporary dental literature. British Dental Journal, 225(1), 68–72. https://doi.org/10.1038/sj.bdj.2018.521
  24. Maricato, J. de M., Mazoni, A., Mugnaini, R., Packer, A. L., & Costas, R. (2023, April 21). SciELO as an open scientometric research infrastructure: General discussion of coverage in OpenAlex, WoS, scopus and dimensions. 27th International Conference on Science, Technology and Innovation Indicators (STI 2023). https://dapp.orvium.io/deposits/6442c231903ef57acd6dc640/view
  25. Noyons, E. (2019). Measuring societal impact is as complex as ABC. Journal of Data and Information Science, 4(3), 6–21. https://doi.org/10.2478/jdis-2019-0012
  26. Overton. (2021, November 23). How international are your sources? https://help.overton.io/article/how-international-are-your-sources/
  27. Overton. (2024a, February 7). About topics, entities and subject areas. https://help.overton.io/article/about-topics-entities-and-subject-areas/
  28. Overton. (2024b, February 7). Documents in different languages. https://help.overton.io/article/documents-in-different-languages/
  29. Overton. (2024c, February 7). How are scholarly references matched in policy documents? https://help.overton.io/article/how-are-scholarly-references-matched-in-policy-documents/
  30. Overton. (2024d, February 7). How does Overton find citation contexts? https://help.overton.io/article/how-does-overton-find-citation-contexts/
  31. Overton. (2024e, February 7). How far back does the database go? https://help.overton.io/article/how-far-back-does-the-database-go/
  32. Overton. (2024f, February 7). What are your criteria for adding new sources? Overton Knowledge Base. https://help.overton.io/article/what-are-your-criteria-for-adding-new-sources/
  33. Overton. (2024g, February 7). What is Overton? https://help.overton.io/article/what-is-overton/
  34. Overton. (2024h, February 7). What sources does Overton track? https://help.overton.io/article/what-sources-does-overton-track/
  35. Overton. (2024i, February 7). What’s your definition of a policy document? https://help.overton.io/article/whats-your-definition-of-a-policy-document/
  36. Overton. (2024j, February 8). Using policy related metrics responsibly in research assessment. https://help.overton.io/article/using-policy-related-metrics-responsibly-in-research-assessment/
  37. Pedersen, D. B. (2014). The political epistemology of science-based policy-making. Society, 51(5), 547–551. https://doi.org/10.1007/s12115-014-9820-z
  38. Pinheiro, H., Vignola-Gagné, E., & Campbell, D. (2021). A large-scale validation of the relationship between crossdisciplinary research and its uptake in policy-related documents, using the novel Overton altmetrics database. Quantitative Science Studies, 2(2), 616–642. https://doi.org/10.1162/qss_a_00137
  39. Robinson-García, N., Torres-Salinas, D., Zahedi, Z., & Costas, R. (2014). New data, new possibilities: Exploring the insides of Altmetric.com. Profesional de la información, 23(4), Article 4.
  40. Sanderson, I. (2002). Evaluation, policy learning and evidence-based policy making. Public Administration, 80(1), 1–22. https://doi.org/10.1111/1467-9299.00292
  41. Szomszor, M., & Adie, E. (2022). Overton: A bibliometric database of policy document citations. Quantitative Science Studies, 3(3), 624–650. https://doi.org/10.1162/qss_a_00204
  42. Tattersall, A., & Carroll, C. (2018). What can Altmetric.com tell us about policy citations of research? An analysis of Altmetric.com data for research articles from the University of Sheffield. Frontiers in Research Metrics and Analytics, 2, 9. https://doi.org/10.3389/frma.2017.00009
  43. Taylor, M. (2020). An altmetric attention advantage for open access books in the humanities and social sciences. Scientometrics, 125(3), 2523–2543. https://doi.org/10.1007/s11192-020-03735-8
  44. Thelwall, M. (2015, October 7). Geometric means for citation counts and altmetrics. Altmetrics for Evaluations. https://altmetrics.blogspot.com/2015/10/geometric-means-for-citation-counts-and.html
  45. Thelwall, M. (2016). The precision of the arithmetic mean, geometric mean and percentiles for citation data: An experimental simulation modelling approach. Journal of Informetrics, 10(1), 110–123. https://doi. org/10.1016/j.joi.2015.12.001
  46. Thelwall, M., & Fairclough, R. (2015). Geometric journal impact factors correcting for individual highly cited articles. Journal of Informetrics, 9(2), 263–272. https://doi.org/10.1016/j.joi.2015.02.004
  47. Thelwall, M., Haustein, S., Larivière, V., & Sugimoto, C. R. (2013). Do altmetrics work? Twitter and ten other social web services. PLOS ONE, 8(5), e64841. https://doi.org/10.1371/journal.pone.0064841
  48. Waltman, L., & Costas, R. (2014). F1000 recommendations as a potential new data source for research evaluation: A comparison with citations. Journal of the Association for Information Science and Technology, 65(3), 433–445. https://doi.org/10.1002/asi.23040
  49. Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., Jones, R., Kain, R., Kerridge, S., & Thelwell, M. (2015). The metric tide: Report of the independent review of the role of metrics in research assessment and management (No. 10.13140/RG.2.1.4929.1363). http://eprints.whiterose.ac.uk/117033/1/2015_metric_tide. pdf
  50. Yin, Y., Gao, J., Jones, B. F., & Wang, D. (2021). Coevolution of policy and science during the pandemic. Science, 371(6525), 128–130. https://doi.org/10.1126/science.abe3084
  51. Yu, H., Cao, X., Xiao, T., & Yang, Z. (2020). How accurate are policy document mentions? A first look at the role of altmetrics database. Scientometrics, 125(2), 1517–1540. https://doi.org/10.1007/s11192-020-03558-7
  52. Yu, H., Murat, B., Li, J., & Li, L. (2023). How can policy document mentions to scholarly papers be interpreted? An analysis of the underlying mentioning process. Scientometrics, 128(11), 6247–6266. https://doi. org/10.1007/s11192-023-04826-y
DOI: https://doi.org/10.2478/jdis-2025-0054 | Journal eISSN: 2543-683X | Journal ISSN: 2096-157X
Language: English
Page range: 63 - 87
Submitted on: Jul 24, 2025
|
Accepted on: Nov 4, 2025
|
Published on: Nov 18, 2025
In partnership with: Paradigm Publishing Services

© 2025 Biegzat Murat, Zhichao Fang, Ed Noyons, Rodrigo Costas, published by Chinese Academy of Sciences, National Science Library
This work is licensed under the Creative Commons Attribution 4.0 License.