Practical operation and theoretical basis of difference-in-difference regression in science of science: The comparative trial on the scientific performance of Nobel laureates versus their coauthors

Huang, Yurui; Tian, Chaolin; Ma, Yifang

doi:10.2478/jdis-2023-0003

Abstract

Purpose

In recent decades, with the availability of large-scale scientific corpus datasets, difference-in-difference (DID) is increasingly used in the science of science and bibliometrics studies. DID method outputs the unbiased estimation on condition that several hypotheses hold, especially the common trend assumption. In this paper, we gave a systematic demonstration of DID in the science of science, and the potential ways to improve the accuracy of DID method.

Design/methodology/approach

At first, we reviewed the statistical assumptions, the model specification, and the application procedures of DID method. Second, to improve the necessary assumptions before conducting DID regression and the accuracy of estimation, we introduced some matching techniques serving as the pre-selecting step for DID design by matching control individuals who are equivalent to those treated ones on observational variables before the intervention. Lastly, we performed a case study to estimate the effects of prizewinning on the scientific performance of Nobel laureates, by comparing the yearly citation impact after the prizewinning year between Nobel laureates and their prizewinning-work coauthors.

Findings

We introduced the procedures to conduct a DID estimation and demonstrated the effectiveness to use matching method to improve the results. As a case study, we found that there are no significant increases in citations for Nobel laureates compared to their prizewinning coauthors.

Research limitations

This study ignored the rigorous mathematical deduction parts of DID, while focused on the practical parts.

Practical implications

This work gives experimental practice and potential guidelines to use DID method in science of science and bibliometrics studies.

Originality/value

This study gains insights into the usage of econometric tools in science of science.

References

Abadie, A., & Imbens, G. W. (2006). Large sample properties of matching estimators for average treatment effects. Econometrica, 74(1), 235–267.
Search in Google Scholar Back to article
Austin, P. C. (2009). Some methods of propensity-score matching had superior performance to others: results of an empirical investigation and Monte Carlo simulations. Biometrical Journal: Journal of Mathematical Methods in Biosciences, 51(1), 171–184.
Search in Google Scholar Back to article
Azoulay, P., Graff Zivin, J. S., & Wang, J. (2010). Superstar extinction. The Quarterly Journal of Economics, 125(2), 549–589.
Search in Google Scholar Back to article
Becker, S. O., & Ichino, A. (2002). Estimation of average treatment effects based on propensity scores. The Stata Journal, 2(4), 358–377.
Search in Google Scholar Back to article
Blackwell, M., Iacus, S., King, G., & Porro, G. (2009). cem: Coarsened exact matching in Stata. The Stata Journal, 9(4), 524–546.
Search in Google Scholar Back to article
Borjas, G. J., & Doran, K. B. (2015). Prizes and productivity how winning the fields medal affects scientific output. Journal of human resources, 50(3), 728–758.
Search in Google Scholar Back to article
Bourdieu, P. (2004). Science of science and reflexivity. Polity.
Search in Google Scholar Back to article
Broadus, R. N. (1987). Toward a definition of “bibliometrics”. Scientometrics, 12(5), 373–379.
Search in Google Scholar Back to article
Chan, H. F., Frey, B. S., Gallus, J., & Torgler, B. (2014). Academic honors and performance. Labour Economics, 31, 188–204.
Search in Google Scholar Back to article
Coca-Perraillon, M. (2007). Local and global optimal propensity score matching. SAS Global Forum, https://support.sas.com/resources/papers/proceedings/proceedings/forum2007/185-2007.pdf
Search in Google Scholar Back to article
Cover, T., & Hart, P. (1967). Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1), 21–27.
Search in Google Scholar Back to article
Donald, S. G., & Lang, K. (2007). Inference with difference-in-differences and other panel data. The review of Economics and Statistics, 89(2), 221–233.
Search in Google Scholar Back to article
Fortunato, S., Bergstrom, C. T., Börner, K., Evans, J. A., Helbing, D., Milojević, S., Petersen, A. M., Radicchi, F., Sinatra, R., & Uzzi, B. (2018). Science of science. Science, 359(6379), eaao0185.
Search in Google Scholar Back to article
Garrido, M. M., Kelley, A. S., Paris, J., Roza, K., Meier, D. E., Morrison, R. S., & Aldridge, M. D. (2014). Methods for constructing and assessing propensity scores. Health services research, 49(5), 1701–1720.
Search in Google Scholar Back to article
Heckman, J. J., LaLonde, R. J., & Smith, J. A. (1999). The economics and econometrics of active labor market programs. In Handbook of labor economics (Vol. 3, pp. 1865–2097). Elsevier.
Search in Google Scholar Back to article
Hill, J., & Reiter, J. P. (2006). Interval estimation for treatment effects using propensity score matching. Statistics in medicine, 25(13), 2230–2256.
Search in Google Scholar Back to article
Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proc Natl Acad Sci U S A, 102(46), 16569–16572. https://doi.org/10.1073/pnas.0507655102
Search in Google Scholar Back to article
Huang, J. M., Gates, A. J., Sinatra, R., & Barabási, A. L. (2020). Historical comparison of gender inequality in scientific careers across countries and disciplines. Proceedings of the National Academy of Sciences, 117(9), 4609–4616.
Search in Google Scholar Back to article
Huber, M., Lechner, M., & Steinmayr, A. (2015). Radius matching on the propensity score with bias adjustment: tuning parameters and finite sample behaviour. Empirical Economics, 49(1), 1–31.
Search in Google Scholar Back to article
Iacus, S. M., King, G., & Porro, G. (2012). Causal inference without balance checking: Coarsened exact matching. Political analysis, 20(1), 1–24.
Search in Google Scholar Back to article
Jin, C., Ma, Y. F., & Uzzi, B. (2021). Scientific prizes and the extraordinary growth of scientific topics. Nature Communications, 12(1), 1–11.
Search in Google Scholar Back to article
King, G., & Nielsen, R. (2019). Why propensity scores should not be used for matching. Political analysis, 27(4), 435–454.
Search in Google Scholar Back to article
King, G., Nielsen, R., Coberley, C., Pope, J. E., & Wells, A. (2011). Comparative effectiveness of matching methods for causal inference. Unpublished manuscript, Institute for Quantitative Social Science, Harvard University, Cambridge, MA.
Search in Google Scholar Back to article
Leuven, E., & Sianesi, B. (2003). PSMATCH2: Stata module to perform full Mahalanobis and propensity score matching, common support graphing, and covariate imbalance testing.
Search in Google Scholar Back to article
Li, J. C., Yin, Y. A., Fortunato, S., & Wang, D. S. (2019). A dataset of publication records for Nobel laureates. Scientific data, 6(1), 1–10.
Search in Google Scholar Back to article
Li, W. H., Aste, T., Caccioli, F., & Livan, G. (2019). Early coauthorship with top scientists predicts success in academic careers. Nat Commun, 10(1), 5170. https://doi.org/10.1038/s41467-019-13130-4
Search in Google Scholar Back to article
Liu, M. J., & Hu, X. (2022). Movers’ advantages: The effect of mobility on scientists’ productivity and collaboration. Journal of Informetrics, 16(3), 101311.
Search in Google Scholar Back to article
Ma, Y. F., Mukherjee, S., & Uzzi, B. (2020). Mentorship and protege success in STEM fields. Proc Natl Acad Sci U S A, 117(25), 14077–14083. https://doi.org/10.1073/pnas.1915516117
Search in Google Scholar Back to article
Ma, Y. F., & Uzzi, B. (2018). Scientific prize network predicts who pushes the boundaries of science. Proc Natl Acad Sci U S A, 115(50), 12608–12615. https://doi.org/10.1073/pnas.1800485115
Search in Google Scholar Back to article
McCaffrey, D. F., & Bell, R. M. (2003). Bias reduction in standard errors for linear regression with multi-stage samples. Quality control and applied statistics, 48(6), 677–682.
Search in Google Scholar Back to article
Merton, R. K. (1968). The Matthew effect in science: The reward and communication systems of science are considered. Science, 159(3810), 56–63.
Search in Google Scholar Back to article
Petersen, A. M. (2018). Multiscale impact of researcher mobility. Journal of The Royal Society Interface, 15(146), 20180580.
Search in Google Scholar Back to article
Petersen, A. M., Fortunato, S., Pan, R. K., Kaski, K., Penner, O., Rungi, A., Riccaboni, M., Stanley, H. E., & Pammolli, F. (2014). Reputation and impact in academic careers. Proc Natl Acad Sci U S A, 111(43), 15316–15321. https://doi.org/10.1073/pnas.1323111111
Search in Google Scholar Back to article
Price, D. J. (1986). Little science, big science... and beyond (Vol. 480). Columbia University Press New York.
Search in Google Scholar Back to article
Radicchi, F., Fortunato, S., & Castellano, C. (2008). Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences, 105(45), 17268–17272.
Search in Google Scholar Back to article
Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.
Search in Google Scholar Back to article
Roth, J., Sant’Anna, P. H., Bilinski, A., & Poe, J. (2022). What's Trending in Difference-in-Differences? A Synthesis of the Recent Econometrics Literature. arXiv preprint arXiv:2201.01194.
Search in Google Scholar Back to article
Rubin, D. B. (1977). Assignment to treatment group on the basis of a covariate. Journal of educational Statistics, 2(1), 1–26.
Search in Google Scholar Back to article
Rubin, D. B. (2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469), 322–331.
Search in Google Scholar Back to article
Sinatra, R., Wang, D. S., Deville, P., Song, C. M., & Barabasi, A. L. (2016). Quantifying the evolution of individual scientific impact. Science, 354(6312). https://doi.org/10.1126/science.aaf5239
Search in Google Scholar Back to article
Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical science: a review journal of the Institute of Mathematical Statistics, 25(1), 1.
Search in Google Scholar Back to article
Uzzi, B., Mukherjee, S., Stringer, M., & Jones, B. (2013). Atypical combinations and scientific impact. Science, 342(6157), 468–472.
Search in Google Scholar Back to article
Wang, D. S., Song, C. M., & Barabási, A. L. (2013). Quantifying long-term scientific impact. Science, 342(6154), 127–132.
Search in Google Scholar Back to article
Wang, J. X. (2021). To use or not to use propensity score matching? Pharmaceutical Statistics, 20(1), 15–24.
Search in Google Scholar Back to article
Yang, Y., Chawla, N. V., & Uzzi, B. (2019). A network's gender composition and communication pattern predict women's leadership success. Proceedings of the National Academy of Sciences, 116(6), 2033–2038.
Search in Google Scholar Back to article