Have a personal or library account? Click to login
How to Run Linear Mixed Effects Analysis for Pairwise Comparisons? A Tutorial and a Proposal for the Calculation of Standardized Effect Sizes Cover

How to Run Linear Mixed Effects Analysis for Pairwise Comparisons? A Tutorial and a Proposal for the Calculation of Standardized Effect Sizes

By: Marc Brysbaert and  Dries Debeer  
Open Access
|Jan 2025

References

  1. 1Austin, P. C., White, I. R., Lee, D. S., & van Buuren, S. (2021). Missing data in clinical research: a tutorial on multiple imputation. Canadian Journal of Cardiology, 37(9), 13221331. 10.1016/j.cjca.2020.11.010
  2. 2Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390412. 10.1016/j.jml.2007.12.005
  3. 3Baguley, T. (2009). Standardized or simple effect size: What should be reported? British Journal of Psychology, 100(3), 603617. 10.1348/000712608X377117
  4. 4Bakeman, R. (2005). Recommended effect size statistics for repeated measures designs. Behavior Research Methods, 37, 379384. 10.3758/BF03192707
  5. 5Bartos, F., Maier, M., Wagenmakers, E. J., Nippold, F., Doucouliagos, H., Ioannidis, J. P. A., Otte, W. M., Sladekova, M., Deressa, T. K., Bruns, S. B., Fanelli, D., & Stanley, T. D. (2024). Footprint of publication selection bias on meta-analyses in medicine, environmental sciences, psychology, and economics. Research Synthesis Methods. 10.1002/jrsm.1703
  6. 6Bates, D., et al. (2022). Lme4: Linear mixed-effects models using ‘eigen’ and S4. Version 1.1-31. https://cran.r-project.org/web/packages/lme4/index.html
  7. 7Ben-Shachar, M. S., Makowski, D., Lüdecke, D., Patil, I., Wiernik, B. M., Thériault, R., Kelley, K., Stanley, D., Caldwell, A., Burnett, J., & Karreth, J. (2023). effectsize: Indices of Effect Size, Version 0.8.3. Available at https://cran.r-project.org/web/packages/effectsize/index.html
  8. 8Blake, K. R., & Gangestad, S. (2020). On attenuated interactions, measurement error, and statistical power: Guidelines for social and personality psychologists. Personality and Social Psychology Bulletin, 46(12), 17021711. 10.1177/0146167220913363
  9. 9Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. (2015). Correlational effect size benchmarks. Journal of Applied Psychology, 100(2), 431449. 10.1037/a0038047
  10. 10Brauer, M., & Curtin, J. J. (2018). Linear mixed-effects models and the analysis of nonindependent data: A unified framework to analyze categorical and continuous independent variables that vary within-subjects and/or within-items. Psychological Methods, 23(3), 389411. 10.1037/met0000159
  11. 11Brehm, L., & Alday, P. M. (2022). Contrast coding choices in a decade of mixed models. Journal of Memory and Language, 125, 104334. 10.1016/j.jml.2022.104334
  12. 12Brown, V. A. (2020, April 11). An introduction to linear mixed effects modeling in R. 10.31234/osf.io/9vghm
  13. 13Brysbaert, M. (2019). How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables. Journal of Cognition, 2(1), 16. 10.5334/joc.72
  14. 14Brysbaert, M. (2020). Basic statistics for psychologists (2nd edition). Macmilan International.
  15. 15Brysbaert, M. (2021). Power considerations in bilingualism research: Time to step up our game. Bilingualism: Language and Cognition, 24(5), 813818. 10.1017/S1366728920000437
  16. 16Brysbaert, M., & Stevens, M. (2018). Power Analysis and Effect Size in Mixed Effects Models: A Tutorial. Journal of Cognition, 1(9), 120. 10.5334/joc.10
  17. 17Clark, H. H. (1973). The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12(4), 335359. 10.1016/S0022-5371(73)80014-3
  18. 18Fernández-Castilla, B., Said-Metwaly, S., Kreitchmann, R. S., & Van Den Noortgate, W. (2024). What do meta-analysts need in primary studies? Guidelines and the SEMI checklist for facilitating cumulative knowledge. Behavior Research Methods, 56, 33153329. 10.3758/s13428-024-02373-9
  19. 19Gallucci, M. (2022). gamlj/gamlj: GAMLj Suite for linear models Version 2.6.6. https://rdrr.io/github/gamlj/gamlj/
  20. 20Garcia-Marques, L., Garcia-Marques, T., & Brauer, M. (2014). Buy three but get only two: The smallest effect in a 2 × 2 ANOVA is always uninterpretable. Psychonomic Bulletin & Review, 21(6), 14151430. 10.3758/s13423-014-0640-3
  21. 21Götz, F. M., Gosling, S. D., & Rentfrow, P. J. (2022). Small effects: The indispensable foundation for a cumulative psychological science. Perspectives on Psychological Science, 17(1), 205215. 10.1177/1745691620984483
  22. 22Hyatt, C. S., Crowe, M. L., West, S. J., Vize, C. E., Carter, N. T., Chester, D. S., & Miller, J. D. (2022). An empirically based power primer for laboratory aggression research. Aggressive behavior, 48(3), 279289. 10.1002/ab.21996
  23. 23Jaeger, B. (2022). Package ‘r2glmm’ Version 0.1.2. Available at https://cran.r-project.org/web/packages/r2glmm/r2glmm.pdf
  24. 24Johnson, P. C. (2014). Extension of Nakagawa & Schielzeth’s R2GLMM to random slopes models. Methods in Ecology and Evolution, 5(9), 944946. 10.1111/2041-210X.12225
  25. 25Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of Personality and Social Psychology, 103(1), 5469. 10.1037/a0028347
  26. 26Kuznetsova, A., Brockhoff, P. B., Christensen, R. H. B., & Jensen, S. P. (2022). Package ‘lmerTest’ Version 3.1–3. https://cran.r-project.org/web/packages/lmerTest/lmerTest.pdf
  27. 28Lewis, M. (2023). Fixing the stimulus-as-a-fixed-effect-fallacy in forensically valid face-composite research. Journal of Applied Research in Memory and Cognition. 10.1037/mac0000128
  28. 27Lovakov, A., & Agadullina, E. R. (2021). Empirically derived guidelines for effect size interpretation in social psychology. European Journal of Social Psychology, 51(3), 485504. 10.1002/ejsp.2752
  29. 29Matuschek, H., Kliegl, R., Vasishth, S., Baayen, H., & Bates, D. (2017). Balancing Type I error and power in linear mixed models. Journal of Memory and Language, 94, 305315. 10.1016/j.jml.2017.01.001
  30. 30Meteyard, L., & Davies, R. A. (2020). Best practice guidance for linear mixed-effects models in psychological science. Journal of Memory and Language, 112, 104092. 10.1016/j.jml.2020.104092
  31. 31Nakagawa, S., & Schielzeth, H. (2013). A general and simple method for obtaining R2 from generalized linear mixed-effects models. Methods in ecology and evolution, 4(2), 133142. 10.1111/j.2041-210x.2012.00261.x
  32. 32Norouzian, R., & Plonsky, L. (2018). Eta-and partial eta-squared in L2 research: A cautionary review and guide to more appropriate usage. Second Language Research, 34(2), 257271. 10.1177/0267658316684904
  33. 33Olejnik, S., & Algina, J. (2003). Generalized eta and omega squared statistics: Measures of effect size for some common research designs. Psychological Methods, 8, 434447. 10.1037/1082-989X.8.4.434
  34. 34Park, J., Cardwell, R., & Yu, H. T. (2020). Specifying the random effect structure in linear mixed effect models for analyzing psycholinguistic data. Methodology, 16(2), 92111. 10.5964/meth.2809
  35. 35Pek, J., & Flora, D. B. (2018). Reporting effect sizes in original psychological research: A discussion and tutorial. Psychological Methods, 23(2), 208225. 10.1037/met0000126
  36. 36Pollatsek, A., & Well, A. D. (1995). On the use of counterbalanced designs in cognitive research: A suggestion for a better and more powerful analysis. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21(3), 785794. 10.1037/0278-7393.21.3.785
  37. 37Richardson, J. T. (2011). Eta squared and partial eta squared as measures of effect size in educational research. Educational Research Review, 6(2), 135147. 10.1016/j.edurev.2010.12.001
  38. 38Rights, J. D., & Sterba, S. K. (2019). Quantifying explained variance in multilevel models: An integrative framework for defining R-squared measures. Psychological Methods, 24(3), 309338. 10.1037/met0000184
  39. 39Schäfer, T., & Schwarz, M. A. (2019). The meaningfulness of effect sizes in psychological research: Differences between sub-disciplines and the impact of potential biases. Frontiers in Psychology, 10, 813. 10.3389/fpsyg.2019.00813
  40. 40Schütze, A. (1964). Studies in Social Theory. Collected Papers, Volume II. Martinus Nijhoff. 10.1007/978-94-017-6854-2
  41. 41Sladekova, M., Webb, L. E. A., & Field, A. P. (2023). Estimating the change in meta-analytic effect size estimates after the application of publication bias adjustment methods. Psychological Methods, 28(3), 664686. 10.1037/met0000470
  42. 42Sommet, N., Weissman, D. L., Cheutin, N., & Elliot, A. (2023). How many participants do I need to test an interaction? Conducting an appropriate power analysis and achieving sufficient power to detect an interaction. Advances in Methods and Practices in Psychological Science. 10.1177/25152459231178728
  43. 43The jamovi project. (2023). jamovi. (Version 2.3) [Computer Software]. Retrieved from https://www.jamovi.org
  44. 44Unsworth, N., Heitz, R. P., Schrock, J. C., & Engle, R. W. (2005). An automated version of the operation span task. Behavior Research Methods, 37(3), 498505. 10.3758/BF03192720
  45. 45Vanhove, J. (2015, February 5). Why I don’t like standardised effect sizes. https://janhove.github.io/reporting/2015/02/05/standardised-vs-unstandardised-es
  46. 46Wagenmakers, E. J., Krypotos, A. M., Criss, A. H., & Iverson, G. (2012). On the interpretation of removable interactions: A survey of the field 33 years after Loftus. Memory & Cognition, 40, 145160. 10.3758/s13421-011-0158-0
  47. 47Westfall, J., Kenny, D. A., & Judd, C. M. (2014). Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Journal of Experimental Psychology: General, 143(5), 20202045. 10.1037/xge0000014
  48. 48Yarkoni, T. (2022). The generalizability crisis. Behavioral and Brain Sciences, 45, e1. 10.1017/S0140525X20001685
DOI: https://doi.org/10.5334/joc.409 | Journal eISSN: 2514-4820
Language: English
Submitted on: Sep 11, 2023
Accepted on: Oct 7, 2024
Published on: Jan 6, 2025
Published by: Ubiquity Press
In partnership with: Paradigm Publishing Services
Publication frequency: 1 issue per year

© 2025 Marc Brysbaert, Dries Debeer, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.