Validity in the Era of Competency-Based Health Professions Education: Tensions and Potential Solutions
References
- Schuwirth LWT, van der Vleuten CPM. A history of assessment in medical education. Adv Health Sci Educ Theory Pract. 2020;25(5):1045–1056. DOI: 10.1007/s10459-020-10003-0
- Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007;29(9):855–871. DOI: 10.1080/01421590701775453
- Ten Cate O, Chen HC, Hoff RG, Peters H, Bok H, van der Schaaf M. Curriculum development for the workplace using Entrustable Professional Activities (EPAs): AMEE Guide No. 99. Med Teach. 2015;37(11):983–1002. DOI: 10.3109/0142159X.2015.1060308
- Hodges B. Assessment in the post-psychometric era: Learning to love the subjective and collective. Med Teach. 2013;35(7):564–568. DOI: 10.3109/0142159X.2013.789134
- Norcini J, Anderson MB, Bollela V, et al. Consensus framework for good assessment. Med Teach. 2018;40(11):1102–1109. DOI: 10.1080/0142159X.2018.1500016
- Bok HGJ, Teunissen PW, Favier RP, et al. Programmatic assessment of competency-based workplace learning: when theory meets practice. BMC Med Educ. 2013;13:
123 . DOI: 10.1186/1472-6920-13-123 - Schuwirth LWT, Van der Vleuten CPM. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach. 2011;33(6):478–485. DOI: 10.3109/0142159X.2011.565828
- Messick S.
Validity . In: Educational Measurement, 3rd Ed. The American Council on Education/Macmillan series on higher education. American Council on Education; 1989. pp. 13–103. - Kane MT. Validating the Interpretations and Uses of Test Scores. J Educ Meas. 2013;50(1):1–73. DOI: 10.1111/jedm.12000
- St-Onge C, Young M, Eva KW, Hodges B. Validity: one word with a plurality of meanings. Adv Health Sci Educ Theory Pract. 2017;22(4):853–867. DOI: 10.1007/s10459-016-9716-3
- Kinnear B, Schumacher DJ, Driessen EW, Varpio L. How argumentation theory can inform assessment validity: A critical review. Med Educ. 2022;56(11):1064–1075. DOI: 10.1111/medu.14882
- St-Onge C, Young M. Evolving conceptualisations of validity: impact on the process and outcome of assessment. Med Educ. 2015;49(6):548–550. DOI: 10.1111/medu.12734
- Morgan DL. Paradigms lost and pragmatism regained: methodological implications of combining qualitative and quantitative methods. J Mixed Methods Res. 2007;1:48–76. DOI: 10.1177/2345678906292462
- Young ME, Ryan A. Postpositivism in Health Professions Education Scholarship. Acad Med J Assoc Am Med Coll. 2020;95(5):695–699. DOI: 10.1097/ACM.0000000000003089
- Custers EJFM, Cate OT. The History of Medical Education in Europe and the United States, With Respect to Time and Proficiency. Acad Med. 2018;93(3S Competency-Based, Time-Variable Education in the Health Professions):S49–S54. DOI: 10.1097/ACM.0000000000002079
- Flexner A. Medical education in the United States and Canada: A report to the Carnegie Foundation for the Advancement of Teaching. Archives of the Carnegie Foundation. 1910. Accessed January 15, 2025.
https://archive.carnegiefoundation.org/publications/pdfs/elibrary/Carnegie_Flexner_Report.pdf - Downing SM, Yudkowsky R. Assessment in Health Professions Education. Taylor & Francis Group; 2009. Accessed October 11, 2024.
https://ebookcentral.proquest.com/lib/uva/detail.action?docID=425408 - ten Cate O, Regehr G. The Power of Subjectivity in the Assessment of Medical Trainees. Acad Med. 2019;94(3):
333 . DOI: 10.1097/ACM.0000000000002495 - Govaerts MJB, van der Vleuten CPM, Schuwirth LWT, Muijtjens AMM. Broadening Perspectives on Clinical Performance Assessment: Rethinking the Nature of In-training Assessment. Adv Health Sci Educ. 2007;12(2):239–260. DOI: 10.1007/s10459-006-9043-1
- Touchie C, Kinnear B, Schumacher D, et al. On the validity of summative entrustment decisions. Med Teach. 2021;43(7):780–787. DOI: 10.1080/0142159X.2021.1925642
- Englander R, de Graaf J, Hauer K, Jonker G, Schumacher D.
Clinical competency committees in an entrustable professional activity-based curriculum and assessment system . In: Entrustable Professional Activities and Entrustment Decision-Making in Health Professions Education. Ubiquity Press; 2024. pp. 249–258. DOI: 10.5334/bdc.u - Cook DA, Lineberry M. Consequences Validity Evidence: Evaluating the Impact of Educational Assessments. Acad Med J Assoc Am Med Coll. 2016;91(6):785–795. DOI: 10.1097/ACM.0000000000001114
- Hauer KE, Edgar L, Hogan SO, Kinnear B, Warm E. The Science of Effective Group Process: Lessons for Clinical Competency Committees. J Grad Med Educ. 2021;13(2 Suppl):59–64. DOI: 10.4300/JGME-D-20-00827.1
- Valentine N, Durning SJ, Shanahan EM, van der Vleuten C, Schuwirth L. The pursuit of fairness in assessment: Looking beyond the objective. Med Teach. 2022;44(4):353–359. DOI: 10.1080/0142159X.2022.2031943
- Hauer KE, Lucey CR. Core Clerkship Grading: The Illusion of Objectivity. Acad Med. 2019;94(4):
469 . DOI: 10.1097/ACM.0000000000002413 - Ryan MS, Randall J. Gaslighting the Racially and Ethnically Minoritized Medical Student: How US Medical School Assessment Practices Perpetuate Systemic Inequities. Teach Learn Med. Published online March 12, 2025;1–16. DOI: 10.1080/10401334.2025.2477094
- Gingerich A, Kogan J, Yeates P, Govaerts M, Holmboe E. Seeing the “black box” differently: assessor cognition from three research perspectives. Med Educ. 2014;48(11):1055–1068. DOI: 10.1111/medu.12546
- Hodges B, Lingard L. eds. The Question of Competence. Cornell University Press; 2014. Accessed February 8, 2024.
https://www.cornellpress.cornell.edu/book/9780801479502/the-question-of-competence/ - Teunissen PW, Watling CJ, Schrewe B, et al. Contextual Competence: How residents develop competent performance in new settings. Med Educ. 2021;55(9):1100–1109. DOI: 10.1111/medu.14517
- Hu K, Hicks PJ, Margolis M, et al. Reported Pediatrics Milestones (Mostly) Measure Program, Not Learner Performance. Acad Med. 2020;95(11S Association of American Medical Colleges Learn Serve Lead: Proceedings of the 59th Annual Research in Medical Education Presentations):S89–S94. DOI: 10.1097/ACM.0000000000003644
- Edgar L, Jones MD, Harsy B, Passiment M, Hauer KE. Better Decision-Making: Shared Mental Models and the Clinical Competency Committee. J Grad Med Educ. 2021;13(2 Suppl):51–58. DOI: 10.4300/JGME-D-20-00850.1
- Schumacher DJ, Michelson C, Winn AS, Turner DA, Elshoff E, Kinnear B. Making prospective entrustment decisions: Knowing limits, seeking help and defaulting. Med Educ. 2022;56(9):892–900. DOI: 10.1111/medu.14797
- Schumacher DJ, Martini A, Michelson C, Turner DA, Winn AS, Kinnear B. A realist evaluation of prospective entrustment decisions in paediatric residency clinical competency committees. Med Educ. 2025;59(3):292–301. DOI: 10.1111/medu.15530
- Maranich AM, Hemmer PA, Uijtdehaage S, Battista A. ACGME Milestones in the Real World: A Qualitative Study Exploring Response Process Evidence. J Grad Med Educ. 2022;14(2):201–209. DOI: 10.4300/JGME-D-21-00546.1
- Tavares W, Kuper A, Kulasegaram K, Whitehead C. The compatibility principle: on philosophies in the assessment of clinical competence. Adv Health Sci Educ Theory Pract. 2020;25(4):1003–1018. DOI: 10.1007/s10459-019-09939-9
- Brennan RL. Generalizability Theory. Springer; 2001. DOI: 10.1007/978-1-4757-3456-0
- Ryan MS, Gielissen KA, Shin D, et al. How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study. Med Educ. 2024;58(7):825–837. DOI: 10.1111/medu.15291
- Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37(9):830–837. DOI: 10.1046/j.1365-2923.2003.01594.x
- Huynh A, Nguyen A, Beyer RS, et al. Fixing a Broken Clerkship Assessment Process: Reflections on Objectivity and Equity Following the USMLE Step 1 Change to Pass/Fail. Acad Med. 2023;98(7):769–774. DOI: 10.1097/ACM.0000000000005168
- Agolia J, Green A, Spain DA, Choi J. Diminishing Objectivity in the Residency Application Process. JAMA. Published online February 6, 2025. DOI: 10.1001/jama.2024.28397
- Dickey CC, Thomas C, Feroze U, Nakshabandi F, Cannon B. Cognitive Demands and Bias: Challenges Facing Clinical Competency Committees. J Grad Med Educ. 2017;9(2):162–164. DOI: 10.4300/JGME-D-16-00411.1
- Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane’s framework. Med Educ. 2015;49(6):560–575. DOI: 10.1111/medu.12678
- Sadler DR.
Making Competent Judgments of Competence . In: Blömeke S, Zlatkin-Troitschanskaia O, Kuhn C, Fege J, editors. Modeling and Measuring Competencies in Higher Education: Tasks and Challenges. Professional and Vet Learning. SensePublishers; 2013. pp. 13–27. DOI: 10.1007/978-94-6091-867-4_2 - Emke AR, Torre D, Aagaard EM. Competency-Based Education and Programmatic Assessment in Undergraduate Medical Education: Describing the Gateway Experience of Successful Design and Implementation. Acad Med. 2025;100(11):1268–1271. DOI: 10.1097/ACM.0000000000006141
- Burk-Rafel J, Sebok-Syer SS, Santen SA, et al. TRainee Attributable & Automatable Care Evaluations in Real-time (TRACERs): A Scalable Approach for Linking Education to Patient Care. Perspect Med Educ. 2023;12(1):149–159. DOI: 10.5334/pme.1013
- Anderson HLK, Xu X, Edwell A, et al. How Might we Build an Equitable Future? Design Justice, a Counternarrative to Dominant Approaches in Medical Education. Teach Learn Med. 1–9. DOI: 10.1080/10401334.2024.2404008
- Kakara Anderson HL, Govaerts M, Abdulla L, Balmer DF, Busari JO, West DC. Clarifying and expanding equity in assessment by considering three orientations: Fairness, inclusion and justice. Med Educ. Published online September 16, 2024. DOI: 10.1111/medu.15534
- Ryan M, Chen A, Nichol H, et al. Assessment fit for (more than one) purpose? Reconciling the dual purposes of assessments in competency-based health professions education. Perspect Med Educ., 2026;15(1):946–958. DOI: 10.5334/pme.2592
- Kinnear B, St-Onge C, Schumacher DJ, Marceau M, Naidu T. Validity in the Next Era of Assessment: Consequences, Social Impact, and Equity. Perspect Med Educ. 2024;13(1). DOI: 10.5334/pme.1150
- Streiner DL, Norman GR, Cairney J. Health Measurement Scales.
- Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638–645. DOI: 10.3109/0142159X.2010.501190
- Pusic MV, Boutis K, Hatala R, Cook DA. Learning curves in health professions education. Acad Med. 2015;90(8):1034–1042. DOI: 10.1097/ACM.0000000000000681
- Ryan MS, Lomis KD, Deiorio NM, Cutrer WB, Pusic MV, Caretta-Weyer HA. Competency-Based Medical Education in a Norm-Referenced World: A Root Cause Analysis of Challenges to the Competency-Based Paradigm in Medical School. Acad Med. Published online March 24, 2023. DOI: 10.1097/ACM.0000000000005220
- De Champlain, AF.
Standard setting methods in medical education: high-stakes assessment . In: Swanwick T, Forrest K, O’Brien BC, editors. Understanding Medical Education: Evidence, Theory, and Practice. Association for the Study of Medical Education; 2019. pp. 347–359. DOI: 10.1002/9781119373780.ch24 - Murray KE, Lane JL, Carraccio C, et al. Crossing the Gap: Using Competency-Based Assessment to Determine Whether Learners Are Ready for the Undergraduate-to-Graduate Transition. Acad Med. 2019;94(3):338–345. DOI: 10.1097/ACM.0000000000002535
- Busari JO, Chan MK, Schwartz A, et al. CBE and the missing ‘E’: Making competency-based education an equitable framework for a majority-world health professions education. Perspect Med Educ., 2026; 15(1):880–893. DOI: 10.5334/pme.2551
- Park YS, Hamstra SJ, Yamazaki K, Holmboe E. Longitudinal Reliability of Milestones-Based Learning Trajectories in Family Medicine Residents. JAMA Netw Open. 2021;4(12):
e2137179 . DOI: 10.1001/jamanetworkopen.2021.37179 - Lifelong learning and nurses’ continuing professional development, a metasynthesis of the literature - PMC. Accessed May 19, 2026.
https://pmc.ncbi.nlm.nih.gov/articles/PMC8045269/ - Setting standards on educational tests - Norcini - 2003 - Medical Education - Wiley Online Library. Accessed December 17, 2025.
https://asmepublications.onlinelibrary.wiley.com/doi/10.1046/j.1365-2923.2003.01495.x - Tavakol M, Dennick R. Modelling the Hofstee method reveals problems. Med Teach. 2014;36(2):181–182. DOI: 10.3109/0142159X.2013.848974
- Bok HGJ, de Jong LH, O’Neill T, Maxey C, Hecker KG. Validity evidence for programmatic assessment in competency-based education. Perspect Med Educ. 2018;7(6):362–372. DOI: 10.1007/S40037-018-0481-2
- Sun T, Park YS, Chen F, Hogan SO, Tanaka P. Longitudinal Reliability of Milestones Learning Trajectories during Anesthesiology Residency. Anesthesiology. 2025;142(5):918–928. DOI: 10.1097/ALN.0000000000005388
DOI: https://doi.org/10.5334/pme.2593 | Journal eISSN: 2212-277X
Language: English
Page range: 959 - 969
Submitted on: Mar 21, 2026
Accepted on: Jul 7, 2026
Published on: Sep 29, 2026
Published by: Ubiquity Press
In partnership with: Paradigm Publishing Services
© 2026 Michael S. Ryan, Claire Touchie, Jason R. Frank, H. Carrie Chen, Benjamin Kinnear, As members of the International Competency-Based Health Professions Education Collaborators, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.