Table 1
Five evaluation focus areas mapped to representative examples of methods, approaches, or frameworks used to evaluate the implementation of undergraduate medical education (UME) or graduate medical education (GME) competency-based health professions education (CBHPE) programs around the globe.
| EVALUATION METHOD, APPROACH, OR FRAMEWORK | SETTING FOR EXAMPLE(S) DESCRIBED | *LEVEL OF TRAINING UME, GME |
|---|---|---|
| 1. Readiness to Implement | ||
| R = MC2 framework | Royal College of Physicians and Surgeons of Canada, Competence by Design [9] | GME |
| 2. Fidelity of Implementation | ||
| Innovation Configuration Mapping | Royal College of Physicians and Surgeons of Canada, Competence By Design [17] | GME |
| Rapid Evaluation | Queen’s University, Canada [12]; Royal College of Physicians and Surgeons of Canada; [17] | GME |
| Realist Evaluation | University of Manitoba, Canada [24] | GME |
| Logic model | Association of American Medical Colleges (AAMC) Core Entrustable Professional Activities for Entering Residency (Core EPAs), 10 school pilot, U.S. [25] | UME |
| College of Family Physicians of Canada – Triple C Curriculum [30] | GME | |
| Other Fidelity-Focused Evaluation | Aga Khan University, Pakistan [31] | GME |
| National program director survey, Finland [32] | GME | |
| Emergency Medicine, Canada CanDREAM Study [33] | GME | |
| 3. Assessment Program Validity | ||
| Extrapolation Inference | Family Medicine residency program, Ambulatory Healthcare Services, Abu Dabi [39] | GME |
| Implications Inference | Association of American Medical Colleges (AAMC) Core Entrustable Professional Activities for Entering Residency (Core EPAs), 10 school pilot, U.S. [25, 43] | UME |
| Comprehensive Validity Argument for Program of Assessment | Accreditation Council for Graduate Medical Education (ACGME) Milestones, U.S. [47, 48] | GME |
| Education in Pediatrics Across the Continuum (EPAC), U.S. [52, 53] | UME, GME | |
| 4. Educational Outcomes | ||
| Descriptive Evaluation | University of Toronto Orthopedic residency program [58] | GME |
| Entrustable Professional Activities (EPAs) in The Netherlands [59] | GME | |
| Stakeholder experience: Resident experience (surveys) | Resident Doctors of Canada (RDOC), Royal College of Physicians and Surgeons of Canada [60] | GME |
| Resident doctors of Fédération des Médecins Résidents du Québec (FMRQ) [61] | GME | |
| Stakeholder experience: Resident experience (interviews and focus groups) | Competence by Design: Surgical and procedure-based Canadian training programs, Canada [62] | GME |
| Competence by Design: Internal Medicine residents, Canada [63] | GME | |
| Large-scale stakeholder analyses | Promotion in Place competency-based time-variable training, U.S. [67] | GME |
| Faculty of Veterinary Medicine, Utrecht University, The Netherlands [68] | UME | |
| Qualitative educational case study | Transitioning in Internal Medicine Education Leveraging Entrustment Scores Synthesis (TIMELESS) Internal Medicine at the University of Cincinnati, U.S. [69] | GME |
| National Core Curriculum implemented in one medical school, Turkey [71] | UME | |
| Mixed Methods Evaluation | Australian Orthopedic Association Training Program, Australia [79] | GME |
| Quantitative Outcome Evaluation | Education in Pediatrics Across the Continuum (EPAC), U.S. [51, 73, 94] | UME to GME |
| Canadian Family Medicine Residency [76] | GME | |
| Pathology residency program at Massachusetts General Hospital, U.S. [77] | GME | |
| 5. Clinical Outcomes | ||
| Evaluation using Objective Structured Clinical Examinations (OSCE) | The COBALIDATION Trial, Spain [83] | GME |
[i] *Level of training: Undergraduate Medical Education (UME) e.g. medical or veterinary school; Graduate Medical Education (GME) e.g. residency and/or fellowship training; Competency-based Health Professions Education (CBHPE).