A Qualitative Textual Analysis of Feedback Comments in ePortfolios: Quality and Alignment with the CanMEDS Roles

Sofie Van Ostaeyen; Mieke Embo; Tijs Rotsaert; Orphée De Clercq; Tammy Schellens; Martin Valcke

doi:10.5334/pme.1050

References

Holmboe ES, Sherbino J, Englander R, Snell L, Jason R. A call to action: the controversy of and rationale for competency-based medical education. Med Teach. 2017; 39(6): 574–81. DOI: 10.1080/0142159X.2017.1315067
Open DOI Search in Google Scholar Back to article
Janssens O, Embo M, Valcke M, Haerens L. An online Delphi study to investigate the completeness of the CanMEDS roles and the relevance, formulation, and measurability of their key competencies within eight healthcare disciplines in Flanders. BMC Med Educ. 2022; 22(260): 1–14. DOI: 10.1186/s12909-022-03308-8
Open DOI Search in Google Scholar Back to article
Frank JR. CanMEDS: a framework for teaching and assessing competencies. In: Sherbino J, Frank JR (Eds.) Educational design a CanMEDS guide for the health professions. Ottawa: Royal College of Physicians and Surgeons; 2011. pp. 17–22.
Search in Google Scholar Back to article
Rodger S, Webb G, Devitt L, et al. A clinical education and practice placements in the allied health professions: an international perspective. J Allied Health. 2008; 37(1): 53–62.
Search in Google Scholar Back to article
Steven K, Wenger E, Boshuizen H, Scherpbier A, Dornan T. How clerkship students learn from real patients in practice settings. Acad Med. 2014; 89(3): 469–76. DOI: 10.1097/ACM.0000000000000129
Open DOI Search in Google Scholar Back to article
Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010; 32(8): 676–82. DOI: 10.3109/0142159X.2010.500704
Open DOI Search in Google Scholar Back to article
Janssens O, Haerens L, Valcke M, et al. The role of ePortfolios in supporting learning in eight healthcare disciplines: a scoping review. Nurse Educ Pract. 2022; 63: 1–12. DOI: 10.1016/j.nepr.2022.103418
Open DOI Search in Google Scholar Back to article
Van De Ridder JMM, Stokking KM, McGaghie WC, Ten Cate OTJ. What is feedback in clinical education? Med Educ. 2008; 42(2): 189–97. DOI: 10.1111/j.1365-2923.2007.02973.x
Open DOI Search in Google Scholar Back to article
Ginsburg S, Watling CJ, Schumacher DJ, Gingerich A, Hatala R. Numbers encapsulate, words elaborate: toward the best use of comments for assessment and feedback on entrustment ratings. Acad Med. 2021; 96(7): 81–86. DOI: 10.1097/ACM.0000000000004089
Open DOI Search in Google Scholar Back to article
Tomiak A, Braund H, Egan R, et al. Exploring how the new entrustable professional activity assessment tools affect the quality of feedback given to medical oncology residents. J Cancer Educ. 2020; 35(1): 165–77. DOI: 10.1007/s13187-018-1456-z
Open DOI Search in Google Scholar Back to article
Lefebvre C, Hiestand B, Glass C, et al. Examining the effects of narrative commentary on evaluators’ summative assessments of resident performance. Eval Heal Prof. 2020; 43(3): 159–61. DOI: 10.1177/0163278718820415
Open DOI Search in Google Scholar Back to article
Renting N, Gans ROB, Borleffs JCC, et al. A feedback system in residency to evaluate CanMEDS roles and provide high-quality feedback: exploring its application. Med Teach. 2016; 38(7): 738–45. DOI: 10.3109/0142159X.2015.1075649
Open DOI Search in Google Scholar Back to article
Fu RH, Cho YH, Quattri F, Monrouxe LV. ‘I did not check if the teacher gave feedback’: a qualitative analysis of Taiwanese postgraduate year 1 trainees’ talk around e-portfolio feedback-seeking behaviours. BMJ Open. 2019; 9(1): 1–9. DOI: 10.1136/bmjopen-2018-024425
Open DOI Search in Google Scholar Back to article
Canavan C, Holtman MC, Richmond M, Katsufrakis PJ. The quality of written comments on professional behaviors in a developmental multisource feedback program. Acad Med. 2010; 85(10): 106–9. DOI: 10.1097/ACM.0b013e3181ed4cdb
Open DOI Search in Google Scholar Back to article
Ginsburg S, Regehr G, Lingard L, Eva KW. Reading between the lines: faculty interpretations of narrative evaluation comments. Med Educ. 2015; 49(3): 296–306. DOI: 10.1111/medu.12637
Open DOI Search in Google Scholar Back to article
Ginsburg S, Van Der Vleuten CPM, Eva KW. The hidden value of narrative comments for assessment: a quantitative reliability analysis of qualitative data. Acad Med. 2017; 92(11): 1617–21. DOI: 10.1097/ACM.0000000000001669
Open DOI Search in Google Scholar Back to article
Hattie J, Timperley H. The power of feedback. Rev. Educ. Res. 2007; 77(1): 81–112. DOI: 10.3102/003465430298487
Open DOI Search in Google Scholar Back to article
Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: A practical guide to Kane’s framework. Med Educ. 2015; 49(6): 560–75. DOI: 10.1111/medu.12678
Open DOI Search in Google Scholar Back to article
de Jong LH, Bok HGJ, Schellekens LH, et al. Shaping the right conditions in programmatic assessment: how quality of narrative information affects the quality of high-stakes decision-making. BMC Med Educ. 2022; 22(409): 1–10. DOI: 10.1186/s12909-022-03257-2
Open DOI Search in Google Scholar Back to article
Ginsburg S. Hidden in plain sight: the untapped potential of written assessment comments. PhD thesis. Maastricht University, Maastricht. 2016. DOI: 10.26481/dis.20160901sg
Open DOI Search in Google Scholar Back to article
Barrett A, Galvin R, Steinert Y, et al. Profiling postgraduate workplace-based assessment implementation in Ireland: a retrospective cohort study. Springerplus. 2016; 5(133): 1–7. DOI: 10.1186/s40064-016-1748-x
Open DOI Search in Google Scholar Back to article
Bleasel J, Burgess A, Weeks R, Haq I. Feedback using an ePortfolio for medicine long cases: quality not quantity. BMC Med Educ. 2016; 16(278): 1–11. DOI: 10.1186/s12909-016-0801-3
Open DOI Search in Google Scholar Back to article
Tham TCK, Burr B, Boohan M. Evaluation of feedback given to trainees in medical specialties. Clin Med J R Coll Physicians London. 2017; 17(4): 303–6. DOI: 10.7861/clinmedicine.17-4-303
Open DOI Search in Google Scholar Back to article
Park S, Griffin A, Gill D. Working with words: exploring textual analysis in medical education research. Med Educ. 2012; 46(4): 372–80. DOI: 10.1111/j.1365-2923.2011.04184.x
Open DOI Search in Google Scholar Back to article
Kivunja C, Kuyini, AB. Understanding and Applying Research Paradigms in Educational Contexts. Int. J. High. Educ. 2017; 6(5): 26–41. DOI: 10.5430/ijhe.v6n5p26
Open DOI Search in Google Scholar Back to article
Cole FL. Content analysis: process and application. Clin nurse Spec. 1988; 2(1): 53–57. DOI: 10.1097/00002800-198800210-00025
Open DOI Search in Google Scholar Back to article
Elo S, Kyngäs H. The qualitative content analysis process. J Adv Nurs. 2008; 62(1): 107–15. DOI: 10.1111/j.1365-2648.2007.04569.x
Open DOI Search in Google Scholar Back to article
Klie J-C, Bugert M, Boullosa B, de Castilho RE, Gurevych I. The INCEpTION Platform: machine-assisted and knowledge-oriented interactive annotation. In: Zhao D (ed.). Proceedings of the 27th international conference on computational linguistics: system demonstrations, 20–26 Aug 2018, Santa Fe, New Mexico, USA. Stroudsburg (PA): Association for Computational Linguistics; 2018. 5–9.
Search in Google Scholar Back to article
Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960; 20: 37–46. DOI: 10.1177/001316446002000104
Open DOI Search in Google Scholar Back to article
Raaum SE, Lappe K, Colbert-Getz JM, Milne CK. Milestone implementation’s impact on narrative comments and perception of feedback for internal medicine residents: a mixed methods study. J Gen Intern Med. 2019; 34(6): 929–35. DOI: 10.1007/s11606-019-04946-3
Open DOI Search in Google Scholar Back to article
Govaerts MJB, van de Wiel MWJ, van der Vleuten CPM. Quality of feedback following performance assessments: does assessor expertise matter? Eur J Train Dev. 2013; 37(1): 105–25. DOI: 10.1108/03090591311293310
Open DOI Search in Google Scholar Back to article
Roshan A, Wagner N, Acai A, et al. Comparing the quality of narrative comments by rotation setting. J Surg Educ. 2021; 78(6): 2070–77. DOI: 10.1016/j.jsurg.2021.06.012
Open DOI Search in Google Scholar Back to article
Davila-Cervantes A, Foulds, JL, Gomaa, NA, Rashid, M. Experiences of Faculty Members Giving Corrective Feedback to Medical Trainees in a Clinical Setting. J Contin Educ Health Prof. 2021; 41(1): 24–30. DOI: 10.1097/CEH.0000000000000322
Open DOI Search in Google Scholar Back to article
Kim B, Rajagopalan A, Tabasky EM, Reddy SS, Topor DR. Sharing perspectives on feedback: a combined resident-faculty workshop. BMC Med Educ. 2022; 22: 1–8. DOI: 10.1186/s12909-022-03253-6
Open DOI Search in Google Scholar Back to article
Nugraheny E, Claramita M, Rahayu G, Kumara A. Feedback in the nonshifting context of the midwifery clinical education in Indonesia: A mixed methods study. Iran J Nurs Midwifery Res. 2016; 21(6): 628–34. DOI: 10.4103/1735-9066.197671
Open DOI Search in Google Scholar Back to article
Bing-You R, Varaklis K, Hayes V, et al. The feedback tango: An integrative review and analysis of the content of the teacher-learner feedback exchange. Acad Med. 2018; 93(4): 657–63. DOI: 10.1097/ACM.0000000000001927
Open DOI Search in Google Scholar Back to article
Christiansen B, Averlid G, Baluyot C, et al. Challenges in the assessment of nursing students in clinical placements: Exploring perceptions among nurse mentors. Nurs Open. 2021; 8(3): 1069–76. DOI: 10.1002/nop2.717
Open DOI Search in Google Scholar Back to article
Li P, Jiang F, Yin L, et al. Perceptions of the CanMEDS competencies of faculty and students in different curriculum systems of a medical school in China. Adv Med Educ Pract. 2022; 13: 1061–70. DOI: 10.2147/AMEP.S367129
Open DOI Search in Google Scholar Back to article
Bugaj TJ, Schmid C, Koechel A, et al. Shedding light into the black box: A prospective longitudinal study identifying the CanMEDS roles of final year medical students’ on-ward activities. Med Teach. 2017; 39(8): 883–90. DOI: 10.1080/0142159X.2017.1309377
Open DOI Search in Google Scholar Back to article
Binnendyk J, Pack R, Field E, Watling C. Not wanted on the voyage: highlighting intrinsic CanMEDS gaps in Competence by Design curricula. Can. Med. Educ. J. 2021; 12(4): 39–47. DOI: 10.36834/cmej.70950
Open DOI Search in Google Scholar Back to article
Puddester D, MacDonald CJ, Clements D, Gaffney J, Wiesenfeld L. Designing faculty development to support the evaluation of resident competency in the intrinsic CanMEDS roles: practical outcomes of an assessment of program director needs. BMC Med Educ. 2015; 15(100): 1–9. DOI: 10.1186/s12909-015-0375-5
Open DOI Search in Google Scholar Back to article
Renting N, Raat ANJ, Dornan T, et al. Integrated and implicit: how residents learn CanMEDS roles by participating in practice. Med Educ. 2017; 51(9): 942–52. DOI: 10.1111/medu.13335
Open DOI Search in Google Scholar Back to article
Renting N, Dornan T, Gans ROB, et al. What supervisors say in their feedback: construction of CanMEDS roles in workplace settings. Adv in Health Sci Educ. 2016; 21: 375–87. DOI: 10.1007/s10459-015-9634-9
Open DOI Search in Google Scholar Back to article
Dudek NL, Marks MB, Wood TJ, et al. Quality evaluation reports: can a faculty development program make a difference? Med Teach. 2012; 34(11): e725–31. DOI: 10.3109/0142159X.2012.689444
Open DOI Search in Google Scholar Back to article
Pelgrim EA, Kramer AW, Mokkink HG, Van Der Vleuten CP. Quality of written narrative feedback and reflection in a modified mini-clinical evaluation exercise: an observational study. BMC Med Educ. 2012; 12(97): 1–6. DOI: 10.1186/1472-6920-12-97
Open DOI Search in Google Scholar Back to article
Van Ostaeyen S, Embo M, Schellens T, Valcke M. Training to support ePortfolio users during clinical placements: a scoping review. Med Sci Educ. 2022; 32(4): 921–28. DOI: 10.1007/s40670-022-01583-0
Open DOI Search in Google Scholar Back to article
Ötleş E, Kendrick DE, Solano QP, et al. Using natural language processing to automatically assess feedback quality: findings from 3 surgical residencies. Acad Med. 2021; 96(10): 1457–60. DOI: 10.1097/ACM.0000000000004153
Open DOI Search in Google Scholar Back to article
Marcotte KM, Ötleş E, Thelen AE, et al. Using natural language processing to determine factors associated with high-quality feedback. Glob Surg Educ – J Assoc Surg Educ. 2022; 1(58): 1–5. DOI: 10.1007/s44186-022-00051-y
Open DOI Search in Google Scholar Back to article
Elamin A, Obeidat M, Davis G. The ePortfolio in UK cardiology training: time for a new digital platform? Br J Cardiol. 2021; 28(3): 1–4. DOI: 10.5837/bjc.2021.031
Open DOI Search in Google Scholar Back to article
Mooney CJ, Pascoe JM, Blatt AE, et al. Predictors of faculty narrative evaluation quality in medical school clerkships. Med Educ. 2022; 56(12): 1223–31. DOI: 10.1111/medu.14911
Open DOI Search in Google Scholar Back to article