Figure 1
“Elements of a Critical Review” adapted with permission from Kahlke R, Lee M, Eva KW. Building blocks for critical reviews in health professions education. Journal of Graduate Medical Education. 2023 Apr 1;15(2):186–9.
Table 1
Five recurrent value judgments observed in postgraduate medical education assessment discussions, supported by representative quotes from meeting transcripts and analytic memos. These value judgments—patient safety, learner development, tradition, talent cultivation, and trust in community—illustrate how underlying values shape interpretations of assessment data and contribute to differing perceptions of validity.
| VALUES | VALUE JUDGMENTS | QUOTES AND RATIONALE |
|---|---|---|
| Patient Safety | It is critical to measure trainee performance to determine their fitness for practice, to ensure patient safety | “To kick somebody out of the program. I have to generate reliable evidence… they have irremediable performance gaps.” “Data that are not reliable should not be trusted to predict future competence… it is unethical to use unreliable data to support decisions that impact which trainees advance.” “For CBME, the goal is only to ensure identification of individuals who are not safe.” |
| Learner Development | Trainee improvement and learning outcomes are the core purpose of health professions education and medical education. (appears throughout transcripts where remediation and coaching are described) | “We just say that we would like to get you more time to learn this because you seem to need more time to be able to achieve this.” “We have shifted away from measurement of pass/fail to being about continued development…” “The goal… is to understand a complex situation with the hope that moves the resident forward.” |
| Tradition | Assessment is a cultural tradition that is important to uphold. | “There is a history and a tradition that we are writing within, so we just have to acknowledge that.” “It serves big testing companies… it is designed for those folks… there have been … pressures to [incorporate] psychometrics that exist right now.” “That is the more traditional way of thinking about how we have been doing assessment in the past… mastery at the end.” |
| Talent | Talent should be fostered and developed within learners to allow for maximal impact. | “We actually have not ever told somebody they are not ever going to be able to learn this… we would like to get you more time to learn this.” “Some people can practice in a lot of environments… some people have really narrow practice environments… they have to choose wisely.” “The act of assessment… is used as the starting point to understand a complex situation with the hope that moves the resident forward.” |
| Community | Assessment systems should include and acknowledge the importance of faculty members. Faculty members are essential to accomplish the goals of assessing trainees. | “We don’t have good performance data on faculty… sometimes the problem is the faculty member, not the resident.” “Bigger concerns arise when raters know themselves to be unreliable or invalid…” “Unless you have very tight circles of people that are trained and savvy to interpret those signals… things fall apart.” |
Table 2
Values, value judgments, beliefs, and risks associated with each, to illustrate how core beliefs can shape faculty decisions, and highlight risks that emerge when any single value is emphasized. For each of the 3 values, we have provided an example of a belief statement, potential implications of this belief for education programs, potential risks or consequences, and examples of how the value and belief might be operationalized in the behaviour or decision making of faculty. Observing the behaviours described in the last column creates opportunities to initiate conversations about individual faculty assessment values and beliefs.
| VALUE JUDGMENTS WE DERIVED FROM ASSESSMENT SCHOLARSHIP | ASSOCIATED BELIEFS & IMPLICATIONS FOR EDUCATION PROGRAMS | RISKS & CONSEQUENCES | OPPORTUNITIES TO DISCUSS BROADER PROGRAM VALUES |
|---|---|---|---|
| Safety: Assessment protects patient safety above all else, ensuring that only clinicians who demonstrate safe practice are advanced. | We believe that assessment influences patient safety. Therefore, programs must shape who enters and progresses within the profession. | Prioritizing this belief can justify overly conservative, exclusionary assessment practices that prioritize avoidance of risk over supporting learning—potentially reducing workforce diversity and worsening shortages. Systemic racism and other forms of discrimination are exacerbated by these priorities. | Faculty raise safety red flags, calling for immediate dismissal of trainees, without attending to process. Faculty fail a learner or delay progression whenever they feel unsure, guided by suspicion rather than fair process. Faculty gravitate toward structured checkboxes to minimize the chance of “missing something,” resulting in narrow assessments that overlook growth, context, or accessibility needs. |
| Development: Assessment supports the development of learners’ competence, rather than ranking or differentiating between individuals. | We believe that learners develop competence through meaningful assessment experiences that inform their growth. Therefore, programs are responsible for learners’ success. | Prioritizing this belief may underplay the importance of accountability, allowing significant performance concerns to go unaddressed in the pursuit of promoting growth. These decisions can mask imminent or future risks to patients or teams, inadvertently preventing early identification of patterns that require intervention. Conversely, learners with accessibility needs can be overlooked or excluded. | Faculty hesitate to document or discuss critical feedback to avoid discouraging the learner. Faculty give higher ratings than warranted to maintain a supportive environment. |
| Agility: Assessment evolves in response to context and should be adapted to better support learners and faculty in health professions education. | We believe that assessment practices reflect the contexts in which they are used. Therefore, programs must adapt as those contexts evolve. | Prioritizing this belief may lead to constant revisions of assessment tools and processes, increasing faculty workload and creating ongoing demands that are difficult to sustain alongside clinical and teaching responsibilities. Consequently, increased faculty workload will further strain healthcare services. | Faculty repeatedly update assessment tools in response to each new context, rotation need, or curricular concern, creating an unending cycle of redesign that adds to administrative burden. Faculty tailor processes for each learner or setting, valuing contextualization but unintentionally generating inconsistent expectations and additional work to manage one-off solutions. |