Investigating the Road to Equity: A Scoping Review of Solutions to Mitigate Implicit Bias in Assessment within Medical Education

Identifying the Research Question

We developed our research question based on a preliminary review of the literature. Our backgrounds also influenced the research question and overall conduct of the review. Our team consisted of 5 health professions education (HPE) researchers based in the US: KM is a second-generation Filipino American woman and clinician-educator interested in actionable change to address implicit bias within medicine and medical education; MS is a White woman HPE researcher interested in the intersection of health disparities, equity gaps, and bias; KB is a queer, disabled, White woman whose research examines the intersection of bias and disability; TW is a medical education researcher, White woman, interested in the intersection of bias and race/ethnicity; and LM is a White woman with a background in information science and expertise in conducting scoping reviews. Our collective experiences lead us to ask questions about what and how; the existing literature pointed us towards type of bias, interventions, and efficacy.

Identifying Relevant Studies

A medical librarian, in close consultation with our team, searched PubMed, Web of Science, Embase, and ERIC databases on August 23, 2023. The search strategies, optimized for each database, incorporated combinations of keywords and controlled vocabulary terms representing the key concepts of implicit bias, medical education, and assessment. Terms included, but were not limited to: bias, equity, underrepresented in medicine, and racism. Medical education search criteria included: medical students, clerkship, residents, and faculty. Assessment terms included: educational measurement, evaluation, letter of recommendation, and interview. For complete search terms, see Appendix A. All citations were uploaded to Covidence, a knowledge synthesis tool for data management.

Selecting Studies

Our literature search was limited to articles published after 2010, when the Association of American Medical Colleges (AAMC) announced an initiative on achieving health equity through inclusion [16]. Because implicit bias is socially situated – a construct of power that is grounded in historical, cultural, geographic, and political contexts [17] – we focused our review on research conducted in North America. To be included, studies needed to have or propose interventions in mitigating implicit bias in medical education. We excluded studies focused on mitigating implicit bias for patient care and clinical decision making, which is important, but outside this study’s scope. We also excluded interventions for fields outside of medical education (e.g. pharmacy).

Two investigators independently screened the article’s titles and abstracts for eligibility. The first author reviewed all articles; the others served as second reviewers. We met biweekly for one year to clarify ambiguities and resolve discordance in eligibility determination. We repeated the process for the full-text screening in which two authors independently read full text articles that met the initial inclusion criteria; disagreements were resolved via group consensus.

Charting the Data

Before extracting the data, we piloted and refined a data charting tool for extraction. Extracted data included: publication type, year, geographic location, and medical specialty. We noted the type(s) of implicit bias addressed and separated interventions into level: individual, interpersonal, program, institutional, or policy. We then further subcategorized interventions into targets and types. If the article was a study, we noted how efficacy was measured and if the intervention(s) was/were effective at mitigating implicit bias. We also extracted: whether the first author self-identified as being a member of the group addressed in the article, the geographic location and institutional affiliation of the first author, whether a theoretical framework was used, and what the article noted as limitations and future directions. We included this information to better understand the circumstances and positionality with which the researchers approached solutions to mitigate implicit bias.

Collating, Summarizing, and Reporting the Results

We consolidated and synthesized the data, analyzing it with quantitative and qualitative lenses. Quantitatively, we described the corpus to include the kind of publication, year, type of intervention, etc. to provide an overview of the articles’ nature, extent, and distribution. Qualitatively, we reviewed the data pertaining to our research questions and identified the current state of knowledge on bias mitigation in medical education. KM and KB used descriptive, In Vivo, and process techniques [18] to code the data and then engaged in thematic analysis [1920]. Throughout the process, we used multiple coders to address coding reliability and created an interpretive codebook to maintain a shared understanding among authors. We also engaged in ongoing reflexivity via memoing – recording our insights and interpretations of the data throughout the research process – and discussed our positionality and varied experiences with implicit bias during team meetings.

Once the data were organized and analyzed, we classified the interventions into Anderson et al’s three orientations. For instance, one study that attempted to address equity in Honor Medical Society selections described expanding eligibility criteria from >50% honors grades in core clerkships to >25% honors grades in core clerkships with a goal to increase eligibility and selection outcomes for UIM vs non-UIM students [21]. Given the goal of the intervention, this article was labeled as “Assessment for Inclusion (AfI)” because it targeted equity through inclusive practices.

Consultation

We shared our results with 3 experts (including Graduate Medical Education leaders and diversity, equity, and inclusion experts) via email to check the relevance of our findings with their experience. While overall the findings resonated with the stakeholders, we incorporated minor edits to delineate assessment definitions and added a discussion of the Computer-Based Assessment for Sampling Personality Characteristics (CASPer); a situational judgment test to measure noncognitive skills as a component of holistic review [2223].

What forms of implicit bias are being addressed?

Anti-bias interventions were proposed and/or designed to minimize bias for different populations of marginalized learners. The majority (n = 37; 69%) targeted UIM individuals, who experience racial disparities and implicit bias throughout their education and in traditional residency selection metrics [6566]. Implicit gender bias was the next most common (n = 22; 41%). This was often described in relation to gender-based differences in the traits ascribed to trainees and highlighted when performing roles as a leader, manager, or within a team [67]. Articles often addressed multiple groups of marginalized learners (i.e.: UIM and women), but not the intersection of both identities. Thirteen articles (24%) used broad phrases, such as “unseen bias” [31] or “bias in the residency application process [30].” The least common was socioeconomic disadvantage (n = 7; 13%), in which authors argued that when implicit bias is coupled with lack of equal opportunities due to structural barriers [68], the disadvantaged experience poorer academic outcomes [69].

What areas of medical education were targeted and what types of interventions were studied or proposed?

Eighty percent (n = 43) described the implementation of interventions and reported outcomes. Eleven articles (20%) featured interventions or potential strategies for mitigating bias. For example, a perspective article proposed developing institutional procedures to evaluate assessments [26] while another discussed the potential of implementing mandatory anti-bias training for evaluators and establishing clear criteria for competency-based assessments [70]. Of the articles that implemented an intervention, 30 (70%) occurred at the program level, 11 (26%) at the institutional level, and two (5%) at both the interpersonal and individual levels [3071].

We categorized the interventions using Anderson et al.’s equity assessment orientations of fairness (n = 43; 96%), AfI (n = 22; 49%), and justice (n = 0; 0%), noting that some articles overlapped orientations. Authors that were fairness-oriented adopted practices and strategies to increase quality/validity of assessment tools, control the quality of group decision-making, and improve assessor expertise/agility. AfI primarily focused on implementing tools for shared decision making instead of co-designing flexible, individualized assessment systems, which is the goal of inclusion equity. There were no studies where a structural approach was taken to acknowledge differences in performance (i.e. justice-orientation).

Anti-bias interventions targeted six areas of medical education. We describe interventions within each category of implementation below (see Table 2).

How did authors describe the efficacy of interventions?

Most studies (85%) examined a single program or institution; therefore, descriptions of efficacy were context-bound. Below, we map how authors characterized efficacy in the 43 studies that reported outcomes. We did not assess the study’s efficacy; that type of analysis is typically outside the bounds of a scoping review.

Frequently (n = 22; 51%), efficacy was measured by comparing the percentage of marginalized learners selected for interviews, admissions, and residency/fellowship programs in terms of pre- and post-intervention(s). This approach aligns with the fairness-orientation, particularly around tracking outcome equity.

Fifteen (35%) studies framed efficacy in terms of increasing the number of marginalized learners at specific points of interaction (e.g., interview invitations, admission, and/or matches). All of these studies were multipronged, incorporating variations of: diverse evaluating faculty, expanding test score eligibility, qualitative test scores, incorporating additional questions, independent reviewers, increased recruitment, masking interviewers to the applicant’s academic record, and using standardized interview questions [2127294650555761727375767779].

Nine (21%) studies measured the efficacy of interventions with surveys on knowledge, skills, or attitude changes before and after the intervention. The majority of these studies were faculty trainings; survey questions asked about training satisfaction, knowledge and skills around bias recognition and mitigation, pre- and post- Implicit Association Tests (IATs) [30455159606480]. Other studies measured perspectives on the blinded interview process [38] and on subjective utility of an added question to an application [42]. No studies in this category measured change or impact on evaluations, promotion, or advancement for marginalized learners.

Seven (16%) studies did not achieve the goal of mitigating implicit bias in assessment. Two studies were algorithms that applied specific scores and weights to application components [6263]. One study masked MCAT scores during the admissions process [25], one masked name and photographs during the screening processes [32], and one masked interviewers to test scores, transcripts, and LORs [34].

One (2%) study had incongruent efficacy for gender and UIM bias, demonstrating that behavioral-based interviews mitigated implicit bias for UIM bias, but not for gender bias [56].

Types of Bias Addressed

Several studies sought to mitigate bias for UIM, gender, and/or socioeconomic disadvantage or often addressed multiple groups of marginalized learners (ie: UIM and women). Yet none addressed the intersection of these biases – overlooking that some learners have unique contexts necessitating interventions that target intersectionality. Further, almost a quarter of the articles did not specify what type of bias they were trying to mitigate, positioning bias as nonspecific and uniform. Understanding the origins and effects of biases is integral to modifying or creating assessment systems. Confirming recent reviews [1112]; ableism, heterosexism, and religion bias were minimally studied.

Targets and Types of Interventions

Interventions to mitigate bias in assessment were focused on the orientations of fairness and AfI, which do not fundamentally acknowledge, incorporate, or shift the structures in society that created bias. No interventions targeted a justice-orientation, which would mitigate bias through co-designed assessment systems with community-led and community-controlled decision-making, acknowledging that what is assessed is frequently designed with the more typical student in mind, those privileged by societal structures. We theorize that this is because it is difficult to dismantle current assessment systems. Rethinking assessment systems for justice is time intensive and would require many stakeholders at multiple points in the system, not the least of which would be regulatory bodies. However, a justice-orientation would explicitly address the root causes of implicit bias and acknowledge that assessment cannot be truly impartial. Further, it would empower marginalized populations to collaborate on assessment systems to bring the medical education community one step closer to equity.

Implicit bias exists because of systems and structures in place that have disproportionately disadvantaged marginalized learners for centuries. Interventions targeting fairness or AfI orientations, like masking academic records, names, and photographs or incorporating CASPer scores as a surrogate for noncognitive skills in the admissions screening process, only skim the surface. Marginalized learners are disadvantaged at multiple levels in the pre-application process beyond test scores, GPAs, and clerkship grades. This includes the ability to engage in extracurricular activities, volunteer work, leadership positions, and scholarly activity. Interventions are being developed outside of the communities that they affect and, instead, need to be co-developed with these learners.

Efficacy of Interventions

The strategies and interventions implemented were frequently effective in bias mitigation for marginalized learners when used in tandem more as a program of assessment towards equity, rather than a single effort [212729464750555761727375767779]. However, researchers’ multipronged approaches made it difficult to assess the most effective anti-bias intervention and provided limited evidence as to what worked and why.

A similar problem occurred with faculty training, which was a common intervention. The efficacy of faculty training was measured with pre/post-intervention surveys [30454951596064], reflecting short-term changes, but there were no longitudinal studies that measured whether faculty training resulted in lasting change or improvement in outcome equity. By focusing only on the intervention itself, researchers miss a developmental perspective that would enable permanent shifts or change. Similarly, mandatory implicit bias training was frequently mentioned as an intervention [27333738394547495051545558596470727380], but limited details were discussed regarding its content or quality, leaving readers with little guidance on how to implement such efforts.

Finally, efficacy was often studied with a fairness-orientation, particularly around tracking outcome equity (i.e. percentage of marginalized learners that applied and those selected). Such an approach is meant to examine how populations increase as a result of the intervention but is a weak form of analysis because it does not attend to nuances in outcomes. For example, although the number of Black individuals may have been advanced or selected for a program using a particular intervention, merely measuring the number of Black residents pre/post intervention tells one nothing about whether these individuals were immigrants from the Caribbean or Sub-Saharan Africa or are descended from slavery, and have therefore experienced generations of racism, discrimination, and educational barriers.