1. Introduction
In the US, our efforts to broaden participation in engineering are in response to a history of exclusion of several overlapping demographic groups. Decades of gender segregation in the labor force during the turn of the 20th century meant women were largely confined to the domestic domain while men dominated the industrial domain and the burgeoning field of engineering (Bix 2002; Pawley 2008). Racial and socioeconomic segregation in education (Slaton 2010) and work (Blackmon 2009; Secules 2019; Zussman 1985) restricted access to engineering to middle-class White men. At a time when racial and gendered exclusion was the norm for professional societies, a colonizing White masculinity came to dominate and define engineering practice (Riley, Slaton & Pawley 2014; Tonso 2014; Villanueva, Mejia & Revelo 2018). In parallel, the normative demographic in engineering defined groups of Black, Indigenous, people of color, LGBTQ people, lower socioeconomic status individuals, and people with disabilities as marginal and non-normative within engineering practice (Riley et al. 2014). Efforts to broaden participation in engineering must address this history of inequity and exclusion.
The reproduction of marginalizing culture in engineering education is a complex and persistent problem that does not rely on singular villains, but on multiple actors and systemic root causes. Therefore, we can conceptualize the educational landscape as a complex system with a structural and cultural inertia that reinforces these norms and their associated values and beliefs over time. For researchers of equity in engineering education, this structural and cultural inertia presents a central challenge as well as a significant opportunity. As we choose our individual and collective research agendas, we should consider our contributions as change agents or as unintentionally complicit in systems of power and oppression—not merely as producers of knowledge. We should examine our roles in the engineering ecosystem and how our research and practice uphold or dismantle systems of oppression. We should interrogate our traditional models of research practice, test the assumptions and outcomes of those practices, and critique and push the boundaries of our own work when it fails to initiate change.
In this paper, we introduce the idea of implicit change models associated with prototypical qualitative research designs to clarify the assumptions embedded within each design. We then use the change models to identify pressure points that act as conservative forces that inherently preserve current structures that shape our research. Lastly, we modify the change models and associated research designs to explore ways to initiate more transformative research on equity and inclusion. We hope this discussion bolsters conversation and reflection on the individual and collective impacts of our work to improve equity in engineering education.
1.1 Positionality Statement
Stephen Secules: I am a White cisgender gay man. I came to education research soon after coming out and with a general sense of mission, specifically a mission to shift inequities related to historically excluded demographic groups in STEM. I shifted careers from being an engineer not because I had my heart set on a career in academia, but because of that mission. The first potential PhD advisor I met with told me I perhaps “cared too much” about my causes to put up with the painstaking process of academic research; he thought I may be better suited to non-profit work. On top of that drive to create change, I’m quite an independent thinker—if it appears there’s only one traditional way of doing something, I will question it, change it, or make it my own. During my education research training, I felt boxed in by some of the conventional methods and sought out support for or my own insight to create unconventional methods that seemed more connected to that sense of mission. I think over time that led me to the philosophy and the sorts of change strategies presented in this paper.
Cassandra McCall: I am a White cisgender woman who came to the field of engineering education because I wanted to improve the educational experiences of engineering students. Conducting research in DEI was never an explicit goal for me, but slowly migrated to the forefront of my work throughout my graduate and post-doctoral research. This work was inspired by my own experiences as an undergraduate student as well as stories of collective experience (e.g., “struggling together” through engineering programs) and of individual struggle (e.g., listening to the heartbreak of a friend after the only woman faculty member in her department said that she “wasn’t cut out for engineering”). I began to ask “why”. Why do we have to “struggle” through engineering? Why was I one of two women in my first Intro to Engineering class, or the only woman in my Trigonometry lab? These later shifted to deeper questions surrounding the exclusionary cultures of engineering, but this time, these questions related to “how”. How can we make engineering more welcoming to and inclusive for all students? In working with Stephen, this paper is my attempt to address the “how” in actionable ways that improve the experiences of all engineering students through inclusion.
The ideas presented come from reflections on our own and others’ work and are presented with an awareness and humility, recognizing that many other researchers come to their work with different motivations. While at the early-to-mid part of our careers, we feel an impatience with slow progress on issues of equity and inclusion and a need to promote new approaches with our work. While we recognize that some of our specific goals or solutions may not apply to every reader’s context, we hope that the thought exercise on how to initiate change with education research is useful to many other researchers.
2. Addressing Change in Engineering Education
A number of intersecting priorities and conversations in the engineering education community demonstrate an increasing focus on producing tangible outcomes from our research. In the mid-20th century, the National Science Foundation’s (NSF) criteria was purely based on the potential for knowledge generation, the scientific merit of the proposal, and the qualifications of the investigators. In the 1980s and 1990s, criteria were added to highlight the importance of broader impacts on society (National Science Foundation 1997, 2022). While this discussion of potential broader impact is now familiar to grant proposers, there is a growing acknowledgement that scholarly research must go further in accounting for not only proposed but realized impact and eventual benefit for society (London & Borrego 2017).
We focus our discussion of research impact on creating change. Change is perhaps particularly important within education and broadening participation research; education is a fundamentally pragmatic discipline and broadening participation connotes an active pursuit of an idealized future that does not yet exist. A large body of work on institutional change notes the difficulties with achieving educational change and specific strategies for doing so. Henderson et al. (2011) developed a system for classifying institutional change efforts as either prescribed or emergent and changing either the individual or the environment. For example, the formation of Departmental Action Teams (DATs) is an emergent environmental approach where teams of institutional personnel assemble to create a shared vision and tackle issues together (Reinholz et al. 2017). Faculty learning communities (FLCs) are an example of an emergent individual approach where faculty reflect on their contexts and likely change as individuals through this reflection (Rooney et al. 2020). Some change strategies are identified by institutional change scholars as less effective than others. The prescribed individual approach of disseminating best practices and the prescribed environmental approach of setting policy have been identified as generally ineffective unless combined with other change strategies (Henderson 2011, pp. 971, 975). Therefore, we can infer that prescribed change strategies, including practices disseminated by research experts or enforced by organizational leaders, fail because true change needs to be at least partially organic and driven by practitioners (Froyd et al. 2017; Karlin 2009).
Related to institutional change strategies, a number of tools are familiar to grant proposers for identifying anticipated change. Organizational Change for Gender Equity in STEM Academic Professions (ADVANCE), Revolutionizing Engineering Departments (RED), and Improving Undergraduate STEM Education (IUSE) grants have encouraged or required specific discussion of the theory and systemic infrastructure that guides and supports the change initiative (National Science Foundation 2019, 2020). Reinholz and Andrews (2020) differentiate theories of change from change theories, where a theory of change is “a particular approach for making underlying assumptions in a change project explicit, and using the desired outcomes of the project as a mechanism to guide project planning, implementation, and evaluation”, and a change theory is a “framework of ideas, supported by evidence, that explains some aspect of change beyond a single initiative” (p. 2).
While some other publications will differ on or invert the usage of these words, the distinction is useful. For Reinholz and Andrews this project-specific theory of change is not the simple input-output process model or logic model that is sometimes equated with the theory of change, but should include rationale, interventions, indicators, outcomes, and the specific context and assumptions that enable the change. A change theory, on the other hand, is broader theory, like a published theoretical framework that guides the spirit of a particular initiative. Examples of change theories cited in STEM higher education literature include communities of practice, diffusion of innovations, and expectancy-value theory (Reinholz, White & Andrews 2021).
We note that theory of change and change theory are not frequently discussed in journal publications. Examples in journal publications typically include review papers and theoretical discussions (e.g., Laursen & De Welde 2019; Sigahi & Sznelwar 2022) rather than explicit discussion of the change or impact strategy for an empirical research study. Theory of change and change theory are more frequently discussed in conference publications such as those associated with the American Society for Engineering Education conferences, commonly within the NSF poster session or discussing NSF grants (e.g., Hug, Convertino & Thiry 2021; Koretsky, Magaña & Shuman 2016; Kukreti & Wondimu Aure 2015; Margherio et al. 2018). Additional discussions, similar to this paper, challenge the community to take new approaches to address institutional change and broader impact (Chan Hilton et al. 2019; Smith et al. 2004).
In our review of the engineering education literature, we note a number of key limitations. First, a relatively small number of studies invoke any organized discussion of change, and most of those are related to specific NSF-funded institutional change efforts. In this paper, we call for a broader consideration of the change and impact of all education and broadening participation work, and we particularly focus on research designs. Second, most of the published theories of change are relatively simple input-output models that do not include assumptions, context, or the necessary actions of multiple stakeholders in order to achieve a desired outcome. We do not think that researchers can singlehandedly create meaningful and lasting change, and if there are assumptions about or required steps from other stakeholders to help bring about that change, we should clarify and explicate those assumptions. Third, while broader change theories (e.g., communities of practice) are valuable, many are discussed in a purely theoretical or vague way and do not make clear how a particular theory helps drive an enacted change. Fourth, many institutional change efforts, their accompanying change theories, and theories of change are focused on pedagogical goals (e.g., expanding the adoption of active learning or evidence-based best practices) and do not place enough focus on issues of equity and inclusion (Reinholz, White & Andrews 2021). Given the additional interpersonal dimensions present regarding equity research and action (Secules et al. 2021), we must become more intentional about the change we seek and the individual and collective action required to achieve it.
2.1 A Generic Change Model for Research on Equity in Engineering Education
We begin by suggesting a simple implicit change model for any researchers hoping to create impact. Similar to the nuanced theory of change suggested by Reinholz and Andrews (2020), our change model incorporates some assumptions about the actions of other stakeholders that would be required in order to achieve a desired outcome. In Figure 1, we introduce a generic research process that includes participants, collected data, a researcher who analyzes and synthesizes that data, dissemination products, and audience. This simplified process is a useful starting place to discuss how researchers create change. Identifying these aspects allows us to consider each stage of the research process and its built-in assumptions. For instance, the process of disseminating research in a journal assumes a targeted readership that, if reached, the researcher can spur change. If it is determined that the target audience does not read that journal, or journals in general, the researcher can consider other forms of dissemination that are more likely to spur change.
Figure 1
Simple Model for Research Process.
Next, we use this model to begin to discuss prototypical research in engineering education, and the implied assumptions about creating change that are associated with the models.
3. Two Change Models for Diversity and Inclusion Studies
Here we introduce two prototypical qualitative research designs for broadening participation in engineering: 1) the Qualitative Interviews with Marginalized Students Study and 2) the Inclusive Faculty Best Practices Study. In both cases, we present the research design as if attempting to impact a particular stakeholder: engineering faculty. We recognize this is a specific interpretive choice we are making which may or may not align with the original intentions of the researchers, but it is one possible interpretation of the intended change model when reviewing this literature.
3.1 Change Model 1: The Qualitative Interviews with Marginalized Students Study
First, we discuss the implied change model present in a Qualitative Interviews with Marginalized Students Study, which comprises a large portion of qualitative diversity and inclusion research. Examples include some of our own prior work (Berhane et al. 2020; McCall et al. 2020) and the work of many of our respected colleagues (Blosser 2019; Cech & Waidzunas 2011; Simmons 2012). The typical research design for this type of diversity and inclusion research (Figure 2) is to find students with a shared and marginalized identity (e.g., women, racially minoritized individuals, LGBTQ+, first generation to attend college) and to conduct qualitative interviews with the students in that population. The researcher then codes for themes and presents findings and recommendations in journals and conferences.
Figure 2
Change Model for the Qualitative Interviews with Marginalized Students Study.
We identify the following implicit assumptions regarding this change model. When we recruit students with a shared identity, we are implying that the other intersecting divergent identities and experiences within that group are less salient. This may encourage an essentializing sense for our audiences of knowing each individual identity group. In general, we conceive of qualitative interviews as knowledge generation activities about this identity group, without questioning, demonstrating, or demanding that tangible value is returned to the initial participants, and without attempting to create change within or by the research process. Next, the process of coding for themes has a tendency to compartmentalize experience into representative example quotes, which can decontextualize their underlying meanings and experiences.
Related to dissemination, the model presumes a passive form of publication impact that relies on faculty to read the journal and conference papers in the first place. Given the length and the format of many academic papers and the time-constrained lives of engineering faculty, this is not a reasonable assumption. The model also assumes that the act of reading or listening to the research at a conference, perhaps hearing best practices, will prompt faculty to act more inclusively. This assumption should be questioned on the basis of our own experience—how often have we as researchers sought out and implemented the recommendations of a parallel scholarly field simply because they tell us to—or on the basis of findings from institutional change scholars (i.e., simple dissemination does not create change). While this characterization will surely be a simplification of the intentions and process of some equity and inclusion research, we think it represents a large portion of qualitative research in engineering education and its limited impact on engineering faculty.
3.2 Change Model 2: The Inclusive Faculty Best Practices Study
Next, we look at a slightly less common prototypical research design, the Inclusive Faculty Best Practices study (Figure 3) (e.g., Mejia & Turns 2020; Moriña 2020; Moriña et al. 2020). The inclusive practices study typically consists of finding faculty with reportedly inclusive practices, conducting qualitative interviews with them, coding for themes, and presenting findings and the inclusive best practices in journals and conference papers. This type of study importantly focuses on generating recommendations for pedagogical and institutional practices, rather than on marginalized students’ experiences.
Figure 3
Change Model for the Inclusive Faculty Best Practices Study.
We find the following implicit assumptions in this prototypical research type. First, there is an assumption in this model that faculty with reported inclusive practices are providing an accurate representation of their practices and that those practices are good examples of inclusion. While, as qualitative researchers, we value the perspectives of our participants, we think the faculty may not be best positioned to represent the impacts of their particular teaching practices. We question whether research solely based on interviews can represent “best practices” as much as it represents the perspectives of those who think they are acting inclusively. We did not bring up credibility during the student interview example, since for students, the perception and experience of marginalization is often the most important phenomenon. The same is not true at the faculty level. If faculty participants in inclusive best practices studies have the perspective that they are acting inclusively but are not actually embodying inclusive practices or achieving inclusive outcomes, the recommendations that this research produces are limited.
Further, by coding for themes, the researcher may be breaking up the nuances, tensions, and tradeoffs that any individual faculty member experiences when trying to act inclusively, distilling their experience into key themes and strategies that represent their intentions on their best days. Education is a complex arena, full of contradictions and tradeoffs, and representing that reality and individuals’ ways of navigating it may be more beneficial but may require different ways of collecting and representing the data. Finally, we once again question whether the papers disseminated in this vein can actually spur change in faculty, as disseminating decontextualized best practices alone is not an effective institutional change strategy.
4. Identifying and Reflecting on Barriers to Change
Having investigated common, prototypical equity and inclusion research designs and their limitations for creating change, we consider root causes for the continuation of these status quo practices. In this section, we highlight the inherent value systems and methodological norms that limit a researcher’s ability to initiate change and impact broader culture and policy (Martin et al. 2022; Riley et al. 2014). These systems and norms are indicative of a cultural inertia established by decades of educational policy and research practice. We challenge ourselves and the research community to critically reflect on these values and norms not as permanent, presupposed structures of the research process, but as opportunities to innovate beyond the bounds of traditional forms of research. We organize our review around three different kinds of barriers to change: barriers due to institutional structures, barriers due to cultural values, and barriers due to methodological norms.
4.1 Barriers Due to Institutional Structures
4.1.1. Institutional Review Boards (IRBs)
A shared value among researchers and IRBs is the protection of human subjects in research. However, navigating IRB review processes can be extremely difficult for researchers wishing to include groups that have been traditionally marginalized and minoritized in research, such as minors, people who are pregnant, prisoners, or people with certain forms of disability (Groen et al. 2018). Inclusion of these groups constitute a specific “red flag” that requires additional scrutiny from IRBs through lengthy full-board reviews and additional documentation, which can take weeks or even months to complete. Similar barriers exist for researchers employing methods that rely on emergence, such as grounded theory, ethnography, and phenomenography in which the study cannot be fully articulated at its outset and its continuation relies on findings from prior phases. For example, research questions for grounded theory and ethnography studies are typically emergent and iterative, whereas an IRB would like researchers to outline all research questions, risks, and benefits before beginning any study. While researchers may submit protocol amendments for an emergent study, the review of these amendments may also be lengthy depending on the type of amendment requested and the nature of the study. Laborious IRB processes serve as a force leading to more conservative research designs with traditional methods, and scholars have argued that such policies and procedures stifle research innovation and minimize potential benefits (e.g., Cross, Pickering & Hickey 2015; Dougherty & Kramer 2005; Marshall 2003; Tierney & Corwin 2007). It is much easier for a researcher to conduct a simple interview or survey study and harder to obtain approval for a design that is unconventional (e.g., action research, self-reflection, or video-based methods). As a result, many of the studies conducted within broadening participation often confirm findings already present in the literature and do not promote change. As researchers employing transformative approaches, it is necessary that we create a continuous dialogue with IRBs by working closely with IRB staff and sharing strategies among researchers (Groen et al. 2018). In this way, we can shift the conventions of IRB review by adopting new research approaches and centering marginalized participants.
4.1.2 Peer Merit Review
A number of institutional structures dictate the lives of engineering education researchers by assessing their merit through a process of peer review. Double-anonymized peer review is the gold standard for journal and conference publication processes, while conference grant reviews are partially anonymous. Tenure and promotion policies (T&P) and dissertation defenses are for the most part not anonymous (except for letter writers in T&P) but are hierarchically situated such that transparency does not give much agency to the person being evaluated (Gendron 2008; Jackson et al. 2017). While we are not against peer review, we join other scholars contending that multiple aspects of the peer review process can make for a more conservative and less innovative academic culture (Beddoes 2014; Douglas & Bumbaco 2012). The nature of paper reviews as anonymous means that authors have to imagine the perspective of reviewers, perhaps basing assumptions on norms for the community, and attempt to appease them. A powerful piece of writing or project that pushes boundaries or challenges thinking could upset or confuse some reviewers, but it may not be because the writing or project was ineffective. A piece of writing that adheres to all norms of a community but adds little value or initiates little change can pass through peer review more easily. T&P committees also tend to enact conservative cultural values, which we explicate in the next section.
4.2 Barriers Due to Cultural Values
4.2.1 Topical Novelty
The value of novelty is introduced in our research training, perpetuated in academic discourse, and reinforced in scholarly evaluation practice (e.g., T&P procedures; Gendron 2008). We join scholars who critically question what it means for research to be novel (e.g., Alajami 2020; Cohen 2017) as a potential barrier for transformative diversity and inclusion research. For example, a great indication of the prioritization of novelty, specifically topical novelty, is the use of the phrase “little is known” for the many research topics on which there are still little to no published studies. Although topical novelty can be important, its prioritization raises several significant concerns. First, while little may be published in education research journals about a specific experience or phenomenon, there typically are people who “know” about it. For example, while little may be known—or published—about the experiences of Black veterans in engineering graduate school, the Black veterans in engineering graduate school certainly know of these experiences. Second, prioritizing topical novelty in this way positions participants as exotic subjects worth of study, rather than as collaborators, allies, and co-constructing agents of change. Third, the prioritization of novelty in broadening participation research presumes that “knowing” more about the marginalized individuals is progress in itself. But research processes can be invasive, research knowledge is not inherently benevolent or good on its own, and we should challenge ourselves to be making real and tangible improvements in the lives of our participants and people like them. Fourth, although a topic may be significantly researched (e.g., women in STEM fields, Brainard & Carlin 1998; Faulkner 2009, 2014; Hatmaker 2013; Jorgenson 2002; Morganson et al. 2010), the progress on the issue may be stagnant or slowed (regarding women’s representation; ASEE 2020). This lack of impact necessitates innovation or methodological novelty to realize the transformative impact that has still not been achieved. We encourage researchers to draw from frameworks and employ research practices not typically seen in engineering education and to be receptive toward research that employs these approaches.
4.2.2. Generalizability
Similar critiques can be made of the cultural value of research generalizability. One misconception regarding engineering education research is that generalizability is synonymous with impact, thus qualitative research with a small number of participants is considered low impact as it only applies to those few individuals (Slaton & Pawley 2018). We align with those who have argued that the opposite is often true (Kellam & Jennings 2022; Slaton & Pawley 2018). Topics in diversity and inclusion and educational/institutional change are so deeply contextual such that we need to understand contextual nuances to inform structural and cultural transformation. If only the most universally encountered experiences are worthy of examination, and if only the universally applicable “best practices” are worthy of publishing, we are likely missing the most crucial contextual information available about how marginalization persists or equity is achieved (Pawley 2019). Here, we highlight Eisenhart’s (2009) qualitative forms of generalizability (i.e., transferability), providing rich description of context and theoretical knowledge that can make lessons learned in one context more “generalizable” and transferable. Rather than conceiving of a single individual’s experience as only important to that individual, we can conceive that everyone is an individual with a parallel and different experience, and the process of deeply understanding or changing that experience for the better is worthy of careful and focused attention by researchers.
4.2.3. Impact
As mentioned previously, impact, or the potential to benefit society (National Science Foundation 2014), is becoming an increasingly emphasized indicator of research quality and, consistent with the vision of this paper, must at some level be personally important for all researchers of education and equity. However, impact can come with risks—if we are not reflexive about our position, intentions, and actions within contexts, we can position our impacts as a colonizing savior complex or “fixing” of participants. The engineering education literature includes several examples of students or researchers creating unintended harm through their impacts. In a mechanical engineering service-learning capstone course, students were tasked with creating a ticket tearing device for David, a person with physical and cognitive disabilities, to improve his job performance as a greeter and ticket taker at a local movie theater. Initially, the device was considered a success (Catalano et al. 2000), but shortly after its implementation, David became withdrawn, quit his job, and by the time his case was reexamined by Catalano in 2006, was institutionalized. Nario-Redmond and colleagues (2017) published a similar example in the field of Disabilities Studies highlighting how conducting disability simulations actually reinforced negative perceptions of people with disabilities and made some non-disabled participants fearful of acquiring a disability. Such cases emphasize the need for researchers to carefully reflect on their own assumptions regarding who the research is for and why it is being conducted (Catalano 2006). We should each reflect on the ways that our impact in our respective contexts is colonizing or is making progress to dismantle systems of oppression.
4.3 Barriers Due to Methodological Norms
4.3.1 Data Collection Practices
The norms that shape how and when we collect data and who we choose to collect data from can produce and reproduce systems of power, privilege, and oppression (Martin et al. 2022). Consistently including the same groups of students from the same research-intensive universities reinforces privileged experiences that dominate decisions related to developing policy and practice in higher education (Pawley 2017). Work in broadening participation has sought to intentionally include marginalized and minoritized stakeholders to make a more inclusive educational culture; however, much of this work has been conducted in a piecemeal fashion. Common practice is to consider factors related to research access and participation only when a particular identity dimension or a set of identity dimensions are salient to the study being conducted, such as considering visual accessibility for a study on disability or gender inclusiveness for a study on LGBTQ individuals. But individuals are multifaceted, and it is difficult to narrowly define when research is “about” any particular identity (Secules et al. 2021, p. 14). One can be a visually impaired woman of color who chooses to participate in a broadening participation study that is not disability-focused, but if recruitment and data collection materials are inaccessible, she may be unintentionally excluded from the study and have her voice silenced. As researchers, we must critically reflect on who is included in our research and, equally as important, who is not (Kristensen & Ravn 2015; McCall, Paretti & Simmons 2019). Identifying individuals who are silenced due to the norms we adopt as a research community can catalyze more inclusive and transformational research.
4.3.2 Data Analysis Practices and Demonstrations of Evidence
Norms for qualitative data analysis can limit what we learn about our research contexts, our participants, and how to impact work in broadening participation. In certain interpretive research traditions, the process of discerning meaning from qualitative data involves coding and sorting data excerpts into themes. This thematic analysis process seems to be the most widely upheld tradition in qualitative research and is useful as a pragmatic step for finding and representing patterns across a large dataset. However, the process of breaking narratives into discrete, themed instances can compartmentalize and decontextualize individual identities and experiences, often presenting emergent categories as the key finding instead of a deeper analysis (Hammer & Berland 2013). This practice can result in a superficial analysis that significantly oversimplifies the phenomenon under study. Overcoming this barrier requires a careful contemplation of the experiences, identities, relationships, and contexts for each participant and phenomenon and can be difficult to communicate in isolated, decontextualized quotes. A recommended practice for researchers to “think with theory” (Jackson & Mazzei 2018) and “[interpret] data in light of theory (Douglas 2017, p. 1) can prompt deeper reflection and interpretation of the topic. Codes are not applied based on singular, isolated segments, but as contributors to the holistic understanding of a phenomenon through the varied lenses of individual context and narrative (McCall, McNair & Simmons 2021). Such analyses require different types of qualitative evidence that move beyond single, decontextualized quotes and toward nuanced description. As a research community, we must be more receptive to these alternative approaches for making meaning of and communicating our work.
4.3.3 Dissemination Practices
We distinguish dissemination from other research practices to emphasize its importance as a step in the research process and as a tool for generating and promoting impact. Similar to other barriers we have identified, there are larger structures that inform when, how, and to whom our work is disseminated. For many scholars, the “publish or perish” culture is a harsh reality that directly connects institutional funding with researcher output and is embedded within decisions related to promotion and tenure (Génova & de la Vara 2018; McNeal et al. 2021). Evaluation metrics for faculty emphasize high-impact journal venues and high citation counts (McKiernan et al. 2019). Institutional-centered approaches tend to decouple the researcher from the outcomes of their work by positioning them as generators and producers of knowledge, shifting the responsibility of outcome engagement to the audience, and devaluing more personal and reflective forms of impact (Secules et al. 2021). Other forms of impact include works that are more accessible to individuals outside of academia, such as magazines, newspapers, blogs, podcasts, websites, and outreach programming, are not weighted as heavily in promotion decisions (Alperin et al. 2019). As a result, faculty must make calculated decisions to balance institutional obligations and roles, time, academic norms, and a desire to create social and societal change. Such structures reward faculty who disseminate work by writing highly cited papers using formulaic and conventional approaches that, in reality, do not prompt change (Alperin et al. 2019). While we acknowledge the need for and potential in the dissemination process, we need to intentionally reflect on our own choices and utilization of dissemination as a tool for impact and change.
5. Alternative Change Models for Research on Equity in Engineering Education
Having considered the many barriers that limit transformative research on broadening participation in engineering education, this section asks: How would we create our research if changing realities for equity were our primary goal? Here we present three alternative change models for research on equity in engineering education. Within these models, we revisit some of our own research and identify emerging research that we find has centered change for issues of equity and inclusion in ways that are relatively uncommon for the field. We do not presume that these are the only research models that prioritize change in broadening participation research; we offer them as possibilities to help spark the imagination of other researchers to think beyond the traditions and barriers that typically dictate our research designs.
Two features are worth noting about these alternative change models. First, we presume that broadening participation is an active and tangible activity that should be present in all broadening participation researchers’ work, their collaborations, or their long-term strategic organizing. Each change model focuses on some activity intended to spur change regarding equity and inclusion. Second, we recognize that many academics are still constrained within the academic system and therefore requirements for publishing are an unavoidable reality. Like the tree that falls in the forest but is only heard if a person is present, publication is the academic process that makes our work legible to the academic community. Thus, each model includes dissemination through traditional publication venues and other familiar parts of the research process, which may or may not be positioned as a form of impact or change.
5.1 Alternative Model 1: Student Agency Dialogue Study
First, we revisit the implied change model related to qualitative interview studies with marginalized students. In an adaptation we call Student Agency Dialogue Study, rather than positioning interviews primarily as information gathering activities to be categorized according to a theory or identity group, they serve as settings for critical reflective dialogue that are deeply examined and interpreted. Examples of these studies include Secules, Gupta, Elby, and Tanu (2018) and McCall et al. (2020, 2021). The alternative change model in these studies is shown in Figure 4 below.
Figure 4
Change Model for the Student Agency Dialogue Study.
Within this alternative change model, the researcher engages in dialogue with students not to elicit information about their marginalization to report out, but to promote agency, growth, and reflective consciousness. Because the dialogue is the site of change, analytical procedures center underlying meaning and contextual influences that prompt the researcher to deeply engage with the data and gain a rich understanding of the phenomenon in question (McCall 2021). The dissemination process presents the process of change and supporting student agency. Of course, since the academic dissemination process prioritizes novelty and new information, the novelty of academic papers that fit precisely into this model (i.e., describing the growth in agency of marginalized students) may have limited shelf value. Still, there is significantly more uncharted territory here, in the capacity of interviews, to promote reflection, increase agency, and create change.
Both authors experienced the need for this reconceptualization of the interview study out of experiences conducting traditional interviews with marginalized students. Groen et al. (2018) included interviews that discussed sensitive and difficult topics related to students’ experiences with attempted suicide and sexual assault. In Secules, Gupta, Elby, and Tanu (2018), a study with a marginalized engineering student grew out of Secules’s graduate course and thus did not come with paid interview incentives. In these studies, the unidirectional interview model seemed designed only to take from the participant and not to give back equally. The eventual dialogic interview model subverts the colonial researcher-to-subject hierarchy that characterizes traditional research, and openly asks what tangible value the participant is gaining from the experience.
5.2 Alternative Model 2: Critical Ethnographic Study
Next, we introduce an alternative change model that centers on creating change in a local classroom setting through ethnographic study. Examples of this research design include Secules, Gupta, Elby, and Turpen (2018) and other ongoing research proposal developments by the two authors. The alternative change model in these studies is shown in Figure 5 below.
Figure 5
Change Model for Critical Ethnographic Study.
Qualitative data collection in this change model includes classroom observations, interviews with students, and interactive dialogue with faculty instructors to provide feedback on equity issues. The focus on change centers the classroom context as a malleable social setting, including faculty practices, student interactions, course texts, assessments, seating arrangements, team pairings, and so on. Because the classroom context is the focal site of change, the dissemination process does not merely present observations or ethnographic themes. Rather, it discusses elements of learning, growth, and change for the entire classroom context. Data analysis in the Secules, Gupta, Elby, and Turpen (2018) study required a deep engagement with a theoretical lens to produce insight about processes of marginalization within otherwise ordinary classroom practices. While simple thematic open coding could help as a first step to assess the entire dataset, the focused analysis examined the data deeply and theoretically (i.e., thinking with theory; Douglas 2017; Jackson & Mazzei 2018).
Challenges involved with impact in this change model include the balance between observation to make correct and informed inferences about the classroom content and feedback to address equity issues quickly and iteratively. Some changes can occur on the time scale of a single class period while others require semester-by-semester time scales to intervene. Another challenge can come with anonymity of feedback and privacy protections. If some equity findings are understood within a confidential interview, it may not be ethical to pass those on directly as faculty feedback in the middle of the semester and risk betraying the confidence of that student. A strategy for navigating these conflicts is to formally bracket processes of observation and inference about equity from processes of dialogue and feedback within separate meetings and perhaps separate individuals responsible for them. This can help with making strategic choices over what the findings are, what should be communicated to faculty, and what changes should be attempted or are feasible within a given time frame. By separating out the processes of data collection and formative pedagogical feedback, the many competing issues of participant privacy, empirical quality, pedagogical quality, and equity can be considered and balanced.
5.3 Alternative Model 3: Audio Dissemination Study
Finally, we consider an alternative change model that creates and tests dissemination products to encourage change in practitioners. These studies move beyond traditionally accepted forms of dissemination and toward practitioner accessibility. Examples of these studies include Secules, Kali, and McCall (2022), Kali et al. (2021), and the Inclusive STEM Teaching Project (ISTP 2022). This alternative change model is shown in Figure 6.
Figure 6
Change Model for the Audio Dissemination Study.
This alternative change model is similar to any qualitative interview study, but instead of disseminating knowledge about the student experience, it focuses on shifting faculty views about marginalized students. This research model is designed to bring student experiences more directly to faculty in formats they would be more likely to consume and in ways that preserve emotion and heighten the empathy of the listener. In this design, the researchers position audio and/or video dissemination as a more accessible and more direct format than traditional journal and conference paper dissemination. Within our own work, we also position the student interview dissemination as a form of contextual, qualitative generalizability (Eisenhart 2009): while the stories shared in these works should not be essentialized as representing all students from a particular identity experience, the faculty who listen to the stories will hopefully recognize the depth of marginalized experiences possible within their own students’ lives and increasingly engage their students with a spirit of empathy and curiosity.
General IRB guidelines and several texts on audio dissemination warn against some of the dangers of sharing participant audio directly with wider audiences including loss of anonymity, going “viral” in an unexpected or uncontrollable way on social media or other platforms, and issues of control and authorship (Singer 2019; Weaver & Spiers 2018). To preserve a measure of both participant anonymity and emotion, we synthesize student interviews into narrative scripts and enlist the help of student actors to read them. This process enables us to member check scripts with participants to ensure that the most revealing identifiable details have been removed while simultaneously preserving overall meaning and purpose. We can also provide emotionality instructions to student actors to preserve underlying meaning communicated using tone, pauses, and emphases (Secules et al. 2022). While creative solutions have been necessary to work within some of the research processes and ethical obligations that protect participants, these approaches seem to have led to greater innovation and impact in our research design.
6. Conclusion
6.1 Change Models for Broadening Participation
Each individual in our community has limited but important power regarding systems of oppression. As individuals we need to take greater concern with conceiving of our individual impact and our coordinated role in collective impact. While this paper drew inspiration from and sees resonance with other enactments of change theories/theories of change across the education research community, we also saw the need for tools that help us conceptualize and communicate that impact. Change models are a required feature of certain institutional change grant proposals (e.g., NSF RED), but we have not seen conceptions of change models to help realize the impact of a wider set of education research activities, or in our case, individual education research designs for broadening participation.
Our version of the change model goes beyond some traditional uses of change theories (e.g., citing Communities of Practice as inspiration) and combines some of the detail of a logic model with the big picture insight and assumptions of a theory of change. Thinking through components of participants, data collection, analysis, dissemination, and audience allows a researcher to look critically and reflexively at each step of their own process. With this tool/process, a researcher can look for the ways their research designs have expanded or limited access of their participants, reinforced or deconstructed colonizing power dynamics between the researcher and researched, essentialized identity groups or provided real insight, and communicated to be best understood by their audience or not.
We hope that the change models we have demonstrated in this paper provide a potential tool and thought process for investigating individual and collective impact. The tool is simple, and only a starting place or reminder to each consider our theories of change. We think that if a researcher produces theoretical accounts (e.g., queer theory, critical theory), they should be able to model or speak to the ways those theories can translate into practice for other stakeholders and to assess whether that has been the case. If a researcher produces accounts of student marginalization to enhance understanding of students, they should be able to model and assess how the work helps faculty or other stakeholders in practice. This may not be a direct impact, it may require the collaboration of faculty developers, industry leaders, or university staff, but these collaborations can be modelled and planned for as well. Through this modelling process, perhaps we find that our research is not creating positive impact but is instead inert or harmful; then our reflection can help us surface those uncomfortable realizations and reorient our research.
6.2 A Focus on Change Towards Equity and Inclusion
In many fields, researchers look at “impact” as synonymous with dissemination, citation counts, and journal and scholar rankings. For pedagogically focused education research, there has been a prominent conversation to disrupt this singular view, to define impact instead as the successful propagation of an effective pedagogy to new contexts (Froyd et al. 2017). Stanford et al. (2017) define a progression of increasing interactivity by PIs, with the most interactive researcher strategies being the most likely to support pedagogical propagation. While useful, this is a researcher-centered model, which examines the interactivity of the researcher but does not discuss the availability for interaction by the instructor or other audience.
Instead, we want to invite a broader conversation about impact and change in broadening participation research focused on equity and inclusion. We note that broadening participation research is distinct from pedagogical propagation, and thus the change models and intended outcomes will look different. We note that change must be a collaborative process, and understanding the perspectives, motivations, and resources of collaborating stakeholders in engineering education is important. Centering those perspectives as we create the models for collective impact is important and a practical necessity.
Finally, we note that change and impact are intuitively important to broadening participation but must be operationalized. We want to invite the collective scholarly community to focus on change for equity and inclusion and to name this process, perhaps via an explicit change model, in publications. While we recognize that broadening participation is multi-faceted and that change will look different for different researchers’ project goals, we think increasing our collective precision about that change strategy will help focus and energize the community on the efforts that matter the most.
6.3 A Call for Innovation in Qualitative Research Design
Within this methodological commentary, we have identified the limitations of typical qualitative research designs in enacting change, identified barriers to innovative research designs, and highlighted a few alternative research designs that circumvent or reimagine those barriers. In a spirit of collegiality with those who seek to broaden participation in engineering, we call for more research innovation that positions change towards equity at the center of the research process. This review has demonstrated that change is complex and difficult, and we must focus on change processes as a central phenomenon to move the needle on broadening participation in engineering and engineering education. We should upend siloed traditional concepts of novelty and impact and look at the novelty that comes from innovative approaches to having local impact on equity. To that end, we should promote research designs that prompt us to think deeply about reality and meaning in findings. For example, as faculty name their own inclusive practices (i.e., their intentions), embedded research designs can examine classroom interactions and student experiences (i.e., the impact) to close the loop about the actual complex processes of equity and inclusion (Secules & Masta 2020). Finally, education researchers should design their entire research process with their intended audience in mind. Impact and dissemination should not be afterthoughts but baked into the research process and outcomes as possibilities for instilling change.
6.4 Strategies for the Catalyzing Innovation
The types of innovations we call for require broader strategies for pushing the traditional boundaries of our field. The first strategy we identify is interdisciplinary collaboration. In other traditions inside and outside of research, some of the norms that we identify as barriers in engineering education are radically different from other contexts. In medical traditions, the idea of research-to-practice dissemination is not an add-on after the research is concluded but is conceived of throughout the laboratory and clinical research process (Dickerson 2021). Similarly in activism, organizing, and action research, the question of impact is not an abstract secondary outcome of a formal research study, but is the priority at the start. Involving interdisciplinary collaborators can help us find creative new approaches to problems encountered in traditional engineering education research. Second, we identify how different resources and contexts bring affordances for different types of research, and how cross-context collaborations can open up new possibilities for participant access. For example, Secules is embedded at a Hispanic Serving Institution and has majority historically excluded racial identity group participants in nearly all studies embedded in his home institution. McCall has access to a wide network of university Disability Resources Centers from her prior work and can prioritize that access in other studies that center on or include those with disabilities. By collaborating across these resources, and many more, we can shift beyond the limitations of our most easily accessible participants to create truly inclusive and innovative research designs.
Third, we suggest that sharing more of ourselves in the research process can open up new possibilities. We have written elsewhere about the importance of reflecting on positionality to gain awareness of impact on equity research (Secules et al. 2021). Deciding to break the rules of researcher impartiality has been a key innovation in our prior work (e.g., McCall et al. 2020). While this researcher involvement may seem risky or impermissible, we think it is one of the keys to unlocking more transformative change towards equity. Fourth, we think more strategies should be shared for how to create innovation. Methods sections of particularly innovative studies could include what steps enabled IRB approval. Tenure and promotion policies that value impact over traditional dissemination could be shared and highlighted, and strategies for convincing supervisors to consider the importance of that impact could be the topic of future papers.
Finally, we encourage members of the research community who are interested in promoting innovative and impactful research approaches to actively engage in the peer-review process. Program officers at funding agencies and editors of peer-reviewed journals are always looking for assistance in reviewing proposals and manuscripts and have created programming to facilitate more involvement (e.g., the Journal of Engineering Education Peer Mentored Review Program, Benson 2021). Volunteering for these roles leverages the peer review systems to help support researchers employing unconventional and action-oriented research practices (Martin 2020; SEE 2022).
***
While we have shared many of our own ideas for embracing more transformative equity and inclusion research, we think that the question of how to craft one’s work to center a process of change towards equity is a personal one, one that each researcher will want to ask and answer. We each bring our whole selves, our own gifts and circumstances, to our research. And we each stand within the tide of a great deal of social and cultural inertia towards inequity. From where each of us are standing, in small but impactful ways, and joining together across our community, we will find out what research can do.
Acknowledgements
Thank you to Maimuna Begum Kali who engaged in early discussions about this topics and helped supply literature examples for the prototypical diversity studies. Thank you to the reviewers and editors who provided their feedback on the prior drafts of the paper. The audio dissemination study discussed is supported by a grant from the National Science Foundation (NSF-2114241 and NSF-2114242). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Publisher's Note
This article originally included an error in the Acknowledgements where the grant number was incorrectly listed as EEC-1636446. As of March 22, 2023 this article has been amended by the publishers to display the correct grant numbers: NSF-2114241 and NSF-2114242. Stephen Secules, one of the authors of the article, brought this error to the editor’s attention.
Competing Interests
The authors have no competing interests to declare.
Author Contributions
Secules and McCall co-generated the concept for this paper through conversations about research norms. Secules and McCall both drew on their prior research experiences for examples of barriers/challenges and innovative change models. Secules was first draft author of most sections of the paper, while McCall led much of the revision and editing process.