Humanizing Data-Driven Methods in Engineering Education Research: A Systematic Literature Review of Four Journals From 2011 to 2021

Opportunities and Issues with Data-Driven Methods

When we discuss data-driven methods, the closest family of techniques that may come to mind is machine learning. Machine learning methods (sometimes cast as educational data mining more broadly) build models by explicitly learning from data that can become adept with prediction and classification tasks. Hence, the data is the mechanism for building the model instead of a researcher testing a presupposed model. Given a large enough dataset, these methods seem to offer a promising means for educational researchers to uncover hidden patterns in service of improving various educational processes (Baker & Yacef 2009).

There are two common types of machine learning approaches—supervised and unsupervised. Supervised machine learning approaches are how people typically perceive such techniques; researchers gather data labeled with some categories (often called classes or class labels) as examples and train the model by showing it these examples. Then, the researchers test the resulting model with data the model has yet to see. Supervised models include decision trees, random forests, Naïve Bayes, neural networks, and other algorithms that try to model the relationship between dependent variables (the classes) and independent variables used during training (Hilbert et al. 2021). The unsupervised machine learning algorithms are exploratory and applied to data where the categories are unknown. The algorithms aim to learn the class labels; typical techniques include cluster analysis, self-organizing maps, and association analysis.

Data-driven approaches, like those offered by machine learning methods, offer a robust framework for analyzing complex, multi-variable, and often non-linear data in ill-defined high-dimensional spaces (Hilbert et al. 2021). In particular, machine learning techniques have become popular in educational research in recent years for addressing a wide variety of purposes, such as developing intelligent tutoring systems, classifying students’ text-based feedback, and predicting students’ academic performance (Korkmaz & Correia 2019). Accordingly, Hilbert et al. (2021) proposed that developing and employing interpretable machine learning techniques (e.g., individual conditional expectation plots and counterfactual explanations) can help researchers discover the sources of inequalities and make corresponding modifications to reduce the potential unfairness; they also suggest researchers be more cautious on the selected input features which are fed into machine learning algorithms—particularly demographics and other social identities. For example, Kotsiantis (2012) showcased that students’ demographics (e.g., gender, marital status, and age) could be used to predict their academic performance. In another study, Kosinski et al. (2013) demonstrated that an individual’s sexual orientation, political views, and other personal attributes with some demographics (e.g., ethnic origin) can be easily predicted with great accuracy by just using people’s likes on their Facebook account, even though they did not intend on sharing such information. These personal attributes are vulnerable to being taken as sources of discrimination if the models are used for more nefarious purposes (Hilbert et al. 2021). Thus, researchers have been building more sophisticated and fair machine learning algorithms to employ sensitive variables like demographics and other personal attributes in the model and mitigate their discriminatory potential (Kilbertus et al. 2020; Kusner et al. 2017).

Despite the analytical power of employing data-driven methods in educational research to detect and minimize different kinds of inequalities rooted in our educational system, Hilbert et al. (2021) warned that their careless application can reinforce existing biases—leading to algorithmic bias and discrimination. For example, hidden feedback loops, where a system’s output is used as an input in future iterations, can potentially perpetuate the extant inequalities when machine learning algorithms are used in contexts that demand more criticality or sensitivity (Kusner & Loftus 2020), especially sub-groups that are vulnerable to the decisions made using the algorithm (Lu et al. 2018), for example, predictive policing. Data-driven approaches adopt a bottom-up framework that prioritizes the relationships embedded within the data itself, devoid of researchers’ presumptions—in theory, at least. This contrasts with traditional statistical methods that follow a top-down approach and aim to establish causality between variables and outcomes (Qiu et al. 2018).

The inductive nature of data-driven approaches is often described as “letting the numbers speak for themselves” (Anderson 2008, p. 2). However, this interpretation becomes questionable when we critically reflect on the origins of these numbers. We have control over how we measure, what we measure, and how we analyze the data. Even if the analysis is claimed to be value-free and without a preconceived model, human choices shape the inputs and are guided by normative values. Consequently, Gillborn et al. (2018) caution that the presumed objectivity associated with quantitative research adopting data-driven method(s) carries the risk of reinforcing racist stereotypes and power structures since “numbers are social constructs and likely to embody the dominant (racist) assumptions that shape contemporary society” (p. 173). These issues can manifest in various contexts, as highlighted by Cathy O’Neil (2017) in the book, Weapons of Math Destruction, which demonstrates how data-driven methods can be exploited for malicious purposes, both intentionally and unintentionally, within domains such as the justice system, job applications, and online advertising. Echoing Irving and Askell (2019), ensuring that machine learning models remain firmly aligned with ethical values will present an ongoing and consistent challenge.

In terms of ethical issues, the broader implementation of data-driven methods can be critiqued, including how the data is sourced to build different models. There are open questions about how to make techniques like those used in machine learning more trustworthy (Shokri et al., 2021), and this discussion of the ethics behind data-driven approaches has been even more pronounced with the public release of generative AI systems like ChatGPT (Wach et al. 2023). When the data used to train a model is poorly sourced, whether in terms of quality or stakeholder representation, the results are also impacted. Facial recognition is one of the most common examples of oversight in data collection with data-driven applications, where systems frequently misidentify non-White and non-male individuals because of the large proportion of the training set being White male faces (Waelen 2023). Therefore, researchers must be keenly aware of their dataset’s quality and disclose its limitations if issues cannot be addressed.

How researchers source the data for data-driven methods is also a contentious issue, especially regarding how consent was obtained from individuals to use their data for different purposes (Jo & Gebru 2020). One particularly salient example of how researchers sidestepped participant consent is the “Tastes, Ties, and Time” dataset curated from the Facebook profiles of approximately 1700 college students over three timepoints, which included copious amounts of personal information (e.g., home state, sexual interests, and political views) merged with students’ academic records (Zimmer 2010). The students were not given a chance to opt out of data collection or even informed their data had been scraped. Despite attempts to de-identify the data, it was quickly determined that the anonymized college was Harvard University, and the dataset was withdrawn after the researchers faced a wave of criticism for the oversights in human subjects protections. Although the underlying data can be more or less public, like the Facebook profiles from the “Tastes, Ties, and Time” dataset, there is considerable divergence in what individuals perceive as appropriate use of their data by companies and governments (Auxier et al. 2019). For example, in the Pew Research Center survey of Americans in 2019, 41 percent of those surveyed deemed it appropriate for fitness apps to share data with medical researchers. In contrast, only 27 percent agreed with social media companies using post activity and engagement to monitor users for depression. Although data ethics and ensuring the rights of human subjects are upheld are not unique to data-driven approaches, the impact of the models derived from them elevates the concerns—especially considering how the approaches are discussed in the literature.

The Premise of QuantCrit

The aforementioned critiques of data-driven approaches can be associated with the analytical framework of QuantCrit, which draws from Critical Race Theory to examine conventional practices in quantitative research that may have harmful implications. Gillborn et al. (2018) extensively discuss five core principles that encompass QuantCrit: (1) the centrality of racism (recognizing the pervasive nature of racism in society, acknowledging that some scholars argue against its quantifiability); (2) numbers are not neutral (researchers decide what to measure, whom to measure, and how to measure); (3) categories/groups are neither natural nor given (such as race and gender, are socially constructed rather than naturally given); (4) data cannot speak for themselves (data cannot convey meaning on their own and require interpretation); and (5) social justice and orientation (QuantCrit challenges the notion of assumed objectivity and political neutrality in the application of quantitative research). These principles are broad rejections of practices that are more or less accepted in quantitative research (e.g., using race as a predictor in a regression model), and it can be challenging to see practical steps forward for embodying a QuantCrit mindset at times. To make these principles more concrete, the idea of person-centered approaches has been advanced as a path forward for researchers to engage in their work more critically.

Person- and Variable-Centered Approaches

To explain what a person-centered approach constitutes, we draw on work by Woo et al. (2018), who described three distinguishable conceptualizations of the term, person-centered, in scientific research. The first use of the term refers to research focused on individual characteristics rather than situational traits. From this point of view, a study qualifies as person-centered when it examines aspects like personality, skills, or people’s abilities. Such a perspective casts the most extensive net on what counts as a person-centered approach to research, perhaps to the extent that entire disciplines in the social sciences could be classified with the person-centered label—for example, the community of psychologists studying personality could be reasonably deemed as entirely person-centered with this first definition. On the other hand, the second use of the term person-centered is to denote research that emphasizes subjective experiences, in contrast to research concentrating on “objective” aspects of individuals (Weiss & Rupp 2011). Here, person-centered approaches seem most aligned with using qualitative approaches, which can capture more of the diversity inherent to peoples’ experiences.

According to Howard and Hoffman (2018), the third use of the term person-centered refers to a set of analytical methods that aim to identify sub-groupings in a sample based on their similarities in the selected variables. Unlike the previous two conceptualizations of person-centeredness, which focus on the constructs under study, this last perspective shifts the attention to the analytical methods in the study. For this research, we adopt the third conceptualization to discuss person-centered approaches because of our interest in exploring how engineering education researchers offset the potential discriminatory power of data-driven methods through their research design.

Because of their ability to provide detailed insights into the study’s sample, person-centered approaches have gradually gained popularity over the years (Lanza et al. 2013; McCutcheon 1987; Vermunt & Magidson 2004). Engineering education was first introduced to the concept most directly by Godwin et al. (2021), who offered the community this way of thinking to differentiate between the underlying mechanisms of various quantitative approaches. A person-centered approach aims to acknowledge and understand the heterogeneity among individuals in a sample by identifying latent groupings based on shared measurements that reflect individual characteristics and their environment. Thus, person-centered approaches are especially useful when the sample contains diversity that is not readily identifiable using pre-existing categories; moreover, such approaches can also uncover different developmental trajectories in longitudinal settings (Muthén & Muthén 2000). Authors writing about person-centered approaches tend to identify a handful of specific analytical techniques that accomplish the goal of uncovering latent diversity in a sample, such as latent profile analysis, latent transition analysis, growth mixture models, and cluster analysis (Godwin et al. 2021; Hofmans et al. 2020; Howard & Hoffman 2018; Muthén & Muthén 2000).

Among the conventional statistical methods in authors’ collections of person-centered approaches, data-driven techniques like cluster analysis can be seen. Godwin et al. (2021) argued that although data-driven methods may be used to conduct a person-centered analysis, they require great caution and critical thought during implementation. For instance, Principal Component Analysis (PCA) is a data-driven method commonly employed to reduce the dimensionality of a dataset. However, it is not inherently performed in a person-centered way because it consolidates variables into composite quantities (i.e., principal components), potentially obscuring relationships among the original variables by creating new variables that may not be theoretically meaningful or generalizable to other settings, despite its aim to minimize information loss during the transformation.

Researchers are likely more familiar with the parallel concept of person-centered approaches, analyses that are variable-centered (Howard & Hoffman 2018). Unlike person-centered approaches, variable-centered approaches presume the homogeneity of individuals in a sample, and every individual in the sample can be represented by the same set of estimated parameters (Morin et al. 2016). These analyses prioritize prediction and relationships among variables (Laursen & Hoff 2006); accordingly, the commonly used analytical methods for variable-centered analysis include standard statistical techniques, such as ANOVA, regression, and factor analysis (Howard & Hoffman 2018).

When conducting empirical research, it is critical to choose a suitable approach based on the research questions (Howard & Hoffman 2018). Thus, we cannot explore variable-centered and person-centered approaches in a vacuum. Person-centered and variable-centered analyses have traditionally been regarded as distinct tasks, each employing unique models and software (Muthén & Muthén 2000). Although person-centered approaches can provide more comprehensive and detailed insights into groupings within a dataset, it does not imply that they should entirely replace variable-centered methods (Godwin et al. 2021). Neither approach is always a better choice than the other (Howard & Hoffman 2018). Instead of viewing variable and person-centered approaches as a binary choice between right and wrong, researchers are encouraged to select quantitative methods based on their research questions and the strengths of different approaches (Laursen & Hoff 2006).

With this new framework for understanding quantitative methods in engineering education, there is an opportunity to reconsider how data-driven approaches are used within the community. Furthermore, given the fresh perspective of person-centered approaches, it remains unclear how extensively engineering education researchers currently employ these techniques and whether they are applied intentionally with a person-centered mindset. By examining the existing literature for examples of thoughtful applications of data-driven methods using person-centered approaches, these studies can serve as models for future endeavors across various research domains within engineering education.

Research Aims

This study explores how engineering education researchers use data-driven methods and examines how they incorporate person-centered approaches into their analyses. The primary research question guiding this investigation was: “How do engineering education researchers employ data-driven quantitative methods and humanize them through the use of person-centered approaches?” Specifically, we aimed to gain insights into the person and variable-centeredness of how different data-driven methods are applied in practice. Godwin et al. (2021) suggested an inherent person-centeredness or variable-centeredness for various common methods, like cluster analysis and PCA. However, zooming out to the broader scope of the design can showcase how these methods are situated in the broader analytical decisions made by the researchers—unveiling nuanced procedures in preprocessing, post-processing, and their general application.

Positionality of the Researchers

As a Master’s student of engineering education who holds a bachelor’s degree in Applied Statistics, Jiafu, always seeks opportunities to apply his statistical knowledge in engineering education research. More specifically, he has been interested in how those powerful data-driven methods are applied in quantitative educational research and how they handle the associated equity issues brought by using these techniques (e.g., algorithmic discrimination).

As a student from China, oftentimes he doubts the appropriateness of aggregating students from other countries into a single category, “international students,” and generalizing the findings to the entire group, which actually contains many subgroups. For example, language is one of the most significant obstacles faced by many students from other countries when they are studying in the US. However, when international students are treated as a homogeneous group in such research, the inappropriateness is evident because some international students speak English as their native language, but some do not. A project funded by Division D (Measurement and Research Methodologies) of the American Educational Research Association that was funded during his studies, which investigated how engineering education researchers apply data-driven methods and how they adopt person-centered approaches to humanize these methods, perfectly fit his research interests and allowed him to unpack some of the uneasiness he has had about certain practices in quantitative methods as an insider of an underrepresented group.

David supervised Jiafu in this research. David’s positionality has been documented extensively in previous publications (Hampton et al. 2021; Hampton & Reeping 2019). Relevant to this effort, David has been reforming his perceptions of quantitative methods since his first introduction to critical quantitative methods, and has been voicing his concerns about quantitative research practices in engineering education. In light of frustration with more nebulous recommendations from the literature for how to engage in critical quantitative methods, his aims to provide more tangible steps for researchers looking to engage in QuantCrit practices and—in a many cases—uncover what practices authors are already doing that align with the principles of critical quantitative methods. Jiafu and David met continuously throughout the project to discuss one another’s perceptions of what it means to engage in QuantCrit and person-centered analyses, which involved revisiting the data, rethinking categorizations of approaches, and delving into different components of the reviewed papers. Thus, this collaboration reflects a reflective construction of themes that support person-centered analyses and highlight what practices already exist in publications using data-driven methods.

Data Collection and Preparation

The initial stage of data collection in an SLR encompasses defining the inclusion criteria, constructing suitable search terms, and selecting relevant databases, which included each journal’s website and Education Research Complete database. Given the exploratory nature of our SLR, we did not impose stringent inclusion criteria. The following list outlines our inclusion criteria:

• Must use a quantitative or mixed methods research design

• Must use one or more data-driven methods in their analytical approach

• Must be an empirical study

• Must involve human subjects in the data analysis, meaning the unit of analysis is the individual

We were specifically interested in articles that applied a data-driven technique with human subjects to learn more about the individuals in the study; this means that studies conducted to create an instrument were not considered within the scope of this SLR. For example, the data-driven technique, Exploratory Factor Analysis, is frequently used to find correlated groups of items in an instrument. This kind of study would be excluded because the unit of analysis is not the individual—instead, it is the set of items in the instrument. More generally, a data-driven method is an inductive research approach that does not depend on an a priori theory or model.

To create our search string, we combined terms for common data-driven methods, such as “decision tree” and “principal component analysis,” with “OR” as the conjunction. More generic terms, such as “machine learning,” were also added. The specific methods were appended based on techniques discussed in Tan et al.’s (2005) Introduction to Data Mining to include articles that did not make general comments about their approach being data-driven or machine learning. The full search string is provided here:

The articles were collected from four leading engineering education journals: the Journal of Engineering Education (JEE), the European Journal of Engineering Education (EJEE), the International Journal of Engineering Education (IJEE), and IEEE Transactions on Education (TOE). We chose the publication period of 2011–2021 to cover the period where data-driven methods were being adopted into more mainstream engineering education research. We retrieved the articles from Education Research Complete and Wiley’s search engine for the Journal of Engineering Education. The PRISMA flowchart for this study is shown in Figure 1.

Figure 1

A total of 236 articles were retrieved based on the established search string. Following the search phase, we manually screened the abstracts and excluded 140 articles based on the inclusion criteria. Three main reasons led to the removal of articles included (1) the failure to employ a data-driven method (e.g., a data-driven method was mentioned as part of the literature review or future work but not used in the study), (2) the utilization of a data-driven method without involving human subjects, and (3) the absence of an empirical study design. Consequently, 96 articles remained and were categorized based on their primary quantitative methods, including Exploratory Factor Analysis (EFA), Logistic Regression, Cluster Analysis, Principal Component Analysis (PCA), Decision Tree, Random Forest, and Others. A temporary “Others” category was created to accommodate articles where the data-driven methods described did not fit into the aforementioned six categories (e.g., simultaneous use of multiple approaches) or where generic terms like “data mining” or “machine learning” were used to describe the quantitative methods. These articles required further investigation to identify the specific data-driven methods employed by the authors.

Subsequently, we proceeded with a thorough full-text review of the remaining articles to ensure compliance with the inclusion criteria. One significant decision we made was to tentatively exclude the logistic regression group. We observed that articles within the logistic regression category predominantly employed traditional statistical modeling approaches rather than adopting an inductive data-driven framework. We eliminated one article from the PCA group, namely Chan and Fong (2018), because their study discussed the results of PCA from other researchers’ studies rather than directly applying PCA themselves. Furthermore, upon further inspection, Pizard and Vallespir (2017) was also excluded from the Cluster Analysis group since it was later found to not involve human subjects after a closer read. As a result, a total of twenty-four articles remained and advanced to the synthesis stage. Note that we were not specifically interested in making judgments about the quality of the papers we reviewed. Thus, no papers were excluded for issues in quality.

Analysis of the Selected Articles

To gain a comprehensive understanding of how engineering education researchers incorporate person-centered approaches to humanize data-driven analyses, we employed a three-round coding process. In the initial round, we used descriptive and In Vivo coding (Saldaña 2013) to categorize each article’s methods and results/discussion sections. The initial In Vivo coding allowed us to survey the breadth of how the data-driven approaches were used and establish preliminary patterns. In the second round, the papers were revisited to extract additional context related to the researchers’ purpose for using data-driven method and any additional procedures to delve deeper into their findings. Subsequently, in the third round, we aggregated the findings from the previous coding cycles using descriptive coding to abstract the In Vivo codes to highlight the person- and variable-centeredness manifested in the application of each technique in different projects. The final round of coding identified tangible strategies for how researchers can balance person- and variable-centered approaches in their designs. To situate the discussion in the broader conversation about the critical application of quantitative methods, we aligned the findings with the QuantCrit framework. This builds off prior work to provide engineering education researchers with practical methods for engaging in a QuantCrit way of conducting quantitative research (Reeping et al. 2023).

Embedding a data-driven technique within a mixed methods study

One way a researcher could use a data-driven technique as part of a person-centered approach is by embedding it in a mixed methods design to expand upon relationships found in the sample. For example, consider the method topic modeling. Topic modeling refers to natural language processing algorithms designed to extract topics from text (Cutumisu & Guo 2019). One of these topic modeling techniques is Latent Dirichlet Allocation (LDA), which is an unsupervised machine learning algorithm assuming a random mixture of hidden topics in different documents in a corpus (Johri et al. 2011). Cunningham-Nelson et al. (2019) leveraged LDA to summarize and visualize student satisfaction from their open-response comments in the end-of-semester course evaluations. Students were asked to rate close-ended questions on a five-point Likert scale, followed by optional free-text responses. Cunningham-Nelson et al. (2019) pointed out that sometimes, the Likert score, as a quantitative value, cannot accurately reflect students’ complete thoughts on a given closed-ended item. For example, a student gave a score of 3 to one of the close-ended questions in the survey, which corresponds to a neutral attitude. However, the comments written by this student conveyed a relatively strong negative satisfaction with the content taught in that course. Put another way, the comments written by students were able to contextualize students’ satisfactions and struggles embedded in their numeric rating, which provides educators with more informed ideas to improve instruction or curricular design. The scores and open-ended comments occasionally diverged, reflecting the QuantCrit principle, “data cannot speak for themselves.” The numbers themselves can be meaningless or misleading depending on how survey items are designed and presented—and to whom they are presented. This application of topic modeling demonstrates how data-driven methods can help uncover a broad range of student experiences or perceptions. This idea also aligns with the premise of initiation in mixed methods research (Greene et al. 1989), which involves seeking divergence and reconciling those findings.

Another illustration of how data-driven methods can be embedded in a mixed methods study was Faber and Benson (2017), who employed cluster analysis to identify latent groups using individual students’ noncognitive characteristics. In this case, the authors aimed to group students based on student’s diverse epistemic motivations and beliefs without examining demographics as a factor in the groupings. After forming the clusters, they sampled students from each group to participate in follow-up interviews. This implementation of cluster analysis aligns with the ideals of person-centered approaches by uncovering latent diversity to describe the sample, and with the QuantCrit principle—categories/groups are neither natural nor given. Faber and Benson’s (2017) approach encapsulates the idea of using a clustering algorithm to go beyond immutable characteristics and use qualitative methods to contextualize the sample’s latent diversity.

Incorporating qualitative context enhanced their exploration of nuanced differences within subgroups, leading to a deeper understanding of individual responses. This is often associated with the complementarity rationale for using a mixed methods design—gaining a more complete understanding of a phenomenon (Greene et al. 1989)—but it can also lead to divergence (Pluye et al. 2009). For example, the study by Ko and Leu (2021) involved binning students into predefined categories (pass and fail) and pitting different machine learning techniques against one another to determine which one had the best predictive power; all seven machine learning algorithms were implemented in variable-centric ways. After testing which algorithm has the most superior predictive power, the researchers employed two unsupervised machine learning methods—association analysis and cluster analysis—to determine the common attributes of students in the pass and fail groups. The researchers found 150 rules and six clusters (i.e., four of these clusters belong to the “pass” group and the other two belong to the “fail” group). Although the researchers began with two predefined groups (i.e., pass and fail), they still adopted a person-centered approach of uncovering latent patterns in these two groups by employing association rules analysis and cluster analysis. Such a situation showcases the complexity and coexistence of person- and variable-centeredness in a single research design. However, they did not stop there.

The latter part of the research design showcases elements of person-centeredness much in the way we observed with cluster analysis. In this case, the students in the potentially poor performance groups were interviewed to learn about their lived experiences, which were compared to the states identified by one of the models (i.e., a Hidden Markov Model). Like Faber and Benson’s (2017) research design, this qualitative layer contextualizes the data-driven method using a person-centered approach, conducting a follow-up qualitative analysis to inform the interpretation of the quantitative model results.

Embracing and exploring outliers

When using traditional statistical techniques, the researcher must be mindful of data points too far from the average because such points can have non-trivial impacts on the results. Although some techniques, like cluster analysis, generally do not have many assumptions that must be satisfied before applying them to a dataset, ignoring observations believed to be “outliers” in the sample could still potentially cause issues with the modeling. For example, Nelson et al. (2015) employed cluster analysis to investigate the learning profiles adopted by students in entry-level courses, where self-regulation, motivation and affect, future time perspective, and course affect were used as input variables. During preprocessing, the authors excluded 10 percent of the data points before applying a two-step cluster analysis, for which the first step used sequential clustering and the second adopted agglomerative hierarchical clustering to avoid the potential effects of extreme values.

Typical advice about handling outliers leans toward a process of screening and removal (e.g., Osborne & Overbay 2004); however, other authors have illustrated the statistical consequences of removing outliers, such as André (2022), who argues that such procedures may inflate Type I errors. In the context of Nelson et al. (2015), 10 percent is a high proportion of the dataset to exclude without examining the underlying reasons for the sample distribution—the considerations of which were not provided. Accordingly, simply excluding these data points can potentially create bias and exclude the perspectives from smaller subgroups of participants. This approach of handling outliers by removing individuals from the sample without further examination showcases the variable-centeredness of seeking for homogeneity of the sample and potentially excluding individuals’ voices from small subgroups, which relates to the QuantCrit principle, “numbers are not neutral.”

On the other hand, researchers can embrace outliers in their analyses—which could take several forms. Once again, Faber and Benson (2017) demonstrated one way to identify and handle “outliers” using a person-centered approach: leaning into a contradictory observation relative to the cluster analysis findings. When examining the intra-cluster similarity in one group during the qualitative analysis, a student was found to have no epistemic aim, which contrasted with the other interviewed members of the same cluster. In this case, the outlier was not based on numerical issues stemming from a stray data point; instead, the outlier emerged when qualitative evidence was juxtaposed with quantitative findings. Instead of dismissing this student as an outlier, the researchers carefully examined the student’s responses to understand why the student differed from the rest. This approach manifests the person-centered element of preserving the diversity in individual’s responses (Godwin et al. 2021) and embodies the QuantCrit principle that data cannot speak for themselves; it moves beyond a static description of the cluster’s “average” and explores counteractive variation within algorithm identified group which the members are supposed to share common characteristics. In a mixed methods design, this person-centered approach is particularly robust for bolstering research quality and avoiding de-emphasizing potentially minoritized voices in the sample.

In Faber and Benson’s (2017) case, they engaged in a practice similar to negative case analysis in qualitative research by digging deeper into the individuals’ experience in the subgroup to understand why the divergence emerged. Negative case analysis involves modifying existing hypotheses by examining cases that go against them—actively seeking outliers (Kidder 1981; Newman & Benz 1998)—although this does not preclude authors from leaning into divergence (Creamer & Edwards 2019).

Allowing fluid membership among groups

Not all data-driven algorithms are designed to assign individuals to a single group; some techniques enable researchers to implement less strict membership rules. For example, Scheidt et al. (2021) aimed to understand how engineering students can be grouped by their noncognitive and affective factors, which included the big five personality traits (i.e., neuroticism, extraversion, agreeableness, conscientiousness, and openness), grit, mindset, and mindfulness. Scheidt et al. (2021) recognized that it was dubious all data points could be perfectly assigned into a set number of clusters; moreover, they realized that most deterministic clustering algorithms would force each individual into a single cluster and not allow partial membership—even though some of the students might not be similar to other members in the cluster. To reflect the reality that not every student can fit into a single profile and to avoid the misassignment of students to clusters because all points are given a label, they chose to use a Gaussian Mixture Model (GMM). Unlike conventional clustering algorithms, GMM is a probabilistic clustering method that allows partial membership among clusters and un-clustered individuals.

Choosing such a clustering algorithm not only aligns with the premise of person-centered approaches by recognizing and preserving the heterogeneity in the sample, but it is also congruent with the QuantCrit principle, “categories are neither natural nor given.” Researchers aiming to perform a similar analysis could benefit significantly from probabilistic approaches to finding latent subgroups. Allowing for some uncertainty avoids ascribing too much weight to a given characterization of the individual. Instead, the individual can be seen as exhibiting qualities of different profiles.

Tracking movement between clusters longitudinally

The terminology distinguishing person-centered and variable-centered approaches can be traced back to Block (1971) in a longitudinal study of personality development. Naturally, it is fitting to see how engineering education researchers combine this origin of person-centered approaches with data-driven methods to explore how students transition between latent categories. Cakir and Gheorghe (2017) offer an example in which they adopted a two-step cluster analysis to investigate students’ academic profiles and identify the factors affecting students’ academic performance. For the first step of clustering, a hierarchical clustering algorithm was used to group courses and define the stages based on the course grouping and context, where three stages were identified for technical courses: stage one was composed of introductory coursework such as calculus, physics, and chemistry, stage two included earlier content in industrial engineering, such as statistics and linear algebra, and stage three referred to the junior and senior level coursework. The second step of their clustering process was to use an expectation maximization clustering algorithm to construct three general student academic profiles over four years, representing low, medium, and high performance. After the stages for the courses were defined and three academic profiles were constructed, students’ membership of the academic profile clusters at the different stages was tracked to unfold their academic performance trajectories. This application of cluster analysis provides a novel way of utilizing cluster analysis as a person-centered approach (i.e., personalizing the academic performance trajectory) by not only creating stages and constructing academic profiles but also showing the change in these latent categories over time.

Exploring social dimensions of race as variables

By carefully selecting variables, one could take a person-centered approach to create more authentic groupings that avoid relying on demographic categories as proxies for broader constructs—most notably, racism. For example, Martin et al. (2015) employed cluster analysis to uncover the social capital profile (i.e., the characterization of students’ social networks and their access to social resources, such as resources provided by kin, friends, and education) of engineering students and investigated how race/ethnicity and gender affect students’ social capital profiles. In their preprocessing, they purposefully excluded students’ demographics as commonly used variables to form groups and constructed their social capital profile instead using variables like the size of their social network, the strength of those ties, the heterophily of those connections, and what resources their connections provide them. This implementation of cluster analysis, in combination with the careful handling of race/ethnicity and other dichotomized social notations, such as gender, by Martin et al. (2015), saliently echoed the concern of the first principle of QuantCrit—the centrality of racism. They argued that using cluster analysis to find latent groupings based on students’ social capital characteristics provides researchers with a more comprehensive lens to view complicated social capital problems compared to simply using pre-determined demographics as the sole categories.

Gillborn et al. (2018) underscored the socially constructed nature of race, and it is not often obvious how one might quantify what we discuss as racism or the impact of racism. Other researchers studying racialized and gendered topics can benefit from collecting these kinds of data to better capture the socialized dimensions of each individual’s fixed characteristics.

Exploring the branches of a decision tree for latent groups

Shifting from the myriad examples of cluster analysis, this approach was specific to the data-driven method, decision trees. Decision trees, by design, work by recursively partitioning the input space into regions referred to as “leaves,” which gradually filter the input into a category based on a series of conditional statements (e.g., “mother attended college?” no à “father attended college?” no à “first-generation student.”).

Demographic categories are often used as inputs to inform the classification. For example, Inkelas et al. (2021) employed decision trees to build a model for predicting incoming college students’ choices of their first calculus course. Incoming students’ high school academic achievement, SAT scores, and demographic information such as gender and race/ethnicity were inspected as input features for the classifier. Even though gender and race/ethnicity were excluded from the final decision tree model because of their subpar predictive power, the purpose of building a predictive model using demographic variables showcases the variable-centeredness of adopting the classification tree method in this study. The intention to use predefined and often dichotomized social notations such as gender and race as candidate predictors highlights the concern of two QuantCrit principles, “centrality of racism” and “categories/groups are neither natural nor given.” Using socialized labels like race can lead one down a path of “racial reasoning” (Zuberi 2008, p. 131) and can obscure the causal mechanisms behind the classification (Gillborn et al. 2018).

Similarly, Tan et al. (2021) used a random forest model, an ensemble learning algorithm based on decision trees, to rank the predictors of engineering major choices of students by their predictive power. The predictors they examined included students’ demographic categories, high school characteristics and experience, and standardized test scores. Students’ gender was fed into the classifier and was recognized by the authors as the most important predictor of student engineering major choice. Tan et al. (2021) wrote that there is “critical predictive power in forecasting students declaring an engineering major” (p. 581) and that “removing the gender variable from the pool of explanatory variables means a 28% (Panel a [of variable importance plot]) and 19% (Panel b [of variable importance plot]) loss the total prediction accuracy” (p. 582). The authors recognize that the random forest method is a black box because of its opaqueness regarding interpretability, which, when combined with socialized identities, can muddy the understanding of precisely what is driving different phenomena, such as engineering major choice. Therefore, to avoid potential discrimination and increase the interpretability of the results, it is essential for researchers who adopt such “black box”-like machine learning algorithms to contextualize, investigate, and justify the selection of variables fed into the model, especially when they are related to individual’s social identities or demographics.

However, person-centered applications of decision trees were present in the sampled articles, such as using the branches of the decision tree as a basis for forming latent groups. In the previous two examples (Inkelas et al. 2021; Tan et al. 2021), decision trees were implemented in a standard variable-centered way, but Singer et al. (2020) demonstrated how to deploy a decision tree classifier using both person- and variable-centered approaches. Singer et al. (2020) used decision trees to investigate the predictive power of certain variables, such as age, gender, and course grades, to predict the academic stability of students with learning disabilities, as measured by the standard deviation in weighted average grades over eight terms; in particular, the researchers wanted to explore if including learning disabilities and accommodations as input variables can significantly affect such a model. Nonetheless, another reason drove the researchers to adopt the decision tree method as the classification method compared to other standard methods, such as neural networks. Unlike random forest models, the mechanism by which a single decision tree filters the data into categories is much more clearly laid out in its branches, and these branches can be used to define latent groupings based on subsets of splits in categories. The researchers contended that finding these subgroups of students can help instructors or professionals provide more effective accommodations based on students’ identified profiles. Thus, even though the primary goal of this study carries a classic variable-centeredness, the researchers implemented the decision tree method for a person-centered purpose—identifying latent groupings—and this implementation embodies the QuantCrit principle “categories/groups are not natural nor given.”

Similar to Singer et al.’s (2020) study, Kaleita et al. (2016) used a decision tree to identify a set of variables that can be used to predict if students will be “at-risk” (i.e., GPA <2.0) before they start their first semester in college, such as ACT scores, demographic categories, and major certainty. The purpose of selecting predictors to build a model showcases the variable-centeredness of adopting decision tree methods—after all, decision trees are techniques designed to bin students into pre-labeled groups. However, Kaleita et al. (2016) argued that there might have been a situation in which a subgroup of students could be impacted heavily by some factors, but these factors do not affect others to the same extent. This consideration recognizes the latent diversity that could exist in the sample and led the authors to uncover these groupings to personalize the interventions for potential at-risk students.

Cluster analysis was the most popular method and most likely to be used in diverse person-centered applications

Cluster analysis was employed in more than half the reviewed articles (n = 13), which demonstrates its popularity as a data-driven method in mainstream engineering education research. Lund and Ma (2021) describe cluster analysis as a data mining algorithm usually employed to uncover latent patterns and group observations in a sample based on the input features; accordingly, it is well-suited for a person-centered approach. There are several different methodologies for clustering data, which include centroid/partition-based (e.g., k-means), connectivity-based (e.g., hierarchical), density-based (e.g., DBSCAN), and model-based (e.g., Gaussian Mixture Models), and fuzzy-based (e.g., c-means) (Tan et al. 2005). Although there is considerable diversity in the approaches researchers can take to group observations together, the centroid-based algorithm called k-means has grown in popularity in practice (Ashabi et al. 2020). Table 3 tabulates the different clustering algorithms used by researchers as the main quantitative method in their research; as noted in the table, the variety of algorithms used demonstrates the engineering education research community has experimented well beyond the popular k-means. Moreover, the diversity in the algorithms is rivaled by the multiple ways cluster analysis was used in a person-centered fashion by leveraging its flexibility in forming different latent subgroups in a sample.

Principal component analysis does not seem compatible with person-centered approaches

PCA, a commonly employed technique for dimension reduction, seeks to identify a set of composite variables that can effectively represent a dataset in a lower dimension while preserving essential information (Kherif & Latypova 2020). We found that authors typically employed principal component analysis with a variable-centric perspective. This is consistent with Godwin et al.’s (2021) initial assessment of PCA as a technique most suited to variable-centered aims. In our sample, PCA was only used twice as the primary technique in the research design but appeared frequently alongside other data-driven techniques. Within these applications, using PCA did not seem compatible with a person-centered approach.

To elaborate on why, consider the process of choosing variables. Besides deciding whom to include and exclude, what to measure is another common analytical decision the researchers need to make before implementing data-driven techniques. Such variable selection processes can be performed using a variable-centered approach in cases where the dimensionality of the data is too large. Because PCA is a dimensionality reduction technique, the studies in the sample would use it to preprocess data to reduce the number of variables at play. For example, Martin and Sorhaindo (2019) examined intrinsic and extrinsic motivational factors as predictors of academic achievement in civil engineering students. They used PCA to consolidate the motivational factors into distinct intrinsic and extrinsic groups. The initial set of 22 motivation variables was reduced to five principal components, explaining 66% of the variance. This application of PCA to combine variables highlights aspects of variable-centeredness, which embodies the QuantCrit principle, “numbers are not neutral.” The researchers replaced the original set of variables with the derived principal components—even though a reasonably large variation loss (i.e., 34%) was recognized. After conducting the PCA, the researchers probed for differences between groups defined by their demographic characteristics, such as gender (male vs. female), national origin (native vs. international), and age.

There are two points to raise here. First, considerable diversity is washed out during aggregation and can bring about misleading conclusions if taken at face value (Shafer et al. 2021). By combining the subgroups comprising the native and international, an aggregation fallacy occurs similar to the underrepresented and minority category often seen in studies with low n’s in populations with racial identities like Black, Latine, and Native American (Xu et al. 2023). These explorations can work against the QuantCrit principle of “categories/groups are neither natural nor given,” particularly in the case of national origin, a dichotomized and highly aggregated category in this study.

Second, principal components are not necessarily informed by theory nor guaranteed to be interpretable and theoretically meaningful. By comparing demographic groups to one another using these uniquely constructed variables, the interpretations of statistical tests can be misused or lead future work down fruitless paths because of spurious correlations in variables from a foundational study. Pairing principal component analysis with a person-centered approach generally does not offset these issues. For example, Gallego et al. (2016) investigated how workload affects students’ academic performance by adopting cluster analysis to group students with similar characteristics by examining the efficiency rate (ER, which is the number of passing students/the number of enrolled students), success rate (SR which is the number of passing students/the number of attending students), drop-out rate (DR which is the number of non-attending students/the number of enrolled students), time (student’s rating of time needed to pass the course), and difficulty (the student’s perceived level of difficulty of a subject).

Before applying cluster analysis, they used principal component analysis (PCA) on the entire set of candidate variables, then chose the two resulting principal components as the aggregated variables to be used to form clusters. These principal components accounted for 70% of the total variance in the sample. Although PCA can aid the researcher by reducing the total number of variables to a more workable number that can be visualized or better aligned with a desirable variable-to-case ratio, the resulting principal components are not guaranteed to be interpretable or theoretically meaningful. In this case, the first principal component was primarily a combination of three variables: ER, SR, and Difficulty. The researchers proposed that this principal component could be interpreted as the “number of students passing the subject” (Gallego et al. 2016, p. 9) because ER and SR are related to the measures of academic performance. However, the other highly associated variable, “Difficulty,” was not clearly addressed, which muddies the interpretation of the composite variable.

Without a clear operationalization, the resulting clusters are likely to be less meaningful, and drawing conclusions from these principal components for classroom interventions can lead the researchers down unproductive paths. The approach to preprocessing, that is, employing PCA to form composite variables, aligns with a variable-centered approach—considering this step explicitly aims to find homogeneity among variables to reduce the data’s dimensionality. Moreover, using principal components without thoroughly interrogating their meaning works against the QuantCrit principle, “numbers are not neutral.” These composite variables are highly contextualized with respect to the data—meaning the actual composite variables from dataset to dataset with the same variables are not guaranteed to be the same. The potential for spurious correlations in high-dimensional datasets is not trivial, so further work based on these principal components could lead to misuse if care is not taken when operationalizing such variables.

(Data-driven) methods are not inherently person-centered or variable-centered

At this point, we contrasted cluster analysis and principal component analysis in terms of person-centered and variable-centered approaches. Although it might be tempting to deem a particular method person-centered or variable-centered, it became clear throughout this review that this thinking can be limiting. For example, cluster analysis is perhaps the most likely technique for researchers to use in a person-centered fashion. In fact, Godwin et al. (2021) associate cluster analysis’s ability to identify latent data structures with person-centered analyses. This aligns with the premise of using a latent diversity perspective when conducting a quantitative analysis to understand relationships among individuals in a sample beyond demographic categories (see Godwin 2017). Although the goal of cluster analysis can be stated simply, the way the technique was used within our sample is a testament to the myriad decisions involved with its implementation.

Although cluster analysis is an inductive technique by design, in the sense that the groupings do not necessarily need to be known as a priori, this does not preclude the researcher from applying the technique deductively in a variable-centric fashion. One example in the sample of such an approach was Reid et al. (2016). In their study, cluster analysis was utilized as a method to justify the stability and repeatability of a psychometric instrument (i.e., Student Attitudinal Success Instrument or SASI-I), which is constructed to gather incoming engineering students’ affective characteristics, such as intrinsic motivation, academic self-efficacy, and leadership. Using the factor analysis results of the SASI-I, the same number of clusters for each cohort from three years (i.e., student cohorts admitted in three different years) were generated, and the homogeneity coefficients for these clusters were found to be high between corresponding clusters over those years. These results were used as evidence for the stability of their instrument. Although subgroups of students were identified, as typical with applications of cluster analysis in these circumstances, the number of the clusters and the similarity of those clusters over years was used to support claims of validity evidence rather than describe the characteristics of the identified subgroups.

Similarly, Quan et al. (2019) selected five representative indicators and conducted a cluster analysis using k-means. They found that the two resulting clusters effectively grouped “at-risk” students (i.e., students who failed at least one course in a single semester), wherein one cluster had a high proportion of “at-risk” students, and the other had a high proportion of “not-at-risk” students. Thus, the cluster analysis result demonstrated the selected indicators’ effectiveness – the two clusters are not only significantly different regarding selected features but also well separate at-risk students and not-at-risk students. Like Reid et al.’s (2016) use of cluster analysis to verify the stability of their instrument, this focus on using groupings of students as the medium to justify an instrument’s usability is inherent to a variable-centric approach.

The other technique that seemed to be an entirely variable-centered method was the decision tree (and its associated approaches, like random forest). Godwin et al. (2021) binned decision trees squarely in the variable-centered category, explaining that these data-driven methods are focused on “reconcil[ing] variables with predefined (and thus potentially biased or inaccurate) categories” (p. 21). There is little to argue about this point because a decision tree is not necessarily designed to assign inputs to categories that are not explicitly used when developing the tree. Moreover, if decision trees are used to filter individuals into categories without careful consideration of the implications of those classifications, such as classifying students as at-risk using demographics among other variables (c.f., Brown 2016), there could be a potential for misuse. However, like how cluster analysis could be leveraged in a variable-centric fashion, examples of how researchers extracted some person-centered applications of decision trees in their studies were present in the sample, such as examining specific branches in the tree and treating the intermediate splits as latent groups. This thinking can generally be applied across quantitative techniques.

Person-centered approaches cannot compensate for poor data collection or operationalization

Although we have discussed person-centered approaches in a positive light so far in this paper, it should be stressed that they are not a panacea for methodological issues in engineering education research. In particular, throughout the papers we discussed, and what is embedded in the usual premise of person-centered approaches, much of the focus is on using psychological measures (e.g., motivation, personality, epistemic beliefs) to group observations and draw comparisons of the latent diversity within the sample. For person-centered approaches to produce legitimate results at a fundamental level, the psychological measures under study must be appropriate in terms of their validity and reliability—or else the result will be a situation of garbage in, garbage out. More broadly, quantification is a key concern of QuantCrit throughout its principles. QuantCrit challenges the idea that concepts like racism are readily quantifiable and that numbers are inherently neutral (Gillborn et al. 2018). Using person-centered approaches does not address issues related to bias in data collection or operationalization of variables, and engineering education researchers should not see person-centered approaches as a fix for such issues.

As suggested by Castillo and Gillborn (2023), researchers should think critically about the metrics they use in their studies by reflecting on the origin of different measurements. Many psychological instruments have only been tested in specific contexts and are laden with cultural and socioeconomic considerations; moreover, the psychological construct may have been operationalized in a way that is not congruent with the researcher’s context. After all, validity is a continuous process, and selecting an instrument from a publication does not imply it will work elsewhere (Douglas & Purzer, 2015). These points underscore that much of the success of a person-centered approach is reliant on the researcher engaging critically with design decisions at the onset of the study. Although bias is never truly eliminated from the analyses, pinpointing sources of bias in their data collection and operationalizations initially better position engineering education researchers to frame the limitations of their work when the time comes to report on their findings.

Future work

One direction for future work can be increasing the timespan of consideration and publication venue, which has the potential to enrich the findings demonstrated in this paper and achieve a more comprehensive and overarching SLR. One large database of publications is the paper repository hosted by the American Society for Engineering Education, which contains conference proceedings for its annual conference and affiliated conferences like the First Year Engineering Experience conference series. Another popular conference that would be useful to review papers from is the Frontiers in Education conference hosted by the Institute for Electrical and Electronic Engineers (IEEE). More data-driven methods, diverse applications, and evolutions of such approaches might emerge.

Another direction for future research can involve focusing on applying a specific data-driven method, such as cluster analysis or decision trees, in engineering education research. We have showcased how versatile some data-driven methods can be when they are applied in different ways—person-centered, variable-centered, or both—according to different research purposes and analytical decisions made by researchers. Thus, such method-specific research is warranted, which can provide a more in-depth understanding of the current state of specific data-driven methods in engineering education research and offer researchers informative exemplars on when and how to apply the data-driven method creatively and effectively.

Finally, although not the focus of this paper, another line of inquiry that can sprout from this work is examining what variables researchers are using to conduct person-centered approaches. As we noted in our discussion, many of the variables considered were psychological measurements. The degree to which these measurements and the conclusions drawn from them are interrogated for validity and reliability—including how the authors reflect with criticality through a QuantCrit lens—can be performed in a deeper analysis of engineering education publications.

Limitations

As mentioned in the “Future work” section of the Conclusion, the main limitation of the current work is the timespan of the reviewed literature. This SLR only considered publications for empirical, human subjects research from 2011 to 2021 in four leading engineering education journals—which yielded a specific subset of data-driven method applications. If any of the constraints were relaxed on the inclusion criteria, the results could likely appear considerably different. Thus, this work only depicts a detailed picture of data-driven methods through the lens of variable- and person-centeredness with a critical eye on a small portion of existing literature. The results should not be taken as a comprehensive review of how data-driven methods are applied in engineering education for broader settings (e.g., research on non-human subjects and looking back from 2010).

A second limitation is our scope in the search string. When forming our search string, we included typical keywords like “machine learning” and “data mining” among more specific techniques commonly associated with data-driven methods. Considering the search string was informed by mainstream data-driven methods, it likely does not capture the full range of niche techniques that might have been employed in the sampled journals. Therefore, a selection of papers could have been overlooked. Moreover, depending on how one defines a data-driven or machine learning method, the boundaries of what would ultimately be included in the review would be reshaped. Thus, the results should not be interpreted as being a complete treatment of all possible data-driven methods.

It is anticipated that this work can encourage more comprehensive studies about the deployment of data-driven methods in engineering education research. Future work can provide researchers with different perspectives to assist them with finding novel ways to balance the benefits and biases brought when adopting powerful data-driven methods in their research.

Final thoughts

As described in the delineation between person-centered and variable-centered approaches, these must coexist and not be viewed as a dichotomy. In the particular case of data-driven approaches, which are aligned with research goals that are often person-centric in nature, we have seen several tangible examples that engineering education researchers can use to provide a balance in their design that can zoom both in and out on the latent characteristics of their sample. In fact, several of them were rooted in situating the data-driven methods in a mixed methods design, which intersected the technique with qualitative approaches to contextualize the findings. However, as we demonstrated, researchers need not always use a mixed methods design to humanize their approach. In particular, by embracing a latent diversity perspective in their research design (Godwin 2017), researchers can avoid an essentialized perspective on the role of race (and other sensitive constructs) in their work. We anticipate spotlighting these strategies will provide engineering education researchers with an expanded, practical toolkit for engaging in a QuantCrit mindset while handling tools that can cause considerable harm if misused.