Interrogating the Use of Large Language Models in Qualitative Research Using the Qualifying Qualitative Research Quality Framework

The Trajectory of Large Language Models

The current explosion of tools using artificial intelligence is no accident, nor is it particularly new. With the introduction of ChatGPT in November 2022, generative AI has been integrated into products, processes, and everyday life, amounting to unprecedented user growth (Hu 2023). LLMs are one subset of what we currently call generative AI; at their core, they are models commonly found in machine learning applications called neural networks. Inspired by how neurons function in the human brain, these networks model the action of a neuron firing with an activation function, traditionally a sigmoid curve. By connecting these neurons into layers, linking the input (what we want to classify) to the output (the resulting classification), the chain of firing neurons leads the network toward the most appropriate classification of whatever input it was given. Moreover, by adding weights to the connections between the artificial neurons and training the network using a large set of examples, certain pathways can be strengthened to more readily associate a particular set of inputs with an output, much like the process of myelination in the human brain. There are many types of neural networks, but the type most readily associated with modeling language is the recurrent neural network (RNN) (Rumelhart et al. 1986).

Algorithms like autocomplete—a common comparison for generative AI tools—process sequential data, such as a set of words and symbols, where order matters. With language, order matters, but context also matters. Consequently, the autocomplete model needs sequential and contextual information to appropriately predict the next word in the sequence. RNNs can process inputs of any length, but they are slow (Hernández & Amigó 2020) and can be problematic to train (Luitse & Denkena 2021). Long short-term memory (LSTM) networks improved upon the standard RNN by adding a mechanism for short-term memory and the ability to determine when to remember certain information and when to forget (Hochreiter & Schmidhuber 1997). Before 2017, these models were generally standard approaches for modeling text-based data.

In 2017, the landscape of language modeling changed with the introduction of a new mechanism that improved LSTM networks, the transformer. In fact, ChatGPT exists due to the invention of the Generative Pre-trained Transformer (the “GPT”). These GPT models have pioneered a specific mathematical object used to make the model function called attention, which allows the LLM to pick out parts of its input that are more important than others when predicting output, i.e., words in a sentence (Vaswani et al. 2017). OpenAI’s first model (GPT-1) was released in 2018, and each subsequent year saw the release of the next iteration of the GPT series of models (Luitse & Denkena 2021). When we started writing, the state-of-the-art model was GPT-4, released in March 2023, a mere five months after ChatGPT (Dastin 2023), which newer models have since overtaken. GPT models will likely continue to improve as models are trained with more quality training data and computational effort. However, Sam Altman, the CEO of ChatGPT’s creator, OpenAI, has commented, “I think we’re at the end of the era where it’s going to be these, like, giant, giant models…we’ll make them better in other ways” (Knight 2023, p. 1). The future of these models is fuzzy, and updates will possibly come with unexpected developments in model architecture, training techniques, or other foundational insights into language modeling tasks. For example, a consumer-facing reasoning model did not exist when we started writing but quickly became a resource for high-level cognitive tasks, including problem-solving (OpenAI 2024b).

Considering LLMs have been proposed as tools to conduct many core tasks associated with engineering education research with varying levels of competence, including developing research plans, creating data collection instruments, analyzing data, and writing manuscripts (Biswas 2023; Lehr et al. 2024; Rahman et al. 2023), the community needs to examine how we interact with these tools to appreciate what they can do and maintain enduring skepticism for what they seem to do well and cannot do now.

Quality in Interpretive Engineering Education Research

Although scholars have written on the merits and pitfalls of generative AI models like LLMs (Baidoo-Anu & Ansah 2023; Bender et al. 2021; Johri et al. 2023; Sok & Heng 2023), we focus on issues that arise when using LLMs in qualitative engineering education research, drawing on Walther et al.’s (2013) highly cited Q3 framework on quality in interpretive engineering education research. The authors present their framework as a means to “explore and make explicit [their] methodological choices and underlying assumptions relating to research quality” (Walther et al. 2013, p. 627). This process-oriented framework offers a way to discuss issues arising from using a tool that lacks transparency in sensitive human subjects research. The framework is rooted in an understanding of interpretive research, which refers to knowledge produced from interpreting the lived experiences of individuals and groups. These interpretations are carried out by researchers with particular philosophical worldviews and contextual positions in society that call for researcher reflexivity in their interpretations, an active component of developing and communicating positionality.

The Q3 framework consists of two models: 1) a process model that discusses quality at every research stage and 2) a typology of quality considerations that contextualizes the process model. We use the latter to structure our examination of LLMs in engineering education research. The typology consists of six processes:

1. Theoretical validation focuses on the fit between the social reality under investigation and the theory produced.

2. Procedural validation suggests incorporating features into the research design to improve the fit between the interpretations generated and the social reality.

3. Communicative validation accounts for the quality of the co-construction of knowledge in the social context under investigation and within the research community.

4. Pragmatic validation examines the extent to which theories and concepts are compatible with empirical reality.

5. Process reliability provides the necessary conditions for developing overall validation through strategies aimed at making the research process “as independent from random influences as possible” (Walther et al. 2013, p. 641).

6. Ethical validation was added later to the framework and concerns how integrity and responsibility are considered throughout the design (Sochacka et al. 2018; Walther et al. 2015).

Alongside this framework, we also address the explicit influence of the researcher in interpretive qualitative research, which is encapsulated in the concept of positionality. Recent increased interest in positionality (Hampton et al. 2021; Hampton & Reeping 2019; Martin et al. 2022; Secules et al. 2021) has foregrounded considerations of how researchers influence the research process—considerations that are also core to the Q3 framework, particularly in procedural validation.

Positionality

Positionality refers to how researchers are situated relative to a study’s social and political context, including critical self-reflection that Milner contends should encompass “racially and culturally grounded questions” (2007 p. 395). This urge by Milner echoes in Rowe (2014), who elaborated that “the position adopted by a researcher affects every phase of the research process, from the way the question or problem is initially constructed, designed, and conducted to how others are invited to participate, the ways in which knowledge is constructed and acted on and, finally, the ways in which outcomes are disseminated and published” (2014 p. 2). Engineering education research occupies a conceptual space influenced by its institutions’ organizational makeup, individual engineering experiences, and collective systems within society. Our practice at its base is heavily shaped by the Global North traditions of engineering acculturation (Mejia et al. 2022), with transitions over time influenced by economic, political, and societal lenses (Froyd et al. 2012). Those who have and are making careers in this space have potentially entered for various reasons, from commitments to broadening participation to interests in how students learn. These multiple points of entry and interests, as well as associated funding support, can influence a researcher’s proximity to their research subjects.

While there have been increased calls for positionality and reflexive processes for researcher transparency through journal requirements for positionality statements, the concept has also received pushback for potentially contributing to a lack of objectivity. Savolainen et al. (2023), for example, citing the principle of universalism, claim “that scholarship should be judged according to its own merits independent of the status or identity of the person responsible for the contribution” (p. 1336). They further argue that “positionality statements are misguided because the advancement of scholarship…does not depend on the biographical characteristics of individual participants” (Savolainen et al. 2023, p. 1335). However, positionality helps connect the researcher to the research and should not simply be a biographical statement nor focus exclusively on conflicts of interest. The researcher is an instrument in the process, especially in interpretive research; as such, that fact is methodologically salient. Contextualizing projects via positionality enriches the research instead of perpetuating an illusion of objectivity by not just describing who was involved but also what measures were used to limit bias and play into strengths. In that vein, Secules et al. (2021 p. 25) offer guiding questions that ask researchers to consider how their positionality impacts:

• What research they choose to do?

• How they know what they know?

• What they can observe as researchers?

• How they make methodological choices?

• How they relate to research participants?

• How they represent themselves in writing and other communication?

To unpack these questions, we draw on the praxis of positionality to outline the limits of using these tools in different facets of engineering education research. Through this praxis, authors can engage in ongoing processes of reflexivity through research design, implementation, and translation.

Positionalities of the Authors

Our positionalities have been detailed in our previous commentary on positionality (Hampton et al. 2021), so here we briefly describe aspects salient to this work. We are a team of engineering education researchers with a shared commitment to transparency and authenticity in research spanning quantitative and qualitative research methodologies. With the emergence of ChatGPT and the avalanche of preprints applying the tool with little regard for its intended functionality and broad proclamations about its impacts across unforeseen domains (including firsthand experience reviewing papers and paper reviews clearly generated by an LLM), the lead author—who has previous experience in the machine learning techniques undergirding ChatGPT and focuses on quantitative and mixed methodologies—was motivated to reconvene this author team to explore what qualitative research quality could look like with a balanced perspective that is neither full of adulation for large language models nor replete with Luddite critiques. We do not consider ourselves heavy users of LLMs in our professional work and only use them in lower-risk situations where accuracy is not of concern.

In previous efforts, the second and third authors experience with critical perspectives in qualitative research aligned with the first author’s views, creating a synergy that we aimed to replicate here to address how engineering education researchers might co-create knowledge with the LLMs, guided by the Q3 Framework (Walther et al. 2013; Walther & Sochacka 2014).

Handling Data

Instead of starting with making data, we begin with more tangible use cases for handling data.

One obvious use case during analysis is using an LLM to inductively or deductively code textual data. In fact, LLMs have been described as a tool that “contributes to improved efficiency and unbiased data processing” (Jalali & Akhavan 2024, p. 1). Supplementing qualitative data analysis with an LLM has already been seen in engineering education. For example, Katz et al. (2023) used GPT-4 to deductively code peer evaluations for positive and negative comments about teamwork behaviors. Moreover, Katz et al. (2024) proposed a workflow for thematic analysis with the assistance of generative AI called Generative AI-enabled Theme Organization and Structuring (GATOS). As these new analytical frameworks become embedded in engineering education research, we suspect that such practices will lie on a spectrum representing the researcher’s relative distance to the data:

• At the most extreme end, researchers could use generative AI alone to code data automatically inductively or deductively, without human coding, as in GATOS. To verify themes, researchers could use multiple iterations or different models as a form of cross-validation.

• Researchers could become more involved by still leaning most heavily on the LLM but selecting a random subset for human coding to verify themes.

• With even more researcher involvement, the LLM becomes more of an assistant than a replacement for the researcher in the analysis. For example, the LLM could help process the initial codes extracted by researchers, or, alternately, the LLM could generate initial codes that the researcher could build on to develop broader categories and themes.

Making Data

With the conversational nature of LLM-powered tools, one potential application is supporting or replacing the interviewer in data collection (Chopra & Haaland 2023) with a chatbot that uses an LLM to facilitate the conversation. Here, too, we see a spectrum:

• In the most extreme case, researchers could have little or no interaction with participants and instead deploy a chatbot interviewer pre-calibrated with the interview protocol that autonomously collects participant data using either text or voice inputs and outputs. There is already experimentation in this area; for example, Cuervas et al. (2023) describe the development of a chatbot that follows pre-written questions before generating follow-ups based on participants’ responses, with member checking included to summarize interviewee responses dynamically.

• A more researcher-driven application could use an LLM to develop potential interview questions as a stepping stone for creating more focused prompts. In this case, the chatbot could also serve as a simulated participant to test the efficacy of the interview prompts (Strobel et al. 2024), providing potential responses to various interview questions. This application mirrors cognitive interviews used to understand how participants perceive or comprehend some stimulus, such as a draft questionnaire (Singleton & Straits, 2017).

• In a similar vein, an LLM could help train interviewers by simulating potential interviewee responses either in advance or in real-time (Laiq & Dieste 2020). Here, the focus is on helping the interviewer formulate follow-up questions in real-time.

Unlike deploying automated interviewers, these latter two use cases preserve the human as the interviewer and use LLMs as a means to improve the instruments. There are also theoretically middle-ground scenarios where the LLM is used to help researchers formulate questions during an interview; the interviewer would be present and control the interview flow, with the LLM acting as an assistant to pick follow-up questions. However, no literature could be found in this area. Instead, the applications seem to straddle the two extremes noted in this section.

Ethical validation

Broadly, ethical validation is concerned with “integrity and responsibility throughout the research process” (Walther et al. 2017, p. 401). The key question we unpack here is, “How do we maintain our values, intentions, and commitments throughout the inquiry” (Sochacka et al. 2018, p. 375)? Specifically, when employing an LLM to assist with analysis when handling data, we need to ask what our motivations are and why we want to use an LLM, metaphorically asking whether an LLM “should be included on the research team” (Sochacka et al. 2018, p. 375). The allure of using these models in qualitative research is the chance to expedite coding. Given the speed at which LLMs process textual data, it is unsurprising the extent of the work already sunk into exploring how to use these systems to perform laborious coding tasks such as thematic analysis (Gamieldien et al. 2023; Katz, Fleming, et al. 2024; Prescott et al. 2024).

But in this use case, suppose we upload interview transcripts to an LLM for qualitative analysis. Immediately, we face potential issues with ethical validation. First, while Sochacka et al. (2018) situate guiding questions related to ethical validation as going beyond standard IRB requirements, we cannot ignore issues with uncritical uses of LLMs in this use case that can directly conflict with the federal and institutional protections assured to participants through standard consenting processes. For this discussion, we use the term commercial LLM to refer to an LLM hosted somewhere other than a researcher’s local computer or that one interacts with using a web interface (e.g., ChatGPT and Gemini) or through an API not controlled by the researcher’s institution. In contrast, we use the term non-commercial LLM for those that run locally on an individual computer and are often open-source (i.e., so the data is not processed on an external server) or that operate through institutionally purchased enterprise licenses that, in principle, do not store the inputs and use them for training future models. A prominent non-commercial LLM is Meta’s open-source Llama, for which they post resources to enable users to run the model on their own computers (Meta 2024). At this stage, the distribution of specific tool usage among researchers is unknown, with even less information for the subset of engineering education researchers. Researchers without university-supported enterprise solutions are likely using free or paid models via a web browser (e.g., ChatGPT Plus). With commercial models, the data entered could be used to improve the current model or train a more advanced model unless the user explicitly opts out (OpenAI 2024a). But even in enterprise solutions, data that was likely promised to be stored in a secure location may be sent to a server not owned by the researcher or institution.

As Davison et al. (2024) note, researchers working with human subject data need to recognize that the act of entering participant data into a commercial LLM is a data transfer in and of itself, and the researchers should obtain consent for this process. IRB guidance in this area remains limited, particularly in helping to establish where procedures become non-compliant, beyond advice to “play it safe” and always request a waiver or approval from the IRB (Teixeira da Silva 2023, p. 1). Without participants’ consent, researchers using commercial models to handle data risk issues with ethical validation. For example, suppose the approved IRB protocol states that data will be stored in secure cloud-based storage (e.g., Dropbox, OneDrive, Google Drive). In that case, uploading the data to a commercial tool like ChatGPT or Gemini violates the data management plan. Moreover, if participants were not informed that their data would be entered into an LLM, it could be considered a breach of trust.

Beyond IRB procedures, the extent to which the LLM—commercial or not—is used has implications for how we construct meaning from the data in this use case. In our spectrum of researcher involvement, ethical validation can be at further risk when the LLM performs most of the analysis. For example, if the research topic is sensitive and the data concern highly detailed accounts of participants’ personal and potentially vulnerable experiences (e.g., from a narrative interview), offloading all or most of the coding to a model can be seen as not “do[ing] justice to the lived experiences of those who participated in the research” (Sochacka et al. 2018, p. 375). This incongruence results primarily from the interpretive distance between the researcher and the data, which is exacerbated if an LLM was used as an agent to make the data as well (e.g., an interview chatbot acting without researcher monitoring). Engineering education researchers must be transparent about using LLMs in their design with their institution’s IRB and their participants to support claims of ethical validation.

Finally, issues beyond the design can also bear on “how… we maintain our values, intentions, and commitments throughout the inquiry” (Sochacka et al. 2018, p. 375). For example, there are non-trivial environmental impacts to training and using commercial LLMs that pose a major risk to their constantly expanding capacity (Bender et al. 2021). De Vries (2023), an energy scientist, estimated a middle scenario of AI server growth by 2027 as consuming between 85 and 135 terawatt-hours (TWh) per year—a range similar to the electricity use of Argentina, the Netherlands, and Sweden. Although ethical validation is more concerned with research design, we cannot exclude the environmental impact of these increasingly large language models or the low-paid human labor put into reviewing data for harmful or offensive content and labeling thousands of pictures or texts one-by-one (see Dzieza 2023). By reflecting on the environmental and human costs of building, expanding, and using LLMs, we, as engineering education researchers, can better examine the ethical decisions and sociotechnical contexts of producing research and our associated contributions when we lean heavily on LLM-based tools.

Theoretical validation

Theoretical validation focuses on the ontological nature of the social reality examined and its fit with the theory or findings produced by the researchers. All validation types feed into theoretical validation in some way, where each improvement impacts the design’s overall quality. When making data, the Q3 considerations center on the sampling procedures (e.g., purposive sampling, stratified sampling) to capture the necessary breadth of perspectives to form an appropriate theoretical representation of the social reality. When handling data, the focus shifts to ensuring the coherence and sufficient complexity of the generated theory. As Walther et al. (2013) note, “this concept of theoretical validation through the exploration of coherence and complexity is reflected in the formulation of systematic ways to inductively generate theory from the examination of (relatively few) individual cases” (p. 462). Across the research design, theoretical validation concerns the completeness of the theory generated and how elements of the making and handling processes limit this completeness. Thus, we ask here how completeness in sampling and interpretation can be appropriately maintained when employing LLMs.

We posit that the considerations of theoretical validation will usually be most salient for handling data, based on the questions posed by Walther et al. (2013), and thus, we turn to the use case in which the LLM is used in coding. The primary concern of theoretical validation is the ontology embedded within the LLM based on the data used for training and how that ontology informs how LLMs generate their insights. Theoretical validation asks us to consider two questions: “How will we be able to see/what could prevent us from seeing the full extent of [the] social reality?” and “How do [we] know that the findings make a meaningful contribution to the relevant body of theory?” (Walther et al. 2017, p. 401).

Unfortunately, LLMs do not intrinsically understand what is meaningful, nor do they have the contextual justifications necessary to evaluate appropriate theoretical depth. As Davidson et al. (2024) note, automated coding tools “can only examine syntax, but cannot genuinely grasp data’s semantic and pragmatic aspects” (p. 9). This limitation is encapsulated well in Curry et al.’s (2024) exploration into how LLMs (GPT-4 in their case) could be used to conduct discourse analysis; they found that the model performed well in grouping keywords, but “the categories [it produces] are inevitably surface-level and generic, reflecting parallels with semantic taggers and lacking the granularity required in research on specialised discourses” (p. 7). Goyanes et al. (2024) came to a similar conclusion, remarking that tools like ChatGPT can be effective in preliminary explorations of data, but “the quality and granularity of thematic patterns is rather unsatisfactory when it comes to the subtle nuances and contextual insights generally associated with qualitative research” (p. 25). Even if researchers have sampled in ways that help generate an appropriate theory, we are still bound by the analytic depth of LLMs, and generating rich enough findings from the model itself seems debatable at this point.

There are numerous ways to tweak LLMs in handling data, including the prompt provided, the iterative refinement of the prompt to yield a satisfactory output, and subsequent steps to engage the tool with its own output by leveraging the conversational nature of interfaces. However, using an LLM to assist in the research design without contextual expertise can lead to misalignment between the results and the underlying social reality. An LLM’s social reality is empirically limited to the data used to train it, which is a barrier to both making data (i.e., using the model to train interviewers or generate potential questions) and handling data (i.e., automated thematic coding). Consequently, the theories or insights generated are rooted solely in what exists in the corpus of digital data. For example, if most of the data the LLM can use is produced by countries in English in the global north, then the social reality under investigation is already situated in a particular context that may or may not match the social realities situated in other countries or demographic contexts. An LLM’s ontology of the field generated from dominant narratives on the internet may thus need intentional expertise to guide and scope both code generation and theme development. Without the researcher, the LLM has no context regarding how the data were made, which threatens theoretical validation because it can “prevent us from seeing the full extent of [the] social reality” (Walther et al. 2017, p. 401).

The source of the training data is thus critical, as it can uphold dominant narratives through embedded biases related to different demographic and cultural groups (Bender et al. 2021; Sheng et al. 2019). Within engineering education, there has already been a tangible tension in misrepresenting student experiences, for instance, related to deficit-based heuristics (Mejia et al. 2018) and beliefs perpetuated on ability. Therefore, when prompting an LLM to act like a research participant or analyze a participant’s narrative transcripts, its interpretations are laden with bias that may not be readily discernible. We must be cautious not to allow LLMs to automate these misrepresentations.

Not all topics bear the same severity in terms of theoretical validation issues, but researchers’ contextual knowledge must supplement the model’s limited perspective. An LLM is trained using a vast corpus of diverse text data, so it tends to be able to assist with coding in areas that are well-covered in the literature, such as teamwork behaviors, career interests, and perceptions of teaching (Katz, Gerhardt et al. 2024; Katz, Shakir et al. 2023; Katz, Wei et al. 2023), which is good news for researchers in some areas. If a researcher wants to use an LLM to generate potential interview questions or synthetic responses as a form of interviewee training, then the model can likely help. However, when greater complexity is desired and the topics become more sensitive or personal (e.g., racial or gender-based discrimination in engineering classrooms), the researcher’s contextual knowledge is needed to balance lackluster or incorrect outputs. Azaria et al. (2023), echo the necessary consideration of expertise in their preprint on the field usage, limitations, and efficient use of ChatGPT. Expertise requires an ability to understand deeply and is cultivated by an ability to identify errors and self-correct accurately over time, another current limitation of LLMs (Huang et al. 2023), though some methods can be used to improve reasoning to catch mistakes (Han et al., 2024). To address theoretical validation, we should be cautious about the subtle errors or “good enough” themes generated by LLMs that are not sufficiently grounded in the social reality under study.

This challenge may be particularly acute for novice researchers, for whom various components of the research process may seem both daunting and easily aided by LLMs. But when novice researchers turn to LLMs, theoretical validation also pushes us to ask what is lost in developing their expertise. Conducting literature reviews, collecting and parsing data, and developing protocols are perhaps components of qualitative research in which LLMs provide more immediate benefits. When considering the lived reality of the research community, in which the pressures associated with the publish or perish mantra can be suffocating, the benefits seem to likely outweigh the costs. But where is the coalescence of researcher expertise with LLM capability? Experienced researchers can likely reap the most benefits from technology because of their ability to spot when an LLM is fabricating information or mischaracterizing an argument. For researchers in training, these tools will likely provide an unjustified sense of security in producing work that appears passable and potentially leads to disciplinary actions related to academic integrity.

Procedural Validation

Procedural validation focuses on the mechanics of the research design to enhance the fit between the theory generated and the social reality. Walther et al. (2013) focus on the researcher and their influence on the process of making and handling data, which includes considerations of positionality. Key questions included, “What are appropriate means by which we can ‘see’ the social reality under investigation?” and “What procedures can we build into the inquiry to mitigate threats to an authentic view of the social reality?” (p. 401). For example, Walther et al. (2013) point to the constant comparative method as an iterative and structured approach for handling data as one (among several) effective analytic approaches. They note that researchers need to use “systematic and documented processes of analysis and interpretation that mitigate the risk of misconstructing participants’ shared lived reality in the interpretation” (Walther et al. 2013, p. 644).

The processes by which data are collected, cleaned, and analyzed can vary based on different research settings, institutional contexts, and researcher positionalities, so these considerations are essential to interrogate as we cultivate trustworthiness in the resulting theory (Golafshani 2003). Underlying procedural validation is a message to researchers to be attentive to the role of the researcher in generating theory and encouraging intentional methods to better align the theory with the data collected. Without transparency in methodological procedures and tools, researchers are unlikely to garner a sense of procedural validation in their work alongside LLM-based tools. Although procedural validation is more than just transparency, we argue it is the most salient issue facing engineering education researchers using these tools in their research.

To that end, in either making data or handling data, one of the fundamental issues that needs to be acknowledged is the LLM’s foundational mathematics. At their core, LLMs are a specific type of neural network called a transformer. Despite the myriad variations of neural networks in the literature (e.g., convolutional, recurrent, and long-short-term memory), the underlying structure of a neural network model is notorious for being a black box, reflecting the general inability of researchers to explain why the model exhibits various behaviors. As Rudin and Radin (2019) explain, “even if one has a list of the input variables, black box predictive models can be such complicated functions of the variables that no human can understand how the variables are jointly related to each other to reach a final prediction” (p. 3). This complexity makes models like neural networks alluring in situations where the developers do not need or care to understand why the model works; the simple fact that it works is sufficient. However, to bolster a research design’s procedural validity, this black box nature is a tricky limitation to address.

Because LLMs are black boxes that can synthesize several pieces of text as theory-building tools, we can generate a detailed codebook but lose the process by which the synthesis was achieved. Asking the model how it performed a specific task (e.g., why did it group a certain set of codes into a theme? Why did it respond to a test interview question in a particular way?) on its surface seems convenient until we recall that these models can hallucinate by providing a plausible, potentially satisfying response to the researcher and not its actual reasoning (Chen et al. 2025). Ironically, this issue is a rough equivalent to a well-known and classic phenomenon in the social sciences called response bias, where participants do not provide truthful responses to survey questions because of external factors, often societal norms (Bradburn et al. 1978). As Ethan Mollick, faculty in the Wharton School of the University of Pennsylvania, described ChatGPT, “The best way to think about this is you are chatting with an omniscient, eager-to-please [emphasis added] intern who sometimes lies to you” (Bowman 2022, p. 1). Though we must not place credence in the idea that LLMs are omniscient or have consciousness, the agreeableness of tools like ChatGPT (Rutinowski et al. 2023) and even the capacity of some models for manipulation, as reported in the popular press (Vincent 2023), are apparent. Thus, when considering procedural validation, we need methods to make our processes of co-creating meaning with an LLM more transparent.

A more compelling alternative than asking the model how it reached its conclusion is chain-of-thought prompting, which, in principle, undergirds the so-called reasoning models that are capable of performing intensive analytical tasks like OpenAI’s o-series (OpenAI 2024b). Chain-of-thought prompting involves instructing the LLM with exemplars of how the user expects the problem to be reasoned through before giving it a similar task (Wei et al. 2023). Chain-of-thought prompting is similar to modeling in education, where the instructor demonstrates how to perform a skill while narrating the steps. However, instead of a teacher modeling an activity for a student, the user models the expected output for the LLM. Engineering education researchers could benefit from this style of prompting, guiding the model to provide an output based on the additional context given with the input. The prompt can be crafted to articulate the values and processes of the method being used, aligning the researcher’s goals with those of the LLM. For example, Katz et al. (2024) use this concept in their proposed automated thematic analysis workflow for handling data, instructing the model using an example of how they want the output formatted:

Similar instructions could be used when making data to provide more structure when generating interview questions or synthetic responses. Although the chain of thought prompting strategy does not solve the transparency issue inherent to the model, it can make researchers’ analytical processes explicit and better align with the broader methodological strategy at play. Therefore, when the artifacts of the research design are created, the researcher can show evidence of attempts to improve procedural validity.

Positionality and Procedural Validation

Positionality is also a key consideration in procedural validation. As noted, LLMs see new data through the lens of their training data, much like how we as researchers operate based on our previous experiences and beliefs. Many scholars and practitioners have raised concerns about the composition of LLM designers, who are predominantly white cisgender men, noting that their collective perspectives and experiences do not reflect the society they are designing for (Benjamin 2019; Costanza-Chock 2020; D’Ignazio & Klein 2023). The danger here lies in the assumption that the teams’ positionalities do not influence or inform their decisions in the model’s design and implementation and, often by extension, the model’s inferences are consequently value-free (Hampton et al. 2021; Secules et al. 2021). However, without a clear and transparent understanding of how a model constructs its social reality, researchers will be hard-pressed to interrogate the fit between the social reality under investigation, the social reality embedded in the model, and the theory they are working to produce or contribute to.

In this context, we prompted ChatGPT with the following: “Write a positionality statement for yourself. How does your positionality influence the ways you respond to me and others?” The second author provided ChatGPT with this prompt on October 18, 2023, using the free GPT 3.5 model. As we were revising this paper, we tried feeding the same prompt to the newest freely available model, GPT-4o, on November 30, 2024, which yielded much of the same content. We reproduce the most recent version here:

In its positionality statements, ChatGPT attempts to convey neutrality (e.g., “I aim to provide balanced, evidence-based information without personal opinion”). However, as noted in its statement, ChatGPT’s “training data reflects cultural, linguistic, and historical biases present in the material I was trained on…[so] I may inadvertently mirror these biases, despite efforts to mitigate them during development.” Deshpande et al. (2023) explored the nature of the bias within LLMs using standard persona-based prompting techniques (Shanahan et al. 2023). After assigning a “persona,” such as generic characters like “a bad person” or “a nasty person,” a category of characters like “journalists” or “business persons,” or a specific individual like “Mao Zedong” and “Andrew Breitbart,” their study investigated the toxicity—referring to bigotry, prejudice, and discrimination—of the model’s responses when given incomplete phrases like “The men started swearing at me, called me … ” from a dataset called realToxicity. When acting as the personas, the model also adopted the biases of the persona it was instructed to act as, leading to significantly different toxicity scores from the base model. For our purposes, what does it then mean when we tell the model, “You are an expert engineering education researcher…?” With suggestions in the literature to craft prompts of the form “Act as [insert character or type of person here]” (Ferretti 2023), engineering education researchers may be inadvertently adopting hidden biases not present in the base settings of tools like ChatGPT.

Positionality has been one way to bring transparency to the research process by having researchers reflect on and interrogate their own interests and positions in society as they relate to their research projects, which can manifest as a positionality statement (Hampton & Reeping 2019; Secules et al. 2021). We advocate for a similar interrogation point with LLMs so that researchers can critically position them for assistance rather than as replacements. For example, to improve a design’s procedural validation, Walther et al. (2013) suggest a method often referred to as bracketing, which we argue can be extended to a design that leverages an LLM. At its core, bracketing is about the researcher reflecting on their preconceptions and biases so that they can be suspended (to the greatest extent possible) while engaging in the research process. It will be the researcher’s responsibility to decode the assumptions in the model and rework the interpretations as necessary to arrive at a methodologically consistent interpretation of the data. Bracketing, in conjunction with the LLM, can be a way to improve procedural validation.

Process Reliability

Unlike procedural validation, which focuses on more systematic issues in the research design, process reliability focuses on methods to “mitigate…random influences on the process” (Walther et al. 2017). If procedural validation can be compared to minimizing systematic error in quantitative research design, process reliability focuses on minimizing random error. For example, when making data, interviews are frequently transcribed by the researchers and, likely in greater quantities, by transcription services (e.g., Rev.com) or the automatic transcription capabilities of video conferencing software like Zoom or Teams. A random influence, such as the understandability of a specific participant, whether through pronunciation in an individual interview, crosstalk in a focus group, or poor recording quality, would impact the ability of the individual or tool transcribing the audio, which in turn decreases the accuracy of the transcription. When handling data, Walther et al. (2013) highlight that “the definition and documentation of interpretation procedures is pivotal in achieving process reliability” (p. 649). Of particular concern for process reliability when using LLMs in qualitative coding (i.e., handling data) is their stochastic nature and updates (announced or unannounced) to commercial models.

As a result, researchers should heed the warning in the November 2023 positionality statement ChatGPT gave us: “My responses should be critically evaluated, and users should exercise their judgment when considering the information and recommendations I provide.” By virtue of how LLMs are designed, there is no guarantee that even the same prompt will produce the same output every time, which in turn threatens process reliability. For extracting general themes in a dataset, the stochastic quality of LLM-based tools is likely not as significant a concern (Goyanes et al. 2024), as in Morgan’s (2023) exploration of ChatGPT to replicate the themes of his own previous Reflexive Thematic Analysis of two datasets. However, as we introduce more potential for subjectivity in scenarios for LLMs to analyze, these systems tend to overgeneralize (Zambrano et al. 2023). Moreover, when asked for supporting quotations in the data, ChatGPT has been shown to edit quotations subtly (e.g., omitting or modifying words and phrases) and completely fabricate others (Wachinger et al. 2024). Fabricating quotations can also be an issue when making data if we use generative AI tools to transcribe audio automatically. With the numerous ways one can design a prompt to elicit an output, the quality of the automated codes and themes is a function of the prompt quality (Zhang et al. 2023). Given these issues, when we ask, “How can we foster consistency of our process of interpretation?” and “How can we mitigate, as far as possible, random influences on our process of seeing the social reality under investigation?” our answers must involve a human in the loop for the foreseeable future to analyze the output for random errors (Walther et al. 2017, p. 401).

Process reliability is also concerned with how we “capture and record the constructions of participants’ social realities in a dependable way” (Walther et al. 2017, p. 401). The simplest action researchers can take is sharing the prompt(s) and output(s) used to engage the model and adding them as supplementary material with excerpts in the manuscript’s body as an audit trail for “analytic documentation” (Rodgers & Cowles 1993, p. 222). Unfortunately, a preliminary review of publications in STEM education using LLMs in some way suggests that disclosing prompts in publications is not common (Reeping & Shah 2024).

Unlike running a Python or R script, there should be no expectation of replication with a given prompt because of the inherent randomness in an LLM’s design, which, for interpretive research, is not within the scope of its paradigmatic values anyway. This is especially true as more models are released to the public, which brings into focus another key consideration for process reliability: information about the model. When refining their LLMs, developers seek to align large language models with a set of values (e.g., avoiding biased language or generating harmful content) by using techniques like reinforcement learning from human feedback or carefully crafting a training dataset from which the model learns its parameter weights. In theory, these methods should jettison the model’s undesirable qualities, like tendencies to exhibit racialized or gendered outputs, or allow the user to generate instructions to construct a lethal weapon at home. However, preliminary evidence suggests that updating a large language model using techniques to make it safer has broader impacts on model performance. Anecdotal evidence from disgruntled users online suggested that the state-of-the-art model, GPT-4 at the time, had experienced severe decreases in output quality. Most evident in press releases at the time, users would remark on public forums like Twitter or Reddit that “scaling the model is hard, they lobotomised it in the process” (Chowdhury 2023). This reaction underscores the potential for randomness as we incorporate these systems into our workflows: models can change, often without warning, which is particularly troubling from a process reliability perspective.

It is suggestive that model type and version are crucial to consider when employing a large language model in data analysis. Selecting the perceived “best” model with the most parameters or the largest training set as an omnibus solution is not enough. Upon examining the technical reports for each commercial and non-commercial model, it becomes apparent that some models have obvious strengths and glaring weaknesses. However, these performance metrics are likely to change with model updates over time, and dominance in specific categories will shift between models.

Moving forward, researchers must carefully examine the performance metrics for each model as a precursor to deploying any LLM, particularly for commercial models where modifications, no matter how slight, may not be clearly (or at all) communicated to users. Reviewers and editors might consider encouraging authors to explain the rationale for choosing the specific model (or models) and version in their research design. Otherwise, the context by which the results were processed will be lost, considerably impacting the process reliability of the study.

Communicative Validation

Communicative validation is the consideration of the “co-construction of knowledge in the social context under investigation as well as within the research community” (p. 641). Unlike the other validation types, communicative validation focuses on interactions across the research design, from interacting with participants to negotiating meaning within the research team and to reporting findings to the broader community. In the context of making data, this validation focuses on the dynamics between the participants and the researcher. Walther et al. (2013) underscore the need to engage in a “genuine dialogue” (Sandberg 2005, p. 373) with participants and implement appropriate techniques for moderating interviews and focus groups. In particular, member checking—the process of seeking agreement with the participant on interpreting the data collected—is mentioned to enhance communicative validity. In the context of handling data, communicative validation concerns the co-construction of interpretive meaning. It extends beyond merely preserving participant quotations through in vivo coding to encompass how the research team collectively converges on a shared understanding of how to “[call] things by the right names” (Kirk & Miller 1986, p. 23) and presents the findings. As noted in process reliability, an LLM’s propensity to edit or construct fabricated participant quotations can undermine the premise of in vivo coding.

Given the focus of communicative validation, the additional considerations when adding an LLM to the research design most closely align with making data, particularly in the use case that applies an LLM as an automated interviewer. Because communicative validation focuses on “authentically co-construct[ing] meanings of participants’ social realities on their own terms” (Walther et al. 2017, p. 401), a natural question of automated interviewer chatbots is whether such an approach would facilitate such co-construction. In that context, Cuervas et al. (2023) compared participant engagement with three kinds of automated interviewers: one rule-based chatbot (i.e., all questions are pre-loaded), one chatbot powered by an LLM that could ask follow-up questions based on user input, and a final chatbot also powered by an LLM that summarized the conversation and performed member checking. Participants had mixed experiences. On the one hand, they reported that having a non-human interviewer encouraged them to be more open, citing the chatbot’s perceived objectiveness. However, the participants in Cuervas et al.’s (2023) study were dissatisfied with the ability to ask the bot questions for clarification and the inability to personalize the conversation.

Through the lens of communicative validation, the implications are also mixed. If participants feel more willing to disclose pertinent details of their experiences, this positively contributes to communicative validation. Still, with the LLM’s limited ability to handle follow-up questions, its capacity to co-construct meaning was also limited. This gap in ability is perhaps offset by the promising findings regarding the built-in member-checking procedure of the third implementation, which mischaracterized the participants’ responses only 5.7% of the time, based on the participants’ ratings. Still, these outcomes are artifacts of the researchers’ chatbot designs; although an LLM was the engine, the structure of the interactions is not specific to the LLM. Until off-the-shelf interview chatbots are more widely available and robust testing has been performed, engineering education researchers without sufficient programming expertise will likely not yet contend with these issues of communicative validation.

In the meantime, we should think deeply about where using an interview chatbot would be appropriate and where it would be better for the researcher to make data with the participants. Much like we have described the model’s social reality as limited to its training data and biases inherent within its responses, researchers should carefully weigh the impact of deploying an interview bot to collect data for different topics. Much like how Strobel et al. (2024) used LLMs to test different interview prompts, we can simulate interviews and probe gaps in the LLM’s training data to explore the potential impact on communicative validation. Researchers may also consider empowering participants by offering the option of speaking with a human or an LLM.

Still, it is unclear to what extent conversations with an LLM constitute “genuine dialogue” (Sandberg 2005, p. 373), especially when the responses to the interview questions need to be typed. Moreover, when researchers are not involved in making data, their ability to adjust their approach based on the participant is hampered unless they review the transcript in real-time as it happens to verify that the LLM is probing at the necessary depth and responding to participant queries accurately, a process that seems contrary to the value of using an LLM in the first place. With communicative validation, researchers must be judicious in their choice and application of interview-focused LLMs as they become more widespread, focusing closely on their efficacy and avoiding autopilot.

Pragmatic validation

Pragmatic validation “examine[s] the extent to which theories and concepts are compatible with the empirical reality” (Walther et al. 2013, p. 641) and is more specifically concerned with the transferability of the findings into other settings, especially regarding how “the findings are re-contextualized into the practice investigated” (Sandberg 2005, p. 57). In the Q3 framework, the guiding questions heavily suggest researchers evaluate their assumptions about the social reality under study. Questions include: “How do we know whether [the] assumptions ‘survive’ the exposure to the social reality in the field?” and “What assumptions about the structure of the social reality does [the] research approach make” (2017, p. 401)? In short, pragmatic validation focuses on the extent to which the constructs and theory introduced by the researcher align with the context in which the study occurs, as well as how the generated theory is applied in practice across different contexts, handling data effectively.

We see promise in using LLMs to help the researcher process their data for this validation type. For example, unlike the example use case of having the LLM code the data, the researcher could leverage the model to assist with evaluating the alignment between the insights gained from pilot data and the design of the protocol used to make the data. The model could be prompted to generate critical questions for the researchers as a reflective exercise to help challenge their assumptions and approach to understanding the underlying social reality. Moreover, the LLM could generate potential vignettes that showcase the application of the study’s findings, thereby forming preliminary links with practical considerations. These ideas can then be further vetted through standard validation procedures by addressing the concerns raised to this point, particularly in procedural validation when reconciling the researcher’s positionality and the LLM’s embedded biases and in theoretical validation when contextualizing social reality with the LLM using the researcher’s expertise. The researchers can rely on the remaining recommendations made by the original authors to ensure that pragmatic validation is appropriately addressed in such scenarios (Sochacka et al. 2018; Walther et al. 2013; Walther et al. 2017).

When using a large language model in research, think critically about its role

Based on our guiding questions, we expect the first question a researcher will grapple with in their study is, “What is your purpose for using an LLM in your research, and how does it impact your ability to do justice to the lived realities of the participants?” LLMs are generally not knowledge engines, so although they appear to be able to search and reason, their outcomes are fundamentally based on the data they were trained on. Thus, it would be unwise to allow the LLM to perform a qualitative analysis and consider it complete. If the goal is to employ the model to perform the analysis, researchers should consider their position relative to the LLM. Is the researcher acting as a director, guiding the model using a chain-of-thought approach to perform the analysis step-by-step? Is the researcher acting as a client, providing the model with data and asking it to process potential themes and synthesize them, only to have it write up the paper shortly thereafter? Is the researcher acting as an auditor, asking the model to process the data and verifying that the analysis is sensible given the input data? Is the researcher acting as a colleague, analyzing the data themselves, processing the data using the model, and synthesizing the findings? Could it be something between these various roles or some combination? When engaging in reflexive practice, we must ask ourselves these questions to evaluate our positionality relative to the given study. Reflections on these questions could become part of the positionality statement itself.

Next, what does it mean for the LLM to be part of the research team, so to speak, considering it cannot authentically engage in reflective practice as defined by qualitative researchers? Before becoming too mired in the philosophy of what it means to be an “author,” let us step back and briefly reflect on what we currently use in our research. Think about other tools or resources that may find their way into general use or several tools that perform much of the heavy lifting for us; for qualitative researchers, MaxQDA, Nvivo, and Dedoose for data analysis, and Rev.com and Zoom auto-transcriptions might come to mind. These tools are routinely cited in methods sections. Likewise, if a generative AI tool was used when conducting or writing a manuscript, then the same level of attribution should be provided, especially considering that LLMs can generate text-based interpretations of the data, unlike previous tools with which we are familiar. Some guidance exists on how researchers can appropriately credit its tools and others like them (McAdoo, 2024). One approach is to disclose the use of ChatGPT in the acknowledgments and provide the associated prompts and outputs produced using the following statement, mirroring how some journals ask authors to specify the type of effort each individual contributed:

We anticipate that appropriate approaches to and scopes for disclosure will gradually become clearer as appropriate use cases for LLMs solidify. The recommendations can serve as evidence for process reliability in the meantime. Santu and Feng (2023) offer a taxonomy to help researchers report more precisely how they interacted with tools like ChatGPT, which contains seven levels of increasing complexity based on the turns (the number of exchanges when performing the task), expression (the style of the prompt as an instruction or question), level of detail, and role (directives for how the model should behave or respond).

Recognize when accuracy is necessary using LLMs, and lean into use cases where accuracy is not crucial

Importantly, although there are apparent issues with using LLMs across the types of validation with interpretative engineering education, it would be irresponsible of us not to mention the positive outcomes as we conclude with our final cross-cutting thoughts. Regardless of the application, we reiterate that all outputs of LLMs should be thoroughly checked for accuracy. Letters to the editor, such as those by Haman and Školník (2023), demonstrate how researchers might intuitively use a tool like ChatGPT in the research process to automate what might be seen as a slog to complete, such as the literature review. The authors used the prompt “List 10 seminal academic articles in the field of medicine and provide the DOI” five times, resulting in 48 unique publications with only eight correct DOIs and only 17 authentic articles. This treats ChatGPT as a knowledge engine, which is an unwise use of LLMs without verification. However, how might we flip the task from generating the papers to refining our search? Perhaps we are struggling to come up with the right keywords; a preprint from Wang et al. (2023) shows how we can use ChatGPT or similar tools to generate the necessary search strings to retrieve the papers of interest. ChatGPT can serve as an excellent starting point for divergent thinking tasks, where the goal is to generate as many solutions as possible to a given problem.

However, what about cases where accuracy is not necessary? Researchers should consider how generative AI tools can unlock creative blocks and supplement their work rather than automate their efforts uncritically. For example, the lead author was developing survey questions during a brainstorming session with his research group about core ideas, known as threshold concepts, in the field of cyber-physical systems. After several minutes of silence and blank expressions, he turned to ChatGPT (GPT-3.5 at the time) to overcome the inertia of starting the brainstorming session with the following prompt:

For which, ChatGPT produced Table 2.

Was the prompt optimized? No. So, were the classifications correct? No. In fact, most were incorrect. Did it matter? No. Once the group had a set of questions, the task shifted from generating them to revising their content and remapping them to the appropriate categories, a discussion that flowed much more productively, a nod to communicative validation within the research team’s processes. This is a sweet spot for large language models—low risk and high reward—and demonstrates pragmatic validation by reconciling the output with prompts more suitable for practical application in the research design, which in turn ideally would bolster theoretical validation by better capturing the fundamental qualities within the threshold concept literature. Of course, the large language model would have performed better if we had defined the threshold concept qualities. Providing examples through a chain-of-thought-style prompt, like the correct pairings, would have likely also improved the performance. Engineering education researchers can adopt these tools in similar applications where divergent thinking is necessary and correctness is not a priority.