Interrogating the Use of Large Language Models in Qualitative Research Using the Qualifying Qualitative Research Quality Framework
By: David Reeping, Cynthia Hampton and Desen Özkan
References
- Azaria, A., Azoulay, R., & Reches, S. (2023). ChatGPT is a Remarkable Tool—For Experts. (arXiv:2306.03102). arXiv. 10.1162/dint_a_00235
- Baidoo-Anu, D., & Ansah, L. O. (2023). Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Journal of AI, 7(1), 52–62. 10.2139/ssrn.4337484
- Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021, March). On the dangers of stochastic parrots: Can language models be too big? 🦜. Paper presented at the 2021 Conference on Fairness, Accountability, and Transparency. 10.1145/3442188.3445922
- Benjamin, R. (2019). Race after technology: Abolitionist tools for the New Jim Code. Polity.
- Berdanier, C. G. P., & Alley, M. (2023). We still need to teach engineers to write in the era of ChatGPT. Journal of Engineering Education, 112(3), 583–586. 10.1002/jee.20541
- Biswas, S. S. (2023). ChatGPT for research and publication: A step-by-step guide. The Journal of Pediatric Pharmacology and Therapeutics, 28(6), 576–584. 10.5863/1551-6776-28.6.576
- Bowman, E. (2022, December 19). A new AI chatbot might do your homework for you. But it’s still not an A+ student. NPR.
https://www.npr.org/2022/12/19/1143912956/chatgpt-ai-chatbot-homework-academia - Bradburn, N., Sudman, S., Blair, E., & Stocking, C. (1978). Question threat and response bias. Public Opinion Quarterly, 42(2), 221–234. 10.1086/268444
- Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., Somani, A., Hase, P., Wagner, M., Roger, F., Mikulik, V., Bowman, S. R., Leike, J., Kaplan, J., & Perez, E. (2025). Reasoning models don’t always say what they think. (arXiv:2505.05410). arXiv. 10.48550/arXiv.2505.05410
- Chopra, F., & Haaland, I. (2023).
Conducting qualitative interviews with AI (SSRN Scholarly Paper 4583756) . Social Science Research Network. 10.2139/ssrn.4583756 - Chowdhury, A. (2023, May 31). ChatGPT may have been quietly nerfed recently—Here’s why.
https://www.videogamer.com/news/chatgpt-nerfed/ - Costanza-Chock, S. (2020). Design justice: Community-led practices to build the worlds we need. The MIT Press.
https://library.oapen.org/handle/20.500.12657/43542 . 10.7551/mitpress/12255.001.0001 - Cuervas, A. C., Brown, E. M., Scurrell, J. V., Entenmann, J., & Daepp, M. I. G. (2023, September 18). Automated interviewer or augmented survey? Collecting social data with large language models.
arXiv.Org .https://arxiv.org/abs/2309.10187v2 - Curry, N., Baker, P., & Brookes, G. (2024). Generative AI for corpus approaches to discourse studies: A critical evaluation of ChatGPT. Applied Corpus Linguistics, 4(1),
100082 . 10.1016/j.acorp.2023.100082 - Dastin, J. (2023, March 14). Microsoft-backed OpenAI starts release of powerful AI known as GPT-4. Reuters.
https://www.reuters.com/technology/microsoft-backed-openai-starts-release-powerful-ai-known-gpt-4-2023-03-14/ - Davison, R. M., Chughtai, H., Nielsen, P., Marabelli, M., Iannacci, F., van Offenbeek, M., Tarafdar, M., Trenz, M., Techatassanasoontorn, A. A., Díaz Andrade, A., & Panteli, N. (2024). The ethics of using generative AI for qualitative data analysis. Information Systems Journal, 34(5), 1433–1439. 10.1111/isj.12504
- de Vries, A. (2023). The growing energy footprint of artificial intelligence. Joule, 7(10), 2191–2194. 10.1016/j.joule.2023.09.004
- Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., & Narasimhan, K. (2023). Toxicity in ChatGPT: Analyzing persona-assigned language models. (arXiv:2304.05335). arXiv. 10.18653/v1/2023.findings-emnlp.88
- D’Ignazio, C., & Klein, L. F. (2023). Data feminism. MIT Press.
- Dzieza, J. (2023, June 20). AI is a lot of work. New York Magazine/The Verge.
https://longreads.com/2023/06/20/ai-is-a-lot-of-work/ - Ferretti, S. (2023). Hacking by the prompt: Innovative ways to utilize ChatGPT for evaluators. New Directions for Evaluation, 2023(178–179), 73–84. 10.1002/ev.20557
- Froyd, J. E., Wankat, P. C., & Smith, K. A. (2012). Five major shifts in 100 years of engineering education. Proceedings of the IEEE, 100 (Special Centennial Issue), 1344–1360. 10.1109/JPROC.2012.2190167
- Gamieldien, Y., Case, J. M., & Katz, A. (2023).
Advancing qualitative analysis: An exploration of the potential of generative AI and NLP in thematic coding (SSRN Scholarly Paper 4487768) . Social Science Research Network. 10.2139/ssrn.4487768 - Golafshani, N. (2003). Understanding reliability and validity in qualitative research. The Qualitative Report, 8(4), 597–606. 10.46743/2160-3715/2003.1870
- Goyanes, M., Lopezosa, C., & Jordá, B. (2024).
Thematic analysis of interview data with ChatGPT: Designing and testing a reliable research protocol for qualitative research . OSF. 10.31235/osf.io/8mr2f - Grant, N., & Metz, C. (2023, March 21). Google releases Bard, its AI chatbot, a rival to ChatGPT and Bing. The New York Times.
https://www.nytimes.com/2023/03/21/technology/google-bard-chatbot.html - Haman, M., & Školník, M. (2023). Using ChatGPT to conduct a literature review. Accountability in Research, 0(0), 1–3. 10.1080/08989621.2023.2185514
- Hampton, C., & Reeping, D. (2019). Positionality: The stories of self that impact others. Paper presented at 2019 ASEE Annual Conference & Exposition, Tampa, Florida, USA. 10.18260/1-2--33177
- Hampton, C., Reeping, D., & Ozkan, D. S. (2021). Positionality statements in engineering education research: A look at the hand that guides the methodological tools. Studies in Engineering Education, 1(2), Article
2 . 10.21061/see.13 - Han, H., Liang, J., Shi, J., He, Q., & Xiao, Y. (2024). Small language model can self-correct (arXiv:2401.07301). arXiv. 10.48550/arXiv.2401.07301
- Hernández, A., & Amigó, J. M. (2020). Differentiable programming and its applications to dynamical systems (arXiv:1912.08168). arXiv. 10.48550/arXiv.1912.08168
- Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. 10.1162/neco.1997.9.8.1735
- Hu, K. (2023, February 2). ChatGPT sets record for fastest-growing user base—Analyst note. Reuters.
https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/ - Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023). Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798.
- Jalali, M. S., & Akhavan, A. (2024). Integrating AI language models in qualitative research: Replicating interview data analysis with ChatGPT. System Dynamics Review, 40(3),
e1772 . 10.1002/sdr.1772 - Johri, A., Katz, A. S., Qadir, J., & Hingle, A. (2023). Generative artificial intelligence and engineering education. Journal of Engineering Education, 112(3), 572–577. 10.1002/jee.20537
- Katz, A., Fleming, G. C., & Main, J. (2024, September 28). Thematic analysis with open-source generative AI and machine learning: A new method for inductive qualitative codebook development.
arXiv.Org .https://arxiv.org/abs/2410.03721v1 - Katz, A., Gerhardt, M., & Soledad, M. (2024). Using generative text models to create qualitative codebooks for student evaluations of teaching. International Journal of Qualitative Methods, 23, 16094069241293283. 10.1177/16094069241293283
- Katz, A., Shakir, U., & Chambers, B. (2023, May 29). The utility of large language models and generative AI for education research.
arXiv.Org .https://arxiv.org/abs/2305.18125v1 - Katz, A., Wei, S., Nanda, G., Brinton, C., & Ohland, M. (2023). Exploring the efficacy of ChatGPT in analyzing student teamwork feedback with an existing taxonomy (arXiv:2305.11882). arXiv. 10.48550/arXiv.2305.11882
- Kirk, J., & Miller, M. L. (1986). Reliability and validity in qualitative research. SAGE. 10.4135/9781412985659
- Knight, W. (2023, April 17). OpenAI’s CEO says the age of giant AI models is already over. Wired.
https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/ - Laiq, M., & Dieste, O. (2020). Chatbot-based interview simulator: A feasible approach to train novice requirements engineers. Paper presented at the 2020 10th International Workshop on Requirements Engineering Education and Training (REET), Zurich, Switzerland. 10.1109/REET51203.2020.00007
- Lehr, S. A., Caliskan, A., Liyanage, S., & Banaji, M. R. (2024). ChatGPT as research scientist: Probing GPT’s capabilities as a research librarian, research ethicist, data generator, and data predictor. Proceedings of the National Academy of Sciences, 121(35),
e2404328121 . 10.1073/pnas.2404328121 - Luitse, D., & Denkena, W. (2021). The great Transformer: Examining the role of large language models in the political economy of AI. Big Data & Society, 8(2), 20539517211047734. 10.1177/20539517211047734
- Martin, J., Desing, R., & Borrego, M. (2022). Positionality statements are just the tip of the iceberg: moving towards a reflexive process. Journal of Women and Minorities in Science and Engineering, 28(4). 10.1615/JWomenMinorScienEng.2022044277
- McAdoo, T. (2024, February 23). How to cite ChatGPT. American Psychological Association.
https://apastyle.apa.org/blog/how-to-cite-chatgpt - Mejia, J., Alarcón, I. V., Mejia, J., & Revelo, R. (2022). Legitimized tongues: Breaking the traditions of silence in mainstream engineering education and research. Journal of Women and Minorities in Science and Engineering, 28(2). 10.1615/JWomenMinorScienEng.2022036603
- Mejia, J., Revelo, R., Villanueva, I., & Mejia, J. (2018). Critical theoretical frameworks in engineering education: An anti-deficit and liberative approach. Education Sciences, 8(4),
158 . 10.3390/educsci8040158 - Menekse, M. (2023). Envisioning the future of learning and teaching engineering in the artificial intelligence era: Opportunities and challenges. Journal of Engineering Education, 112(3), 578–582. 10.1002/jee.20539
- Meta. (2024). Running Meta Llama on Windows | Llama Everywhere.
https://www.llama.com/docs/llama-everywhere/running-meta-llama-on-windows/ - Milner, H. R. (2007). Race, culture, and researcher positionality: Working through dangers seen, unseen, and unforeseen. Educational Researcher, 36(7), 388–400. 10.3102/0013189X07309471
- Morgan, D. L. (2023). Exploring the use of artificial intelligence for qualitative data analysis: The case of ChatGPT. International Journal of Qualitative Methods, 22, 16094069231211248. 10.1177/16094069231211248
- OpenAI. (2024a). How your data is used to improve model performance | OpenAI Help Center.
https://help.openai.com/en/articles/5722486-how-your-data-is-used-to-improve-model-performance - OpenAI. (2024b, September 12). Introducing OpenAI o1.
https://openai.com/o1/ - Prescott, M. R., Yeager, S., Ham, L., Saldana, C. D. R., Serrano, V., Narez, J., Paltin, D., Delgado, J., Moore, D. J., & Montoya, J. (2024). Comparing the efficacy and efficiency of human and generative AI: Qualitative thematic analyses. JMIR AI, 3(1),
e54482 . 10.2196/54482 - Rahman, M. M., Terano, H. J., Rahman, M. N., Salamzadeh, A., & Rahaman, M. S. (2023). ChatGPT and academic research: A review and recommendations based on practical examples (SSRN Scholarly Paper 4407462). Social Science Research Network.
https://papers.ssrn.com/abstract=4407462 - Reeping, D., & Shah, A. (2024, June 23). Board 50: Work in progress: A systematic review of embedding large language models in engineering and computing education. Paper presented at the 2024 ASEE Annual Conference & Exposition, Portland, Oregon, USA.
https://peer.asee.org/board-50-work-in-progress-a-systematic-review-of-embedding-large-language-models-in-engineering-and-computing-education - Rodgers, B. L., & Cowles, K. V. (1993). The qualitative research audit trail: A complex collection of documentation. Research in Nursing & Health, 16(3), 219–226. 10.1002/nur.4770160309
- Rowe, W. (2014).
Positionality . In D. Coghlan & M. Brydon-Miller (Eds.), The SAGE encyclopedia of action research (p. 628). Sage. - Rudin, C., & Radin, J. (2019). Why are we using black box models in AI when we don’t need to? A lesson from an explainable AI competition. Harvard Data Science Review, 1(2). 10.1162/99608f92.5a8a3a3d
- Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), Article
6088 . 10.1038/323533a0 - Rutinowski, J., Franke, S., Endendyk, J., Dormuth, I., & Pauly, M. (2023). The self-perception and political biases of ChatGPT (arXiv:2304.07333). arXiv. 10.1155/2024/7115633
- Sandberg, J. (2005). How do we justify knowledge produced within interpretive approaches? Organizational Research Methods, 8(1), 41–68. 10.1177/1094428104272000
- Santu, S. K. K., & Feng, D. (2023). TELeR: A general taxonomy of LLM prompts for benchmarking complex tasks (arXiv:2305.11430). arXiv. 10.48550/arXiv.2305.11430
- Savolainen, J., Casey, P. J., McBrayer, J. P., & Schwerdtle, P. N. (2023). Positionality and its problems: Questioning the value of reflexivity statements in research. Perspectives on Psychological Science, 18(6), 1331–1338. 10.1177/17456916221144988
- Secules, S., McCall, C., Mejia, J. A., Beebe, C., Masters, A. S., Sánchez-Peña, M. L., & Svyantek, M. (2021). Positionality practices and dimensions of impact on equity research: A collaborative inquiry and call to the community. Journal of Engineering Education, 110(1), 19–43. 10.1002/jee.20377
- Shanahan, M., McDonell, K., & Reynolds, L. (2023). Role play with large language models. Nature, 623(7987), 493–498. 10.1038/s41586-023-06647-8
- Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2019). The woman worked as a babysitter: On biases in language generation (arXiv:1909.01326). arXiv. 10.48550/arXiv.1909.01326
- Singleton, R. A., & Straits, B. C. (2017). Approaches to social research (6th edition). Oxford University Press.
- Sochacka, N. W., Walther, J., & Pawley, A. L. (2018). Ethical validation: Reframing research ethics in engineering education research to improve research quality. Journal of Engineering Education, 107(3), 362–379. 10.1002/jee.20222
- Sok, S., & Heng, K. (2023). ChatGPT for education and research: A review of benefits and risks (SSRN Scholarly Paper 4378735). 10.2139/ssrn.4378735
- Strobel, J., Medina, M., & Guzman, E. S. (2024). Exploring AI Bots as simulators in human subject research: A novel approach to ethical and efficient experimentation in engineering education research. Paper presented at the IEEE Frontiers in Education Conference 2024, Washington DC, USA. 10.1109/FIE61694.2024.10893007
- Surovell, E. (2023, February 8). ChatGPT has everyone freaking out about cheating. It’s not the first time. The Chronicle of Higher Education.
https://www.chronicle.com/article/chatgpt-has-everyone-freaking-out-about-cheating-its-not-the-first-time - Tangermann, V. (2023, January 21). A new scientific paper credits ChatGPT AI as a coauthor. Futurism.
https://futurism.com/scientific-paper-credits-chatgpt-ai-coauthor - Teixeira da Silva, J. A. (2023). Is institutional review board approval required for studies involving ChatGPT? American Journal of Obstetrics & Gynecology MFM, 5(8),
101005 . 10.1016/j.ajogmf.2023.101005 - Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.
https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html - Vincent, J. (2023, February 15). Microsoft’s Bing is an emotionally manipulative liar, and people love it. The Verge.
https://www.theverge.com/2023/2/15/23599072/microsoft-ai-bing-personality-conversations-spy-employees-webcams - Wachinger, J., Bärnighausen, K., Schäfer, L. N., Scott, K., & McMahon, S. A. (2024). Prompts, pearls, imperfections: Comparing ChatGPT and a human researcher in qualitative data analysis. Qualitative Health Research,
10497323241244669 . 10.1177/10497323241244669 - Walther, J., Pawley, A. L., & Sochacka, N. W. (2015). Exploring ethical validation as a key consideration in interpretive research quality. 26.726.1–26.726.21.
https://peer.asee.org/exploring-ethical-validation-as-a-key-consideration-in-interpretive-research-quality . 10.18260/p.24063 - Walther, J., & Sochacka, N. (2014). Qualifying qualitative research quality (The Q3 project): An interactive discourse around research quality in interpretive approaches to engineering education research. Paper presented at the 2014 IEEE Frontiers in Education Conference (FIE), Madrid, Spain. 10.1109/FIE.2014.7043988
- Walther, J., Sochacka, N. W., Benson, L. C., Bumbaco, A. E., Kellam, N., Pawley, A. L., & Phillips, C. M. L. (2017). Qualitative research quality: A collaborative inquiry across multiple methodological perspectives. Journal of Engineering Education, 106(3), 398–430. 10.1002/jee.20170
- Walther, J., Sochacka, N. W., & Kellam, N. N. (2013). Quality in interpretive engineering education research: Reflections on an example study: Quality in interpretive engineering education research. Journal of Engineering Education, 102(4), 626–659. 10.1002/jee.20029
- Wang, S., Scells, H., Koopman, B., & Zuccon, G. (2023). Can ChatGPT write a good boolean query for systematic review literature search? (arXiv:2302.03495). arXiv. 10.48550/arXiv.2302.03495
- Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2023). Chain-of-thought prompting elicits reasoning in large language models (arXiv:2201.11903). arXiv. 10.48550/arXiv.2201.11903
- Zambrano, A. F., Liu, X., Barany, A., Baker, R. S., Kim, J., & Nasiar, N. (2023).
From nCoder to ChatGPT: From automated coding to refining human coding . In G. Arastoopour Irgens & S. Knight (Eds.), Advances in quantitative ethnography (pp. 470–485). Switzerland: Springer Nature. 10.1007/978-3-031-47014-1_32 - Zhang, H., Wu, C., Xie, J., Kim, C., & Carroll, J. M. (2023, October 10). QualiGPT: GPT as an easy-to-use tool for qualitative coding.
arXiv.Org .https://arxiv.org/abs/2310.07061v1
DOI: https://doi.org/10.21061/see.174 | Journal eISSN: 2690-5450
Language: English
Page range: 1 - 23
Submitted on: Mar 21, 2024
Accepted on: Jun 4, 2025
Published on: Jul 4, 2025
Published by: Virginia Tech Publishing
In partnership with: Paradigm Publishing Services
Keywords:
© 2025 David Reeping, Cynthia Hampton, Desen Özkan, published by Virginia Tech Publishing
This work is licensed under the Creative Commons Attribution 4.0 License.