Interrogating the Use of Large Language Models in Qualitative Research Using the Qualifying Qualitative Research Quality Framework

What is your purpose for using an LLM in your research, and how does it impact your ability to do justice to the lived realities of the participants?

How will you consider participant privacy when handling the data?

Will the participants know that an LLM will be used in the research process, and have you obtained their consent?

How is the social reality under study reflected in the training data?

If the model is proprietary (training data is unavailable), how does the ambiguity in the training data influence the study of the full social reality in your research?

How will you determine if the findings generated by an LLM constitute a meaningful contribution to the literature or if findings over-emphasize depth?

What expertise do you possess to evaluate the model output’s quality in the research context?

How can you interrogate the black-box nature of large language models to enhance the trustworthiness of the results?

How will you interrogate the biases in the model and manage them while analyzing the data? And how can you do this in combination with interrogating your positionality?

What robust methods can be used for co-constructing interpretative meanings with the LLM?

To what extent can meaning be co-constructed between the participant and an LLM in the absence of the researcher? And how involved does the researcher need to be in making data?

What methods can be used to reconcile the model’s output with practice?

How will you assess the meaningfulness of the LLM’s output with respect to the social reality under investigation in other contexts?

Because large language models are probabilistic systems, how will you manage the randomness in the output? What about potential inaccuracies?

If the model is commercially hosted and updated (potentially without warning), how can consistency in the findings be achieved?

What information can be disclosed about your use of LLMs to document the process of making and handling the data? (e.g., chat logs, prompts, model version, date of conversation).