Human-Gen-AI Collaboration in Developing Situational Judgment Tests for Self-Regulated Problem Solving

Teachers play a crucial role in promoting self-regulated learning (SRL) and problem solving (PS) in classrooms, yet their knowledge in these domains remains understudied and challenging to evaluate. While traditional assessment tools like surveys exist, there is a need for diversification, particularly through situational judgment tests (SJT). This study leverages human-AI collaboration to develop an SJT tool for evaluating teachers’ SRL-PS knowledge. Human interaction with generative artificial intelligence (Gen-AI) enables overcoming difficulties and mutually complementing each other. The purpose of this paper is to present an initial attempt to develop an SJT tool for evaluating teachers’ SRL-PS knowledge using the assistance of ChatGPT and to examine how the researcher-ChatGPT interaction contributed to the development of this tool. Through systematic interaction between researchers and ChatGPT, we developed a comprehensive tool comprising 15 difficulty categories and 20 scenarios. The research demonstrates how scenarios created by researchers can be effectively complemented by ChatGPT-generated content. A synthesized map of difficulty categories and scenarios from both researchers and ChatGPT revealed patterns of over- and under-representation across categories, and identified opportunities for synthesized new difficulty categories that could facilitate new scenarios. The study advances SRL knowledge evaluation tools through SJT methodology while demonstrating the benefits of human-Gen-AI collaborative interactions.