Human-Gen-AI Collaboration in Developing Situational Judgment Tests for Self-Regulated Problem Solving

2.1 Self-regulated learning problem solving (SRL-PS)

The development of self-regulated learning (SRL) among learners is known to be of great importance nowadays, due to its connection to academic success, effective learning habits, positive emotions and high motivation for learning (Burns et al. 2018; Li et al. 2023). SRL is also known to be of great importance in preparing learners for lifelong learning, and it is important to foster these skills from the early stages in education (de Bruijn-Smolders et al. 2016).

SRL is defined as an active process, during which the learner sets goals for learning, plans times, chooses learning strategies, monitors and controls understanding, motivation and behavior (Boekaerts 1999; Pintrich 2000; Zimmerman 2002). SRL is a circular process consisting of three stages: The preparation stage, in which students set goals and objectives and plan times; the performance phase, which involves monitoring and control processes carrying out the task; and the evaluation phase, in which reflection processes on the entire process take place (Boekaerts 1999; Jansen et al. 2017; Panadero 2017; Zimmerman 1998). SRL includes several key strategies. Cognitive strategies relate to handling information and are specific to various fields. Metacognitive cognitive strategies encompass planning, scheduling, monitoring, and evaluating learning. Motivational strategies involve learners’ beliefs in their abilities and self-efficacy in light of failures and successes in previous tasks, as well as beliefs pertaining to values and interests in the task. These three types of strategies work together to support self-regulated learning. (Boekaerts 1999; Dignath et al. 2008; Pintrich 2000).

In the context of SRL, the problem-solving (PS) process was found to be an important cognitive skill in the 21st century by the OECD (OECD 2021). In the process of solving problems in learning activities, learners utilize a variety of cognitive and metacognitive strategies, such as time management, feedback, critical thinking, and the like, that help them overcome difficulties and solve problems. These PS strategies are cognitive skills that are required for self-regulated learners (Assi & Cohen, 2023).

2.2 Teacher SRL-PS knowledge evaluation

Teachers have a central role in promoting SRL in their classrooms. Teachers who promote SRL are increasing learners’ responsibility and encouraging their participation in the learning process. They also help students develop cognitive, metacognitive, and motivational skills (Alvi & Gillies 2020; Dignath et al. 2008; Porter & Peters-Burton 2021). This is due to the fact that teachers largely control the learning environment through which they provide students with learning experiences that help students develop SRL skills and regulate their learning (Dignath-van Ewijk & Van der Werf 2012). In addition, teacher behavior can play a significant role in the development of students’ SRL skills, especially in the early stages of SRL development when learners are primarily influenced by social supports (Porter & Peters-Burton 2021).

This fact requires a better understanding of which special abilities teachers should possess in the SRL domain, in order to develop SRL skills among their students (Karlen et al. 2020). Teachers often do not feel prepared or adequately supported to integrate SRL into their teaching (Cleary et al. 2022). Therefore, teachers need help to improve their abilities to better support students’ SRL through professional development. Indeed, SRL is an important component of teachers’ professional knowledge (Cleary et al. 2022; Karlen et al. 2020; Porter & Peters-Burton 2021). Despite this, teachers’ required SRL knowledge as well as its evaluation remain unclear (Chen & Jang 2019). Furthermore, since very little is known on how teachers promote SRL in their classes, it is important to assess teachers SRL-PS knowledge (Dignath & Büttner 2018; Kistner et al. 2010).

There are a few basic ways to assess teachers’ SRL-PS knowledge, such as using questionnaires or interviews to gather information about their SRL-promoting activities. These are usually self-reports that are collected before, after, or independently of a specific learning task (Dignath et al. 2008). Another common method of assessment is a systematic observation, to evaluate the teachers’ teaching behavior during lessons (Kistner et al. 2010). Each of the tools has its advantages and limitations. There is a need for more tools to assess teachers’ SRL-PS knowledge in a deeper and broader way (Vosniadou et al. 2020).

2.3 SJT evaluation tools

Situational Judgment Tests (SJTs) are tools that can be used to characterize difficulties and scenarios across various situations and fields, serving both as a learning and evaluation tools (Chao et al., 2020). SJT tools have been used in military settings (Lievens et al. 2008), in retail organizations (Weekley & Jones 1999) and in the selection of medical candidates (Patterson et al. 2016) and aim to assess the decision-making ability of subjects. The subjects are presented with realistic or hypothetical scenarios related to their field of occupation and are required to choose or rate effective responses (Chao et al. 2020). Due to the lack of objective assessment tools designed to define the specific requirements for the teaching profession, it is difficult to empirically investigate what types of knowledge and skills are required for the teaching job in general and in SRL in particular (Aldrup et al. 2020; Gold & Holodynski 2015). Studies in recent years demonstrate the potential of SJT during admission procedures for elementary teacher training programs in middle school and in evaluating different practices in the teaching work (Aldrup et al. 2020; Chao et al. 2020; Klassen et al. 2020). The integration of SJT in evaluating SRL knowledge has not yet been explored.

Development of SJT tools is a long and complex process that follows three steps: The first step is job analysis, where scenarios of work situations are collected by experts in the field or from the literature. In the second step, experts or inexperienced workers create responses for each scenario in order to create a range of effective and ineffective responses, and in the last step, a coding key is developed for responses (Gold & Holodynski 2015; Lievens et al. 2008). In light of the stated above regarding the importance of promoting SRL-PS among learners and evaluating teachers’ knowledge in SRL-PS using diverse tools, this study presents an attempt to develop an SJT tool for evaluating teachers’ SRL-PS knowledge.

2.4 Gen-AI assisted evaluation tools

Artificial intelligence (AI) systems are taking more and more places in our lives in recent years. A significant breakthrough has occurred in the last few years with the developments of base models and large language models (LLM) as well as supporting conversational Gen-AI chatbots that provide the ability to chat with LLMs (Pegrum 2024). These models and chatbots allow the use of a variety of applications and have brought about changes in human-computer interaction. In Gen-AI, using prompts, content is created based on large sources of information that have been entered into the Gen-AI systems as part of its training. An example of such a system is ChatGPT, developed by OpenAI (European Parliamentary 2022).

The ubiquitous impact of Gen-AI throughout society has inevitably permeated the educational domain. Gen-AI tools are now receiving a great deal of attention among those involved in education who deal with the implications of these tools on teaching, learning and assessment (Nasution 2023; Owan et al. 2023). The subject of Gen-AI assisted evaluation in education receives a lot of attention among researchers (Owan et al. 2023; Xia et al. 2024). A review of studies in the field of Gen-AI assisted evaluation in education spans different regions of the world and different disciplines, such as medicine, computer science, and teaching, the leading disciplines in Gen-AI assisted evaluation (Xia et al. 2024).

The development of evaluation tools with the help of Gen-AI can occur at different aspects of development, such as defining the objectives of the tool, identifying the key knowledge and skills to be assessed in the content area, creating a pool of potential test items, scoring the test items, and interpreting the test results (Owan et al. 2023). Previous studies that have used ChatGPT for developing standard multiple-choice tests indicate the development of a test which quality corresponds to experienced humans. Considering that building an evaluation tool is a complex and time-consuming process, development with the help of Gen-AI was found to be faster (Cheung et al. 2023; Ionescu & Enescu 2023; Lee et al. 2023; Nasution 2023).

Regarding SJT, studies in the medicine field used ChatGPT to respond to SJT tests (Borchert et al. 2023; Sareen 2023). In these studies, ChatGPT presented praiseworthy average accuracy response and consistent performance across domains. However, as best as we know, studies have not yet used ChatGPT to develop SJT tests. This paper presents a preliminary attempt to use ChatGPT to bridge the gap in academic research in the development of SJT to evaluate SRL-PS knowledge among teachers.

In light of the fact that the human-Gen-AI interaction is currently expanding, we must also recognize the risks that worry a wide variety of stakeholders (Fui-Hoon Nah et al. 2023). Risks in human-Gen-AI interaction manifest in over-reliance on Gen-AI that can lead to unsafe use and loss of social connection. Another risk is delegating essential decisions to Gen-AI, which may cause loss of skills and autonomy or feelings of disempowerment (Slattery et al. 2024). On the other hand, human-centered AI (HCAI) ensures avoidance of these risks. Delegating power to the various agents according to their capabilities and advantages is important (Drori & Te’eni 2024; Cohen et al. 2021; Te’eni et al. 2023). Thus, AI literacy has been coined to reflect an ability to collaborate with Gen-AI in an efficient and ethical manner with minimal risks (Fui-Hoon Nah et al. 2023). However, research on good practices for collaborating with Gen-AI in education, particularly in assessment, is still emerging (Drori & Te’eni 2024; Te’eni et al. 2023). In this study we examined the contribution of the collaboration between researchers and Gen-AI in the development of an SJT evaluation tool.

4.1 The teachers’ training

The current study was conducted as part of a training program for in-service teachers held at the researches’ university. The main goal was to provide the teachers with an educational program to promote SRL-PS. The training instructed the teachers on a new SRL-PS model developed by the researchers. The SRL-PS model guides the learners along the PS prosses which is divided into four stages: problem identification; problem exploration; providing a solution; and solution evaluation. During these stages learners are instructed with a mix of SRL’s cognitive, metacognitive, and behavioral strategies. The program also included a digital toolbox containing an online checklist of the PS stages enriched with supporting interactive videos (Assi & Cohen 2023; Cohen et al. 2022; Cohen & Cohen 2023). In the middle of the training, the teachers also started implementing the SRL-PS model in their classrooms.

4.2 Participants

The study involved ten in-service female teachers who participated in a training program. These teachers predominantly work in middle and high schools, with a minority teaching in elementary schools. They are teaching various subject areas, represent different sectors of society, and have teaching experience ranging from five to 21 years. The participants enrolled in the training program in response to an advertisement jointly issued by the Office of the Educational Chief Scientist and the university.

4.3 Research procedure

In this article, an attempt was made to harness Gen-AI for the enhancement of the SJT tool for evaluating teachers’ SRL-PS knowledge while saving time and manpower resources. The research procedure was conducted in three sequential phases. The first two phases constituted the development processes of the researchers and Gen-AI of the SJT tool according to previously suggested procedures for developing SJT tools (Aldrup et al. 2020; Chao et al. 2020; Klassen et al. 2020). Thus, the first phase in the research procedure involved the development of the SJT tool by the research team based on teachers’ experience. Subsequently, the second phase of the research procedure utilized ChatGPT4 for the SJT tool development. The first two phases in this development process facilitated the third phase in the research procedure, in which comparative analysis between the researcher-generated and Gen-AI-generated SJT products was enabled. Thus, we studied how human-machine interaction could potentially help improve the development of SJT assessment tools.

Phase two – the joint process: ChatGPT4 development

The researchers examined the potential points in the various steps of the SJT tool development process where Gen-AI could be integrated. It was decided that ChatGPT was relevant in all the steps except for step three. The researchers interacted with ChatGPT in the following way:

• Step 1—SRL-PS knowledge: The researchers examined ChatGPT’s knowledge about SRL-PS by comparing the researcher’s knowledge with ChatGPT’s output. The researchers then uploaded the SRL-PS model as the main body of teachers’ knowledge to be evaluated.

• Step 2—SJT collection methods and categories: The researchers asked ChatGPT to identify difficulty categories that could arise from the implementation of the SRL-PS model in a real classroom.

• Step 4—scenario collection and step 5—scenario formulation: The researchers asked ChatGPT to provide scenarios that reflect the difficulty categories and write them according to the SJT writing standards.

• Step 6—scenario selection: The researchers asked ChatGPT to select the top ten scenarios from the scenarios formulated in a previous step.

In order to interact with ChatGPT to perform these steps, gradual prompt strategies were employed in line with suggested guidelines for prompt engineering (Svendsen & Garvey 2023). Figure 1 shows the dialog with ChatGPT using the prompts. The first column shows the sequence number of the prompt, the second column the instruction given to ChatGPT, and the third column describes the purpose of the prompt.

Figure 1

Figure 2 illustrates the combined researchers and ChatGPT’s processes for developing the SJT tool for evaluating teachers’ SRL-PS knowledge. The six steps are presented in the middle. The researchers’ role in the process is described in the upper part, and the ChatGPT’s role is described in the lower part.

Figure 2

4.4 Research tools

After teachers started implementing the SRL-PS model in their classrooms their input regarding the new model was collected during a session dedicated for this purpose. This data was collected from the teachers in order to assist in developing the SJT tool scenarios, as teachers’ raised difficulties constituted the basis for creating the researchers’ developed scenarios. Additionally, data was collected from ChatGPT to provide its categories and scenarios.

Scenario Questionnaire: In the Scenario Questionnaire, the teachers were required to come up with authentic scenarios related to various difficulties applicable to the teacher or students, according to a literature-based set of predefined difficulty categories (Mishra & Koehler 2006). The categories covered difficulties that focused on teachers and students. For example, pedagogical difficulties focused on teachers, and social difficulties focused on students. These difficulty categories were defined to raise teachers’ awareness of the various aspects of difficulties in implementing the SRL-PS model in their classroom.

Shared SWOT presentation: In a shared presentation the teachers reported about strengths, weaknesses, opportunities, and threats adhering to the SWOT method (Helms & Nixon 2010) to further their feedback about the model and its implementation.

Shared success story presentation: Drawing on the Learning from Success method (Rosenfeld 2016), we asked the teachers to identify one difficulty that they had overcome and tell us how they faced this difficulty. We also asked teachers to comment on other teachers’ responses.

Textual reflection: We asked the teachers to write an overall reflection about the training in the new SRL-PS model.

ChatGPT: We asked ChatGPT to provide difficulty categories and scenarios relating to the SRL-PS model.

4.5 Data analysis

The data was qualitatively analyzed to enable the comparative examination between the products developed by the researchers and ChatGPT and made it possible to examine the potential contributions of human-machine collaboration in the development of the SJT evaluation tool. This comprehensive analysis was collaboratively performed by the five authors of this paper. Thus, discussions were first held with regard to scenarios comparison in general; then, the five researchers jointly assigned their top ten scenarios to ChatGPT’s difficulty categories following a systematic assignment method. Subsequently, the five researchers followed a similar systematic method to assign ChatGPT’s scenarios into the researchers’ difficulty categories.

5.1 Researchers’ difficulties and SJT scenarios for evaluating SRL-PS Knowledge (RQ1)

The SJT tool for evaluating SRL-PS knowledge, as formulated by the researchers, encompasses difficulty categories based on research literature, scenarios title, and the scenarios themselves. Some categories were focused on teachers’ difficulties and some categories focused on students’ difficulties. For example: the pedagogical category, which focused on teachers’ difficulties, produced two scenarios: ‘learning gaps between students,’ and ‘lack of knowledge about students’ progress.’ The social category, which focused on students’ difficulties, produced one scenario: ‘difficulty in group work.’ Altogether, the scenarios built based on the data collected from the teachers, as well as the researchers own input, led to the formulation of scenarios in all difficulty categories. Figure 3 shows the difficulty categories (first column), the title of the scenario (second column), and the ten selected scenarios (third column). Each scenario is assigned an abbreviated code (for example: RS1 stands for research scenario 1).

Figure 3

5.2 ChatGPT’s difficulties and SJT scenarios for evaluating SRL-PS knowledge (RQ2)

The SJT tool for evaluating SRL-PS knowledge, as formulated by ChatGPT, includes difficulty categories and scenarios corresponding to these difficulties. The eight difficulty categories produced by ChatGPT are listed below. The explanation given by ChatGPT is written next to the category title. The titles of each difficulty category and their explanations are presented in the exact output from ChatGPT:

1. Cognitive Development: Students at this age have varying levels of cognitive development, which may affect their ability to grasp complex problem-solving processes.

2. Motivational differences: There can be significant variability in students’ motivation levels, impacting their engagement with the model.

3. Skill diversity: Students may have diverse skill sets, making it challenging to tailor the model to all learning needs.

4. Classroom management: Managing the classroom dynamics while implementing a structured model can be challenging, especially with larger class sizes.

5. Time constraints: The school curriculum may have tight schedules, making it difficult to allocate sufficient time for each stage of the model.

6. Resource limitations: Lack of adequate resources (materials, technology, etc.) can hinder the effective implementation of the model.

7. Assessment alignment: Aligning the model with standard assessment methods and ensuring it meets educational standards and objectives may be challenging.

8. Teacher training: Teachers may require additional training and support to effectively implement and adapt the model in their classrooms.

As seen, the difficulty categories raised by ChatGPT focused on teachers’ and students’ difficulties. Another insight is that the difficulty categories produced by ChatGPT referred to different aspects of SRL knowledge. More specifically, the difficulty categories produced by ChatGPT can be divided into two main class of categories: Management included teachers’ management of classroom dynamics, time allocation, resources, assessment procedures, and professional knowledge. Skill gaps included the categories related to disparities among students in cognitive, motivational, and other skills. After producing the eight difficulty categories, ChatGPT formulated scenarios for these eight categories, and eventually selected the top ten scenarios. Thus, for each difficulty category, ChatGPT selected one or two scenarios. For example: the cognitive development difficulty category produced two scenarios that deal with simplifying a complex task for a fifth grade student and clarifying the constraints of the problem to the student. Figure 4 shows, in the first column, the eight difficulty categories, and in the second column the ten selected scenarios. Each scenario is assigned an abbreviated code (for example: CHS1 stands for the first ChatGPT scenario).

Figure 4

5.3 Researcher-ChatGPT interaction contribution to SJT tool development (QR3)

Combining the findings from Figures 3 and 4 above, during the comparative analysis (phase three described in Research Methodology, above) revealed some preliminary insights concerning the researchers’ interaction with ChatGPT. The researchers’ difficulty categories and ChatGPT’s difficulty categories complemented each other and presented similar and different ways to classify difficulties. Both the researchers and ChatGPT presented difficulty categories in the domain of SRL-PS that focused on students and teachers. However, some noticeable differences stood out: the researchers split difficulty categories into ones which explicitly focused on teachers’ scenarios and ones which explicitly focused on students. ChatGPT did not explicitly suggest such sub-classification focusing on teachers and students’ difficulty categories. However, the researchers noticed that the ChatGPT’s difficulty categories could be divided into management and skill gap categories.

Regarding the scenarios, occasionally, the researchers’ scenarios and the ChatGPT’s scenarios were quite similar (for example: lack of teaching materials for the teacher – RS3, CHS8). Alternatively, some scenarios formulated by ChatGPT expanded or presented another perspective of the difficulty (for example: evaluation and feedback – RS2, CHS9); yet in other instances, new scenarios were proposed by the ChatGPT (for example: Classroom Management – CHS5, CHS6). In addition, in certain cases the researchers’ suggested scenarios which were absent from ChatGPT’s pull of scenarios, primarily the scenarios reflecting lack of students’ understanding of the SRL-PS model itself (RS5–RS8). This indicates that the researchers’ and ChatGPT’s scenarios complement each other.

This analysis led to the creation of the map, which assigns the seven difficulties categories created by the researchers (focused on teacher and focused on student) into the eight ChatGPT difficulty categories. Thus, the comparative analysis enabled the creation of a map consisting of the 20 scenarios created by the researchers and ChatCPT (Figure 5). The dark grey colored squares represent the researchers’ scenarios, code named RS1, RS2, … up to RS10, and the ChatGPT scenarios, codenamed CH1, CH2, … up to CH10. The light grey squares represent difficulty categories where no scenarios were given, but this area has coverage by the parallel agent (researchers or ChatGPT). The white squares represent difficulty categories that lack any scenarios.

Figure 5

Thus, the map highlights several findings. First, difficulty categories were observed that lacked any scenarios (the white squares); these require further treatment. The second finding points to the difficulty categories consisting only of researchers’ scenarios or ChatGPT scenarios (dark grey and light grey squares), which somewhat requires treatment. The third finding highlights the difficulty categories in which several scenarios are offered both by the researchers and ChatGPT (dark grey), suggesting further treatment might not be required. The final finding revealed by the map concerns the emergence of the new, refined integrated categories of the researchers and ChatGPT.

6.1 Contributions and Implications

This paper outlines an initial attempt to develop an SJT tool, utilizing Gen-AI, to assess teachers’ knowledge of SRL-PS following professional development. Additionally, the paper examines how the interactions between the researchers and ChatGPT contributed to the tool’s development. Thus, the contributions of this study are twofold: First, it expands evaluation tools for teachers’ SRL knowledge, which need diversification (Vosniadou et al. 2020) through the SJT methodology. Second, it highlights the beneficial elements of human-centered AI practices (Drori & Te’eni 2024; Te’eni et al. 2023), illustrating the advantages of collaborative interactions between humans and Gen-AI.

The map created serves as a foundation for the next step in developing the SJT tool for assessing teacher knowledge in SRL-PS. This map aids in formulating new and refined scenarios, as well as in selecting the final scenarios based on the integrated categories from researchers and ChatGPT. In terms of utilizing Gen-AI, this study establishes a basis for conducting further experiments on various Gen-AI platforms. As Gen-AI systems are rapidly evolving, they have the potential to spark significant advancements in SJT development, possibly widening the gap between researchers who choose to adopt these technologies and those who do not. Future research should investigate more human-centered AI practices to enhance Gen-AI-assisted evaluation.