Human-Gen-AI Collaboration in Developing Situational Judgment Tests for Self-Regulated Problem Solving
By: Dafna Avidov, Orit Ezra, Guy Cohen, Anat Cohen and Alla Bronshtein
References
- Aldrup, K., Carstensen, B., Köller, M. M., Klusmann, U., & Frenzel, A. C. (2020). Measuring teachers’ social-emotional competence: Development and validation of a situational judgment test. Frontiers in Psychology, 11(May), 1–20. 10.3389/fpsyg.2020.00892
- Alvi, E., & Gillies, R. M. (2020). Teachers and the teaching of self-regulated learning (SRL): The emergence of an integrative, ecological model of SRL-in-context. Education Sciences, 10(4), 98. 10.3390/educsci10040098
- Assi, A., & Cohen, A. (2023, September). Video-based SRL program for teachers and students. Proceedings of the 15th International Conference on Education Technology and Computers (pp. 175–182). 10.1145/3629296.3629323
- Boekaerts, M. (1999). Self-regulated learning: Where we are today. International Journal of Educational Research, 31(6), 445–457. 10.1016/S0883-0355(99)00014-2
- Borchert, R. J., Hickman, C. R., Pepys, J., & Sadler, T. J. (2023). Performance of ChatGPT on the situational judgement test—A professional dilemmas–based examination for doctors in the United Kingdom. JMIR Medical Education, 9(1),
e48978 . 10.2196/48978 - Burns, E. C., Martin, A. J., & Collie, R. J. (2018). Adaptability, personal best (PB) goals setting, and gains in students’ academic outcomes: A longitudinal examination from a social cognitive perspective. Contemporary Educational Psychology, 53(February), 57–72. 10.1016/j.cedpsych.2018.02.001
- Chao, T. Y., Sung, Y. T., & Huang, J. L. (2020). Construction of the situational judgment tests for teachers. Asia-Pacific Journal of Teacher Education, 48(4), 355–374. 10.1080/1359866X.2019.1633621
- Chen, Y. H., & Jang, S. J. (2019). Exploring the relationship between self-regulation and TPACK of Taiwanese secondary in-service teachers. Journal of Educational Computing Research, 57(4), 978–1002. 10.1177/0735633118769442
- Cheung, B. H. H., Lau, G. K. K., Wong, G. T. C., Lee, E. Y. P., Kulkarni, D., Seow, C. S., … & Co, M. T. H. (2023). ChatGPT versus human in generating medical graduate exam multiple choice questions—A multinational prospective study (Hong Kong SAR, Singapore, Ireland, and the United Kingdom). PLoS One, 18(8),
e0290691 . 10.1371/journal.pone.0290691 - Cleary, T. J., Kitsantas, A., Peters-Burton, E., Lui, A., McLeod, K., Slemp, J., & Zhang, X. (2022). Professional development in self-regulated learning: Shifts and variations in teacher outcomes and approaches to implementation. Teaching and Teacher Education, 111, 103619. 10.1016/j.tate.2021.103619
- Cohen, A., Ezra, O., Hershkovitz, A., Tzayada, O., Tabach, M., Levy, B., Segal, A., & Gal, K. (2021). Personalizing mathematical content in educational applets repository: human teacher versus machine-based considerations. Educational Technology Research and Development, 69(3), 1505–1528. 10.1007/s11423-021-10002-x
- Cohen, G., Assi, A., Cohen, A., Bronshtein, A., Glick, D., Gabbay, H., & Ezra, O. (2022).
Video-Assisted Self-Regulated Learning (SRL) Training: COVID-19 Edition . In I. Hilliger, P. J. Muñoz-Merino, T. De Laet, A. Ortega Arranz & T. Farrell (Eds.), Educating for a New Future: Making Sense of Technology-Enhanced Learning Adoption. EC-TEL 2022. Lecture Notes in Computer Science (pp. 59–73). Springer, Cham. 10.1007/978-3-031-16290-9_5 - Cohen, G., & Cohen, A. (2023).
Exploring Students’ Problem-Solving Challenges in Self-regulated Learning Through Training Video Prompts . In O. Viberg, I. Jivet, P. Muñoz-Merino, M. Perifanou, & T. Papathoma (Eds.), Responsive and Sustainable Educational Futures. EC-TEL 2023. Lecture Notes in Computer Science, vol 14200. Springer, Cham. 10.1007/978-3-031-42682-7_38 - De Bruijn-Smolders, M., Timmers, C. F., Gawke, J. C. L., Schoonman, W., & Born, M. P. (2016). Effective self-regulatory processes in higher education: Research findings and future directions. A systematic review. Studies in Higher Education, 41(1), 139–158. 10.1080/03075079.2014.915302
- Dignath, C., Buettner, G., & Langfeldt, H. P. (2008). How can primary school students learn self-regulated learning strategies most effectively? A meta-analysis on self-regulation training programmes. Educational Research Review, 3(2), 101–129. 10.1016/j.edurev.2008.02.003
- Dignath, C., & Büttner, G. (2018). Teachers’ direct and indirect promotion of self-regulated learning in primary and secondary school mathematics classes–insights from video-based classroom observations and teacher interviews. Metacognition and Learning, 13, 127–157. 10.1007/s11409-018-9181-x
- Dignath-van Ewijk, C., & Van der Werf, G. (2012). What teachers think about self-regulated learning: Investigating teacher beliefs and teacher behavior of enhancing students’ self-regulation. Education Research International, 2012(1), 741713. 10.1155/2012/741713
- Drori, I., & Te’eni, D. (2024). Human-in-the-loop AI reviewing: feasibility, opportunities, and risks. Journal of the Association for Information Systems, 25(1), 98–109. 10.17705/1jais.00867
- European Parliamentary Research Service, Scientific Foresight Unit (February 2022). AI Definition & Glossary Key Terms and Concepts.
https://www.europarl.europa.eu/cmsdata/244992/AI%20Glossary%20STOA%2014.02.2022.pdf - Ezra, O., Cohen, A., & Shany, D. (2024). Human experts and Generative AI interaction: From real-world and real-life language learning diary-research app to a diary-learning app. Ubiquity Proceedings, 4(1), 19. 10.5334/uproc.141
- Fui-Hoon Nah, F., Zheng, R., Cai, J., Siau, K., & Chen, L. (2023). Generative AI and ChatGPT: Applications, challenges, and AI-human collaboration. Journal of information technology case and application research, 25(3), 277–304. 10.1080/15228053.2023.2233814
- Gold, B., & Holodynski, M. (2015). Development and construct validation of a situational judgment test of strategic knowledge of classroom management in elementary schools development and construct validation of a situational judgment test of strategic knowledge of classroom management. Educational Assessment, 7197(20), 226–248. 10.1080/10627197.2015.1062087
- Helms, M. M., & Nixon, J. (2010). Exploring SWOT analysis–where are we now? A review of academic research from the last decade. Journal of Strategy and Management, 3(3), 215–251. 10.1108/17554251011064837
- Ionescu, V. M., & Enescu, M. C. (2023, June). Using ChatGPT for generating and evaluating online tests. In 2023 15th International Conference on Electronics, Computers and Artificial Intelligence (ECAI) (pp. 1–6).
IEEE . 10.1109/ECAI58194.2023.10193995 - Jansen, R. S., van Leeuwen, A., Janssen, J., Kester, L., & Kalz, M. (2017). Validation of the self-regulated online learning questionnaire. Journal of Computing in Higher Education, 29(1), 6–27. 10.1007/s12528-016-9125-x
- Karlen, Y., Hertel, S., & Hirt, C. N. (2020). Teachers’ professional competences in self-regulated learning: An approach to integrate teachers’ competences as self-regulated learners and as agents of self-regulated learning in a holistic manner. Frontiers in Education, 5(September), 1–20. 10.3389/feduc.2020.00159
- Kistner, S., Rakoczy, K., & Otto, B. (2010). Promotion of self-regulated learning in classrooms: Investigating frequency, quality, and consequences for student performance. Metacognition and Learning, 5(2), 157–171. 10.1007/s11409-010-9055-3
- Klassen, R. M., Kim, L. E., Rushby, J. V., & Bardach, L. (2020). Can we improve how we screen applicants for initial teacher education? Teaching and Teacher Education, 87, 102949. 10.1016/j.tate.2019.102949
- Lee, U., Jung, H., Jeon, Y., Sohn, Y., Hwang, W., Moon, J., & Kim, H. (2023). Few-shot is enough: Exploring ChatGPT prompt engineering method for automatic question generation in English education. Education and Information Technologies, 1–33. 10.1007/s10639-023-12249-8
- Li, L., Zhu, M. L., Shi, Y. Q., & Yang, L. L. (2023). Influencing factors of self-regulated learning of medical-related students in a traditional Chinese medical university: A cross-sectional study. BMC Medical Education, 23(1), 87. 10.1186/s12909-023-04051-4
- Lievens, F., Peeters, H., & Schollaert, E. (2008). Situational judgment tests: A review of recent research. Personnel Review, 37(4), 426–441. 10.1108/00483480810877598
- Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge related papers. The Development of Technological Pedagogical Content Knowledge in Design Seminar. 10.1177/016146810610800610
- Nasution, N. E. A. (2023). Using artificial intelligence to create biology multiple choice questions for higher education. Agricultural and Environmental Education, 2(1), 4–8. 10.29333/agrenvedu/13071
- Nota, L., Soresi, S., & Zimmerman, B. J. (2004). Self-regulation and academic achievement and resilience: A longitudinal study. International journal of educational research, 41(3), 198–215. 10.1016/j.ijer.2005.07.001
- OECD. (2021). AI and the future of skills, volume 1: Capabilities and assessments. OECD Publishing. 10.1787/5ee71f34-en
- Owan, V. J., Abang, K. B., Idika, D. O., Etta, E. O., & Bassey, B. A. (2023). Exploring the potential of artificial intelligence tools in educational measurement and assessment. Eurasia Journal of Mathematics, Science and Technology Education, 19(8),
em2307 . 10.29333/ejmste/13428 - Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8(Apr), 1–28. 10.3389/fpsyg.2017.00422
- Patterson, F., Zibarras, L., & Ashworth, V. (2016). Situational judgement tests in medical education and training: Research, theory and practice: AMEE Guide No. 100. Medical Teacher, 38(1), 3–17. 10.3109/0142159X.2015.1072619
- Pegrum, M. (December 2024). Generative AI.
https://markpegrum.com/tools-for-digital-learning/generative-ai/ - Pintrich, P. R. (2000).
The role of goal orientation in self-regulated learning . In Handbook of self-regulation (pp. 451–502). Elsevier. 10.1016/B978-012109890-2/50043-3 - Porter, A. N., & Peters-Burton, E. E. (2021). Investigating teacher development of self-regulated learning skills in secondary science students. Teaching and Teacher Education, 105, 103403. 10.1016/j.tate.2021.103403
- Rosenfeld, J. M. (2016). From exclusion to reciprocity: Learning from success. Rowman & Littlefield.
- Sareen, K. (2023). Assessing the ethical capabilities of Chat GPT in healthcare: A study on its proficiency in situational judgement test. Innovations in Education and Teaching International, 1–11. 10.1080/14703297.2023.2258114
- Schubert, S., Ortwein, H., Dumitsch, A., Schwantes, U., Wilhelm, O., Kiessling, C., … & Kiessling, C. (2008). A situational judgement test of professional behaviour: Development and validation. Medical Teacher, 30(5), 528–533. 10.1080/01421590801952994
- Schwartz, D., & Te’eni, D. (2024). AI for knowledge creation, curation, and consumption in context. Journal of the Association for Information Systems, 25(1), 37–47.
https://aisel.aisnet.org/jais/vol25/iss1/12/ - Slattery, P., Saeri, A. K., Grundy, E. A., Graham, J., Noetel, M., Uuk, R., … & Thompson, N. (2024). The AI risk repository: A comprehensive meta-review, database, and taxonomy of risks from artificial intelligence. arXiv preprint arXiv:2408.12622.
https://airisk.mit.edu/ . 10.70777/agi.v1i1.10881 - Svendsen, A., & Garvey, B. (2023). An outline for an interrogative/prompt library to help improve output quality from generative-AI Datasets. (May 2023). 10.2139/ssrn.4495319
- Te’eni, D., Yahav, I., Zagalsky, A., Schwartz, D., Silverman, G., Cohen, D., … & Lewinsky, D. (2023). Reciprocal human-machine learning: A theory and an instantiation for the case of message classification. Management Science. 10.1287/mnsc.2022.03518
- Vosniadou, S., Lawson, M. J., Wyra, M., Van Deur, P., Jeffries, D., & Ngurah, D. I. G. (2020). Pre-service teachers’ beliefs about learning and teaching and about the self-regulation of learning: A conceptual change perspective. International Journal of Educational Research, 99, 101–495. 10.1016/j.ijer.2019.101495
- Weekley, J. A., & Jones, C. (1999). Further studies of situational tests. Personnel Psychology, 52(3), 679–700. 10.1111/j.1744-6570.1999.tb00176.x
- Xia, Q., Weng, X., Ouyang, F., Lin, T. J., & Chiu, T. K. F. (2024). A scoping review on how generative artificial intelligence transforms assessment in higher education. International Journal of Educational Technology in Higher Education, 21, 40. 10.1186/s41239-024-00468-z
- Zimmerman, B. J. (1986). Becoming a self-regulated learner: Which are the key subprocesses?. Contemporary educational psychology, 11(4), 307–313. 10.1016/0361-476X(86)90027-5
- Zimmerman, B. J. (1998).
Developing self-fulfilling cycles of academic regulation: An analysis of exemplary instructional models . In D. H. Schunk & B. J. Zimmerman (Eds.), Self-regulated learning: From teaching to self-reflective practice (pp. 1–19). Guilford. - Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory into Practice, 41(2), 64–70. 10.1207/s15430421tip4102_2
DOI: https://doi.org/10.65043/eurodl.156 | Journal eISSN: 1027-5207
Language: English
Page range: 5 - 5
Submitted on: Dec 25, 2024
Accepted on: May 8, 2025
Published on: Jun 19, 2025
Published by: EDEN Digital Learning Europe
In partnership with: Paradigm Publishing Services
Keywords:
© 2025 Dafna Avidov, Orit Ezra, Guy Cohen, Anat Cohen, Alla Bronshtein, published by EDEN Digital Learning Europe
This work is licensed under the Creative Commons Attribution 4.0 License.