Bridging the Gap: ChatGPT’s Role in Enhancing STEM Education

Research Focus and Questions

This study aims to examine the effects of ChatGPT feedback on improving critical thinking, long-term information retention, and collaborative learning dynamics in higher education. The research questions posed to guide this inquiry are as follows:

The relevance of each of these questions is situated in the current educational dilemmas and the hypothetical promise of ChatGPT. The present study intends to respond to these questions to contribute to our current understanding regarding the role of AI in education and its pedagogical potential. The discoveries of this research would have significant implications for how ChatGPT can be used as an instructional tool, possible cultural and hybrid learning approaches, and student involvement in higher education. Knowing how ChatGPT influences learning upsides is key for educators, policymakers, and curriculum creators to use this new type of modern technology.

The Importance and Effectiveness of Feedback in Learning

The feedback that students receive in educational settings profoundly impacts on their learning experiences and the caliber of their course achievement. These impacts are greatly influenced by the source and manner in which the feedback is provided. The theoretical framework of feedback prioritizes precision and clarity to ensure that students comprehensively understand their present performance status and the necessary steps to enhance their performance. Such precise, unambiguous, and meaningful feedback was highly beneficial in guiding students’ learning and improving their performance in reaching learning objectives (Hattie & Timperley, 2007; Luan, 2016; Nicol & Macfarlane-Dick, 2006). Studies that have investigated the impact of feedback supplied by humans in contrast to input provided by an AI tool like ChatGPT have uncovered a variety of findings around student involvement and learning. Researchers also point out how AI can deliver timely, relevant, and individualized feedback. Unlike traditional feedback’s delay or bulkiness, AI can process feedback simultaneously as it delivers it. It is possible to cite several authors (Luan et al., 2023; Naseer et al., 2024; Vashishth et al., 2024; Yesilyurt, 2023).

There has been consensus that AI feedback generated by ChatGPT has the potential to enhance learning with its instant response to queries and submissions, though there has been a paucity of empirical evidence as to the quality improvements (Grassini, 2023; Luan et al., 2023; Michel-Villarreal, 2023). Additionally, the natural language processing (NLP) talents of ChatGPT enable it to produce context-specific responses, furnishing feedback in ways that go far beyond what “push-button” rules permit (Nepal, 2024; Rane, 2023; Wu et al., 2023). These are critical for STEM education, where students’ advanced critical thinking and problem-solving should be prioritized but need ongoing and prompt feedback and assistance. Thus, AI-generated feedback in these domains can significantly benefit students by offering highly customized assistance, which human instructors often struggle to accomplish proficiently (Mollick & Mollick, 2024). Therefore, instead of replacing human feedback systems, integrating AI into educational systems becomes an extra help for the student’s learning process and a source of personalized and instantaneous reactions. Moreover, it is fundamental for educational institutions to reconcile and adjust human and AI feedback to achieve an optimal equilibrium between the volume of knowledge imparted and the total cumulative learning impact (Luan et al., 2024; Nikolopoulou, 2024; Sevnarayan & Potter, 2024).

AI and the Enhancement Critical Thinking and Problem-Solving Skills

Acquiring analytical and problem-solving skills is an essential requirement for achieving proficiency in academic subjects. Hence, AI applications in schooling bring a lot of new prospects and at the same time challenges in fostering such skills. Engagement in critical thinking and problem-solving is a hallmark of learning, and that is central in the STEM science, technology, engineering, and mathematics (STEM) curricula, where students get multifaceted problems that demand novel answers (Ardianti et al., 2020; Baharin et al., 2018; Madden et al., 2013; Stein et al., 2007). The relationship and significance of these talents to the utilization of technical tools and AI remain unclear. The majority of the current study presents comparison data that exhibits significant variation. Reports on how digital tools and AI may affect the cultivation of these skills bring out disparate findings. For instance, many researchers have concluded that AI can contribute to developing critical reasoning and problem-solving, while this can be achieved through a personalized learning experience and learners’ engagement in complex problem-solving situations (Chi & Wylie, 2014; VanLehn, 2011).

Nevertheless, there is an abundance of research findings that are incongruous and present confounding conclusions regarding the influence of AI in education. Therefore, the remaining research interest is dedicated to examining the case of AI – ChatGPT – and its connection to the specified elements. This pertains to two aspects: the ability of ChatGPT to facilitate the expression of supplementary images and concepts alongside a person’s statements, and the techniques involved in the planning process, emotional intelligence, social skills, and goal setting related to the specific case mentioned (Bozkurt et al., 2023; D’Mello et al., 2017; Dwivedi et al., 2023; Nazir & Wang, 2023). Theoretical perspectives on heuristic learning and cognitive engagement suggest that ChatGPT can support the development of critical thinking and problem-solving skills by simulating real-world problem-solving dialogues and providing immediate, context-specific feedback (Baradel, 2023; Graesser et al., 2014; Liu et al., 2023). This interaction engages students in deep cognitive processes and encourages them to apply heuristic strategies to navigate complex problems, enhancing their problem-solving abilities (Lin et al., 2024).

Incorporating artificial intelligence (AI) in educational institutions offers remarkable opportunities for striking a chord with students’ cognitive processes and fostering their analytical and problem-solving skills. For instance, with the advent of AI technologies, particularly conversational AI like ChatGPT, there is now a high likelihood that those tools will enhance heuristic learning and cognitive engagement in STEM education. It is also worth noting that these technologies still need in-depth scientific investigations to facilitate a full evaluation of their impact on learner outcomes and best practices for their utilization in educational setups.

The Role of AI Supporting Knowledge Retention

A significant issue of conventional teaching approaches is the way in which students forget knowledge, consequently failing to transfer it. Nevertheless, AI-enhanced methods can increase the power of influence (Abdellatif et al., 2022; Kamalov et al., 2023; Martin, 2019; Vashishth et al., 2024). The difficulties of learning new information and keeping it for the long term are well-known; studies show that students often forget a lot of what they’ve learned right after class (Roediger & Karpicke, 2006). The cognitive factors like encoding, storage, and retrieval processes, which are responsible for the retention of new knowledge, can be influenced by multiple factors, including the engagement of audiovisual material and usage of reinforcement methods during the instruction (Cepeda et al., 2006, Dai et al., 2021, Dai & Arnberg, 2022, Dai et al., 2023; Luan, 2021; Terry, 2017). Through spaced repetition, research in AI technologies of personalized learning, learning efficiency, and achieving high levels of information retention has been observed. Thus, hand in hand with the custom intervals, spaced repetition is geared towards sharpened memory retention (Karpicke & Roediger, 2007).

AI-based personalized learning systems include adaptive learning, which allows for tailored practice to ensure better recall (Xie et al., 2020). Through ChatGPT and its fantastic text analysis and reasoning capabilities, student retention can be further improved through personalized content and interactive activities. ChatGPT, on the other hand, speeds up the process and eliminates the uncertainty as students get immediate answers to their questions as soon as they confront them. They also do the learning through dialogue with the possibility of execution so that, in the end, the learning experience becomes very tangible (Babu & Akshara, 2024; Dinçer et al., 2023). Moreover, this approach proposes that successful feedback should provide students with information regarding their present performance and learning goals and offer guidance on bridging the gap between the two (Kerfoot et al., 2010). In conclusion, AI-powered dialog systems are among the promising methods that can be used to retain knowledge in the educational process. These personalized, interactive learning practices, which are based on cognitive principles and which students utilize, would likely increase retention rates significantly, thus swelling the entire education outcome.

AI’s Impact on Collaborative Learning and Group Dynamics

Team-based learning and collaborative problem-solving are essential features of modern education. They benefit from technologies such as AI, enabling new forms of student collaboration. The field of educational psychology has recognized the importance of collaborative learning, mainly through social interdependence theory and cooperative learning. These approaches have positively impacted social interaction, knowledge construction, and various aspects of student development, including cognitive, social, and psychological domains (Johnson & Johnson, 2009). Collaborative learning allows students to interact socially, to negotiate, and to share their various and individual perspectives. In his work, Slavin (1996) has shown that all student interactions resulted in a much broader grasp and recollection of the important knowledge content. Research on AI in education reveals the advantages and disadvantages of this innovative change. AI can play this role through the precision of decision-making and shortening the steps of many different processes (Javaid et al., 2022; Rane, 2023). Nevertheless, the use of AI technology in educational settings is not without concerns like equity among students for participation and the tendency of students to rely overly on technology (Pedro et al., 2019).

The role of ChatGPT must be emphasized in a group assignment, and this fact should not be underestimated. It has the possibility to carry conversations that can illustrate how such conversations affect student-student relationships, the assignment of tasks, and the learning assistance between peers when in a group project. Another example could be ChatGPT’s method of placing responsibilities according to certain fields and employing a simulated dialogue to solve disagreements (Ray, 2023). Group dynamics and collaborative learning can be improved with the use of ChatGPT’s communication and feedback features (Abujaber et al., 2023; Amin, 2023; Baskara, 2023; Syahputra, 2023). In addition, by facilitating discussion and directing students to relevant resources, ChatGPT can enhance interaction and collaborative learning. ChatGPT, possesses both capabilities and difficulties when utilized in conjunction with collaborative learning. While ChatGPT can enhance collaborative work, it is important to remain cautious of these concerns to maximize the benefits of working in a group. Future research should examine the impact of AI on crowd behavior and develop strategies to enhance students’ learning in a scientific setting.

Bridge the Gap: The Unique Position of ChatGPT in Educational Feedback

Undoubtedly, AI has become part of almost every industry, including education. However, ChatGPT is peculiar in that it provides comprehensive analysis and profound responses to students. The AI and its applications are gaining popularity due to its capability to adjust with the context and provide the required content regardless of the subject area (Kalla & Smith, 2023; Kasneci et al., 2023; Tlili et al., 2023). ChatGPT’s ability to analyze and produce natural language enables it to provide prompt and deeply congruent feedback with the learner’s comprehension and requirements (Javaid et al., 2023; Koraishi, 2023). The flexibility of this approach guarantees individualized input, which is crucial for successful learning and self-motivation (Dai, et al., 2022; Lin, 2023; Lin & Dai, 2022; Lin et al., 2022; Ray, 2023). However, the level of feedback that human feedback can provide is limited as it is often personal, which contrasts sharply with the scalability of ChatGPT.

Even though Human feedback is unparalleled in its ability to discern variations in context, this can be a minimal method for scaling solutions. However, other AI tools may provide expansion on such surfaces; ChatGPT’s multilayered and deep language model appears to take the lead in this area (Brown et al., 2020; Ray, 2023). This dichotomy of scale and ability to form a varied relationship make ChatGPT a potentially game-changing tool for schools. However, we are just at the beginning of the process of using ChatGPT in education. Also, ongoing experimentation should optimize the blending of ChatGPT in learning environments, emphasizing the bettering and enriching of the human-instruction experience. Newer improvements may involve more advanced robotics, automatic personalization, and the ability to comprehend and adjust to the learner’s pattern (Hirschberg & Manning, 2015). In summary, ChatGPT’s distinctive role in educational feedback underscores AI’s transformational capacity in education. As AI advances, it is crucial to explore its applications to guarantee it remains a formidable ally in achieving educational excellence.

Participants

The participants’ pool consisted of undergraduate students enrolled in the “Introduction to Cognitive Neuroscience” course at a research institute in eastern China. This class was delivered in English, and students completed the assignments and interaction in English as well. Out of the 120 students who were registered for the course, 106 actively participated in the study, resulting in a participation percentage of 88.3%. Participants were identified through classroom announcements and email, a process that took place during the fourth week of the Spring 2023 semester. The Institutional Review Board approved the study. All possible subjects signed an informed consent form that described the study’s objective, methods, signs, risks associated with research, and their rights as research participants.

Students were encouraged to participate in the study and were informed that they would be offered course credits for their involvement. However, participation remained entirely voluntary, and subjects were assured that their participation would not be detrimental to their course success. The participants agreed to join the study, signed consent forms, and returned them to the researchers. Participants ranged in age from 18 to 25 years (M = 20.7, SD = 1.8). The gender distribution was 52% female (n = 55) and 48% male (n = 51). Academic backgrounds were diverse: 65% (n = 69) from social sciences, 25% (n = 26) majoring in psychology, and 15% (n = 16) from computer science and other disciplines. A significant fraction of the class was engaged in social sciences, of which the psychology core curriculum comprised 25%. Other disciplines, including Computer Science (15%), comprised the remaining 35%. Participants were assigned to groups – defendants and plaintiffs – per their preferences and academic interests.

The data and information were anonymized to safeguard the privacy of the participants. Furthermore, all digital data was saved on encrypted, password-protected servers. This process of selecting participants and ethical concerns is carried out appropriately, as it is in good research in education; this ensures a varied sample that represents the group while maintaining ethical standards.

Research Topics and Group Roles

In order to reduce the potential introduction of biases, the study presented a selection of 20 research subjects about cognitive neuroscience for the students to select from. These topics included the influence of music on the brain, cognitive function, neural mechanisms of memory, and the impact of sleep on cognition. Students selected their preferred topics based on their interests and academic backgrounds, ensuring that each topic had a diverse representation of student preferences.

We allowed students to self-select their group members and choose their roles within the groups (Defendant or Plaintiff). The Defendant groups were responsible for creating presentation slides based on their chosen topics, while the Plaintiff groups evaluated these presentations and provided feedback. This process promoted organic group formation, and the task design was centered on students’ interests.

Procedure

We introduced the study objectives and methodology to students during the fourth week of the Spring 2023 semester on the study’s objectives and methodology. They selected their role as Defendant or Plaintiff groups based on their preferences and academic interests. The lecturer presented 20 study topics related to cognitive neuroscience for the groups. The students were given relevant material for each research topic, including references and critical arguments, as well as the tasks that they were anticipated to accomplish. Defendant groups were required to create the presentation slides based on their selected topics and strategies, while Plaintiff groups evaluated these presentations and provided suggestions for improvement. Evaluation criteria were standardized and disseminated to all groups beforehand to ensure fairness and objectivity. Group members were self-selected, promoting organic group formation, with the lecturer approving the final team compositions.

Defendant and Plaintiff groups were required to submit their work to the lecturer via email in a specific order. First, Defendant groups were required to submit their presentation slides, which the lecturer then graded (this constitutes their pre-feedback score). The lecturer sent human or AI feedback to the Defendant group according to the random assignment results. Subsequently, the lecturer forwarded the Defendant group’s slides to the corresponding Plaintiff groups. The Plaintiff groups were given one week to review the slides, identify issues in the arguments, and provide their feedback. After one week, the Plaintiff groups were required to submit their feedback to the lecturer, who then graded it (this constitutes the Plaintiff group’s pre-feedback score). The lecturer sent human or AI feedback to the Plaintiff groups based on the random assignment results.

Both groups were given one week to revise their work based on the feedback they received. After this period, the Defendant groups needed to upload their revised presentation slides to Canvas. The corresponding Plaintiff groups were required to upload their final work, which incorporated the feedback (human or AI), as a post under the corresponding Defendant group’s submission. The lecturer then re-evaluated the updated work of both groups using the same evaluation criteria, assigning post-feedback scores.

Feedback Assignment and Presentation Preparation

Participants in the Defendant and Plaintiff groups were randomly assigned to receive either human feedback from the lecturer or AI-generated feedback from ChatGPT. This randomization aimed to minimize selection bias and bolster the study’s internal validity.

Research Questions

The study’s analysis was structured to examine three research questions:

• RQ1: How does ChatGPT feedback influence the development of critical thinking and problem-solving skills in students compared to traditional human feedback? This question addresses the potential of ChatGPT to foster analytical thinking and the application of knowledge in problem-solving scenarios, which are crucial skills in STEM education.

• RQ2: What is the effect of ChatGPT feedback on the long-term retention of course material? Knowledge retention is a significant challenge in education, and this question explores whether the immediacy and personalization of ChatGPT feedback can lead to more durable learning outcomes.

• RQ3: How does the integration of ChatGPT into group projects affect collaborative learning dynamics and outcomes? Given the importance of teamwork and communication in higher education, this question examines the role of ChatGPT in facilitating and enhancing group interactions and learning processes.

Analysis Plan

We structured our analysis into three main components: grade change analysis (grade change = post-feedback score – pre-feedback score), retention proxy analysis, and collaborative learning dynamics analysis.

Measures

Data Analysis Software and Tools

• The analysis was conducted using R, employing packages such as dplyr for data manipulation, ggplot2 for visualization, rstatix and report for statistical testing, and APA-style reporting.

Research Question 1: Impact of Feedback Type on Grade Changes

A Wilcoxon rank-sum test was conducted to compare the change in grades between students who received feedback from ChatGPT and those who received human feedback. The results indicated a statistically significant difference in the change in grades between the ChatGPT feedback group (median = –12) and the human feedback group (median = –5), W = 5.5,p < .001, with both groups having 10 participants each. This suggests that the type of feedback received significantly impacts students’ grade improvement, with students who received ChatGPT feedback experiencing a more significant median decline in grades than those who received human feedback. The negative median grade changes for both groups suggest that, on average, grades declined from the initial to the final assessment within both the ChatGPT and Human feedback groups, as shown in Figure 1. However, the decline was more pronounced in the ChatGPT group. The statistically significant result indicates that the difference in grade changes between the two feedback types is not likely due to random chance. The larger decrease in grades among students who received ChatGPT feedback may suggest issues related to the effectiveness of the feedback, its relevance, or how it was understood and acted upon by the students. It can reflect differences in the quality or applicability of feedback or how personalized and actionable the feedback was compared to human-provided feedback.

Figure 1

Research Question 2: Influence of Feedback Type on Long-Term Retention

The Retention Proxy measure, composed of equally weighted scores for General Experience, Feedback Usefulness, Relevance, Engagement, and Motivation, provides insight into factors contributing to long-term course material retention. Our findings show that human feedback (M = 5.78, SD = 0.405) was associated with significantly higher Retention Proxy scores than ChatGPT feedback (M = 4.06, SD = 0.411). A Wilcoxon rank-sum test indicated a statistically significant difference between the Retention Proxy scores for the two feedback types, W = 0.5, p < .0001. The effect size, measured by Cohen’s d, was substantial (d = –4.22, 95% CI [–5.34, –3.08]), indicating that human feedback was associated with a significantly higher Retention Proxy score than ChatGPT feedback. This suggests that human feedback may be more effective in promoting long-term retention, possibly due to its ability to provide nuanced, contextually relevant guidance that resonates more deeply with students, as shown in Figure 2.

Figure 2

These findings support the hypothesis that the feedback received can significantly influence factors related to the long-term retention of course material. Human feedback shows a more pronounced positive impact.

Research Question 3: Integration of ChatGPT in Collaborative Learning

The sentiment analysis results indicate a higher net sentiment score for feedback categorized under “Human” (sentiment score = 63) compared to “ChatGPT” (sentiment score = 40). This suggests that human-generated feedback is more positive or less negative than feedback associated with ChatGPT, which can foster a more positive collaborative environment, as shown in Figures 3 and 4.

Figure 3

Figure 4

The word count results show that “Human” feedback contains more words (939) than “ChatGPT” feedback (610). This discrepancy might suggest that human participants provide more detailed feedback than ChatGPT, which can be crucial for learning as it may offer more specific guidance and insights, potentially leading to better understanding and improvement in group projects. Specifically, the human feedback group students perceive feedback as being more valuable as it offers comprehensive details and precise assistance for their work, and more details were recalled and provided in the open-ended question in Figure 5.

Figure 5

In Table 1, we compare the main ten verbs for every feedback type received from people and ChatGPT; Table 1 highlights the distinctions in the actions emphasized by the two. Examples of these are “add,” “make,” and “understand,” which are the most frequently used in human feedback, while “be,” “explore,” and “lead” are the most frequently used in ChatGPT feedback. The dissimilarity in the verbs used for remarks can indicate a focus on slightly different things: human feedback can be more directed at improvements and even understanding ideas, while ChatGPT can be more centered on exploration and leadership. Such differences may lead to a more apparent distinction in the dynamics of collaborative learning, whereby the responses of individual group members vary depending on the status of the feedback.

The analysis of the top 10 N-grams (bigrams) within each group provides detailed insights into the themes and topics emphasized in the feedback, as shown in Figure 6. This linguistic analysis, particularly focusing on verbs, is crucial as verbs often indicate the type of cognitive processes and actions encouraged in educational feedback. Verbs are crucial in educational feedback, and the lack of English language proficiency will limit the understanding of these cognitive processes and actions in generating feedback (Biber & Conrad, 2019; Hyland & Tse, 2004). In the case of feedback generated by ChatGPT for groups labeled “lecturers feedback” or “cognitive functions,” it is likely of educational type feedback and cognitive processes aspect of the learning. The use of action verbs like “explore” and “lead” in ChatGPT feedback (Table 1) shows a preference for student-centered learning by incorporating the principles of active exploration and inquiry. This aligns with contemporary educational theories that put students in the center of the learning process (Weimer, 2013). Differently, human-generated feedback classes having bigrams, such as, “cognitive neuroscience” and “biological aspects,” provide evidence of the more scientific or detailed questioning. The overuse of verbs such as “add,” “make,” or/and “understand” in human feedback, as just example, can mean that the focus is on empirical upheaval or bereft understanding. This is quite the opposite of the student-centered learning technique, which majors on knowledge building and use (Anderson & Krathwohl, 2001). The variance in theme and structure and grammar of the containing contained in feedback would importantly, among other things, divert attention from what students should concentrate on in their projects. Employing such verbs in feedback can affect the way students engage in any cognitive activities and approach learning tasks.

Figure 6

Consequently, the incorporation of ChatGPT into group projects modifies the way the collaborative learning interactions and outcomes are formed in several dimensions. When compared with ChatGPT’s data, human feedback is generally characterized by a greater degree of positivity, detail, and focus on enhancing understanding and students’ performance. The thematic emphasis of feedback also diverges between humans and ChatGPT, which might draw the attention and focus of team members in different directions. These results reveal that although human feedback is still unreplaceable for creating an accessible, comprehensive, and progress-oriented learning environment, ChatGPT is a powerful teaching-and-learning assistant tool for group feedback delivery. We suggest future research delve deeper into the approaches in which these variations in feedback affect particular learning outcomes and group interaction processes.

Grade Changes and Critical Thinking Development

The study found that the ChatGPT feedback group’s grades statistically significantly improved compared with the human feedback group. The students in the ChatGPT feedback group exhibited a much more significant decline in their grades (median = –12) than those receiving human feedback (median = –5). These research results contradict the theory that any personalized feedback can be duplicated by an AI program in classroom (Luan et al., 2023). However, the high reduction in grades of the AI ChatGPT group indicates that the AI-generated feedback might not be effective when it comes to the tasks that require critical thinking and problem-solving skills, which are essential in STEM education (Ardianti et al., 2020; Baharin et al., 2018). Although the study suggests that AI improves learning outcomes in many ways, human instructors are still unreplaceable in individualized guidance (Grassini, 2023; Michel-Villarreal, 2023).

Long-term Retention and Feedback Quality

Human feedback produced superior retention scores compared to ChatGPT feedback by a substantial margin. Meanwhile, a considerable Cohen’s effect size (d = –4.22), supporting the view that human feedback may be more effective in fostering the long-term retention of learning materials, was detected. This corroborates earlier studies with simple or direct evidence on learning retention and consolidation of memory due to affective feedback (Karpicke & Roediger, 2007; Xie et al., 2020). The human feedback conveyed the information in the most engaging way, and the learning remained deep and memorable. This is because it is more adaptive, specific, and relatable advice that connects better with the learners than generic automated feedback. This concept aligns with the heuristic learning and cognitive engagement learning processes, which posit that individualized and interactive educational experiences can substantially enhance recall rates (Graesser et al., 2014; Liu et al., 2023).

Collaborative Learning Dynamics

The experiments that were conducted on the applications of ChatGPT in group projects showed a variety of collaborative learning dynamics. On the aforementioned score, our analysis exhibited a better sentiment score sent by human-generated feedback (63), which was way higher than that one sent by ChatGPT feedback (40), signaling a more affirmative sentiment. Furthermore, human feedback encompassed a more significant number of words (939) than ChatGPT feedback (610), which is a norm in human-to-human communication. The examination of verb and bigram usage proved the source of knowledge in the two systems, human and artificial intelligence. The human approach incorporates more verbs, such as “add,” “make,” and “understand,” whilst ChatGPT only uses “explore” and “lead.” Given the emerging evidence that students may notice these pattern differences as differences between AI writing and human writing, and they may stake their response mostly to this pattern, the implications of these findings are that while ChatGPT may serve as the primary means for providing feedback in collaborative learning settings, it lacks the capacity to foster a positive or optimistic environment and be detailed or problem-centric as human feedback. The outcome has affirmed the proposition that it is supposed to be a combinational approach for tasking purposes where both AI and the human mind are much needed to optimize learning dynamics and the set Sharif outcomes of light on (Adıgüzel et al., 2023; Rasul et al., 2023).