Introduction
Feedback is the reinforcing or corrective information provided to students regarding their performance in a learning environment (Bardwell, 1981). In writing instruction, formative feedback plays a crucial role in enhancing students’ writing proficiency (Graham et al., 2011; MacArthur, 2016; Saliu-Abdulahi et al., 2017). Formative feedback not only helps students understand their current level of writing achievement and guides them on how to improve but also encourages continuous development throughout the learning process (Balta & Türel, 2013; Çetinkaya & Yolcusoy, 2020; Graham et al., 2012; Timperley & Parr, 2010).
Although teacher feedback is preferred by students (Çetinkaya & Hamzadayı, 2015) and often more effective than peer and self-feedback (Beach & Friedrich, 2006), the effort required to provide feedback significantly increases teachers’ workload (Ballet & Kelchtermans, 2009; Dibbon, 2004). Additionally, given that teachers typically instruct multiple large classes, the time required for providing feedback may limit their ability to focus on teaching fundamental writing skills (Applebee & Langer, 2011; Graham, 2019). Reducing teachers’ workload related to feedback provision in writing instruction could allow them to allocate more time to other aspects of the writing process (Biber et al., 2011; Graham et al., 2015; Grimes & Warschauer, 2010; Warschauer & Grimes, 2008; Wilson & Czik, 2016; Wilson & Roscoe, 2020). Consequently, alternative methods of providing feedback in writing education have become increasingly necessary.
Research and advancements in writing instruction have led to a broader range of feedback sources available to students. In addition to teacher feedback, peer feedback, self-assessment, writing workshops, oral conferences, and computer-assisted feedback have been incorporated into writing instruction (Hyland & Hyland, 2006). One such computer-assisted feedback method, automated writing evaluation (AWE), has attracted attention from educators due to its potential to assess student writing, assist teachers, and reduce their feedback workload (Kellogg et al., 2010; Liu et al., 2024; Wilson & Roscoe, 2020; Wilson et al., 2022). These systems generally utilize natural language processing (NLP) techniques or artificial intelligence (AI) to evaluate students’ written texts. Numerous studies have demonstrated that feedback provided through such systems positively impacts students’ writing performance (Graham et al., 2015; Grimes & Warschauer, 2010; Lee, 2019; Roscoe et al., 2017; Stevenson & Phakiti, 2014; Wilson & Czik, 2016; Wilson & MacArthur, 2016; Wilson & Roscoe, 2020; Zhai & Ma, 2022).
On the other hand, some researchers (MacArthur, 2016; Wilson et al., 2022; Yancey et al., 2004) have argued that AWE applications predominantly focus on scoring rather than providing meaningful feedback, and due to the variability in students’ responses to such feedback, these tools may be detrimental to students’ writing skills. Additionally, they have emphasized that integrating these tools into learning environments takes time (Chen et al., 2022; Moore & MacArthur, 2016; Shermis, 2014; Ware & Warschauer, 2006; Xie et al., 2020). While these discussions continue, the use of both paid and free AWE applications has been increasing, particularly in technologically advanced countries like the United States and China. However, while studies suggest that these applications offer scoring comparable to—or even higher in quality than—that of teachers (Shermis & Hamner, 2013; Stevenson & Phakiti, 2014), their long-term effectiveness in improving students’ writing skills remains highly limited. The available evidence is insufficient, and researchers have emphasized the need for more comprehensive studies on this topic (Xiao et al., 2025).
Amid ongoing debates about the advantages and drawbacks of integrating AI-based systems into education, OpenAI’s 2022 release of ChatGPT has sparked both excitement (Derakhshan & Ghiasvand, 2024; Lee, 2023) and concern (Akgun & Greenhow, 2022; Chomsky et al., 2023; Harari, 2023). The fundamental distinction between ChatGPT and similar models (Claude, Poe Aı, Gemini, Grok, Bing, etc.) and traditional AI-driven autonomous systems is their ability to not only mimic human language but also process and generate text, comprehend generated content, ask and answer questions, and interact dynamically with users (Bozkurt, 2020, 2023a). The introduction of generative AI (GenAI) tools has not only increased the popularity of automated written feedback (AWF) but also heightened expectations regarding the quality of feedback these systems can offer (Barrot, 2023; Burstein, 2020). Based on these developments, AWF shows significant potential for providing feedback in writing instruction, and with advancements in GenAI models, educators are likely to incorporate these systems more frequently into their teaching. Additionally, some studies (Hsieh et al., 2017; Kluger & Adler, 1993) have classified AWF as a form of computer-assisted feedback, similar to AWE, that includes both automated feedback and GenAI responses.
With the introduction of GenAI tools, AI-supported chatbots are increasingly being used to provide feedback on students’ writing (Guo & Wang, 2024; Kohnke et. al., 2023; Xiu-Yi, 2024; Yang & Kyun, 2022; Yang & Li, 2024). Additionally, GenAI platforms such as Poe AI enable users to create AI-powered chatbots trained for specific purposes, using a foundational GenAI model, at no cost.
Despite the growing body of research on the use of AWE and AWF in writing education—particularly in English and Chinese—an extensive literature review reveals no applied research on this subject in Turkish writing instruction. A previous review suggested that AI could be utilized to provide feedback to students (Akkaya & Çıvgın, 2021); however, no empirical studies have been conducted in the Turkish context. Therefore, the present study aims to examine the impact of feedback provided by a GenAI chatbot developed by the researchers on the informative text writing performance and retention of middle school students. Furthermore, the study seeks to analyze the nature, similarities, and differences of the feedback provided. This study is significant, as it represents the first applied research on the use of GenAI systems for feedback provision in Turkish writing instruction. Moving forward, this research is expected to support the development of GenAI-based AWE and AWF systems, particularly in the context of native language education, and further advance feedback-driven AI applications in writing pedagogy.
Research AIM
The aim of this study is to examine the impact of feedback provided by a teacher and a chatbot on informative texts written by fifth-grade students in Turkish language courses. Specifically, the study investigates how the content of the feedback influences students’ writing performance in informative text composition.
In line with this objective, the research seeks to answer the following questions:
Is there a significant difference in the effect of formative feedback provided by a chatbot versus a teacher on students’ performance in writing informative text?
Does the effect of formative feedback from a chatbot or a teacher on students’ informative text writing performance persist over time?
What are the similarities or differences between the formative feedback provided by the chatbot versus the teacher?
Literature
Types of Feedback
Although the literature presents various classifications of teacher feedback on written texts, it is generally categorized into feedback type and feedback level. Feedback types are further classified into direct and indirect feedback based on how they are presented to the student (Biber et al., 2011). Direct feedback provides explicit instructions for revising the text, while indirect feedback requires students to identify and correct their own mistakes based on cues provided in the feedback. Feedback can include questions, informative elements, or praise (Cho et al., 2006; Nelson & Schunn, 2009; Peterson et al., 2004; Shute, 2008). Feedback level refers to the teacher’s focus when providing feedback, which can be categorized into five areas: vocabulary, grammar, mechanics, organization, and content (Biber et al., 2011). In their study on the effects of automated assessment and teacher feedback on student motivation and writing quality, Wilson and Czik (2016) classified feedback types as directive, questioning, informative, or praising, and feedback levels into lower-order writing skills (spelling, capitalization, punctuation, sentence structure, grammar, formatting, and word choice) and higher-order writing skills (ideas and elaboration, organization, style, and self-reflection).
Studies at the K–12 level have shown that teacher feedback plays a significant role in improving students’ writing skills. However, its effectiveness varies depending on the type and level of feedback. Direct feedback is generally preferred by both students and teachers (Aridah et al., 2017) as it enhances clarity and comprehensibility (Wahyudi et al., 2024). Notably, teachers often provide more feedback than students expect (Aridah et al., 2017), which can sometimes lead to confusion and anxiety (Wahyudi et al., 2024). Teachers tend to focus on standardizing the written text rather than improving its content or organization (Matsumura et al., 2008). Regardless of the type and level, feedback involves corrections, and students may respond by asking questions, accepting, or ignoring the feedback (Rismawati & Hikmat, 2018). To maximize the effectiveness of feedback in writing instruction, teachers should carefully consider both the content and delivery of their feedback (Wahyudi et al., 2024).
In conclusion, the method and delivery of feedback play a critical role in enhancing students’ writing performance and quality. Therefore, feedback should be systematically analyzed in terms of its source and effectiveness.
Studies on Feedback
Feedback in writing instruction is typically analyzed from four main perspectives: teacher feedback, peer feedback, self-feedback, or computer-assisted feedback (Dawson et al., 2023; Loncar et al., 2023). The following section outlines key studies associated with each approach.
Teacher feedback plays a crucial role in enhancing students’ writing skills, offering revision opportunities that help them grow into proficient writers (Jerry et al., 2013). Both teacher and peer feedback contribute positively to students’ writing performance and their attitudes toward the writing process (Ghani & Asgher, 2012). Teacher feedback can be either direct or indirect, with its effectiveness depending on how well it addresses student needs and classroom realities (Srichanyachon, 2012). However, teachers sometimes ignore students’ feedback preferences and perspectives, which can hinder the learning process (Raihany, 2014). To maximize the benefits of feedback, teachers should provide clear explanations and incorporate positive comments to enhance student motivation (Srichanyachon, 2012). Teacher feedback can also be complemented by feedback from peers, which offers immediate clarification and alternative perspectives, fostering genuine communication among students (Ghani & Asgher, 2012).
Recent research on teacher feedback in writing instruction highlights its importance in the cognitive, affective, and psychomotor dimensions of writing. Mamad and Vigh’s (2024) study examined teacher and student perceptions and practices regarding written feedback, revealing a need for further research on the relationships and differences between these perspectives. One identified gap involves the disconnect between fieldwork and theoretical knowledge in preservice writing teacher training (Peck & Kavanagh, 2024). In second-language writing, feedback from multiple sources has shown the greatest impact on student motivation (Cen & Zheng, 2024). For university entrants, both direct and indirect corrective feedback enhances writing performance, with direct feedback combined with metalinguistic explanations proving particularly beneficial (Wondim et al., 2024). These studies highlight the complex nature of teacher feedback in writing instruction and its potential to enhance student achievement at various levels.
In K–12 writing education, peer feedback has shown more positive outcomes than teacher feedback in some cases, and can be as effective as teacher feedback in improving students’ writing performance (Cui et al., 2021; Gielen et al., 2010). When combined with teacher feedback, peer feedback can significantly enhance the quality of writing across multiple drafts (Pojslová, 2022). However, students often require structured guidance to provide high-quality feedback. Teaching guides and examples can improve the comprehensiveness of peer feedback, but students may still struggle to offer actionable suggestions (Gyamfi et al., 2024). This raises questions regarding the multisource functionality of peer feedback. A meta-analysis study found that multisource feedback had the most substantial motivational effect on L2 writing, followed by single-source feedback, including peer feedback (Cen & Zheng, 2024). In practice, peer review has been shown to significantly improve students’ descriptive writing scores compared to those who did not receive peer feedback (Angraini, 2023). While these studies highlight the potential of peer feedback in writing instruction, they also emphasize the need for structured guidance to maximize its effectiveness.
Self-feedback in writing instruction has been examined alongside peer and teacher feedback, revealing significant effects on student outcomes. While self-feedback can contribute to writing development, peer and teacher feedback have proven more effective in fostering collaborative writing skills. However, when combined with peer feedback, self-feedback has shown notable benefits in promoting critical thinking and increasing both behavioral and emotional engagement in learning (Zou et al., 2022). Research suggests that the application of feedback, rather than the quantity of feedback, plays a more significant role in supporting writing development (Lu et al., 2021).
Research on computer-mediated feedback in writing instruction has demonstrated its effectiveness in both feedback quality and teacher workload reduction. AWE systems can provide effective feedback while alleviating some of the burden on teachers (Fan & Ma, 2022). Wilson and Roscoe (2020) found that AWE feedback increased students’ writing self-efficacy and performance on language skill assessments, while Sarı & Han (2022) found that combined automated tutor feedback was as effective as full tutor feedback in improving analytical writing scores and more effective in reducing grammatical and mechanical errors. However, feedback’s effectiveness may depend on factors such as feedback type and students’ analytical language abilities. For example, in a group receiving direct metalinguistic feedback, language analytic ability exhibited a stronger correlation with acquisition compared to a group receiving only direct feedback (Sheen, 2007). These studies suggest that when implemented appropriately, AWE systems can serve as valuable tools in writing instruction.
Recent studies have obtained varying results regarding the effectiveness of AWF. A systematic review of 83 studies examining AWF’s performance, students’ perceptions, usage, engagement, and the factors influencing these aspects found that AWF has both positive, negative, and neutral effects on advanced-level language and writing courses (Shi & Aryadoust, 2024). A meta-analysis of 20 studies found that AWF has a moderate overall effect on students’ writing performance (g = 0.55) but also highlighted significant heterogeneity in the results (Fleckenstein et al., 2023). In the context of open and distance English language instruction, a 9-week quasi-experimental study investigated the impact of AWF. During the first 3 weeks, students received feedback from an English instructor on their writing tasks, and in the last 3 weeks, they received feedback from an automated system. Pre- and posttest results indicated that both teacher and automated feedback contributed to improvements in academic writing proficiency, with students in the teacher feedback group achieving slightly higher writing performance scores (Taskıran & Göksel, 2022). These findings suggest that AWF can serve as a valuable tool in writing instruction, potentially reducing teacher workload while providing students with timely and personalized feedback.
The literature review highlights the significant role of feedback in writing education and its positive impact on student achievement. As technological advancements and evolving educational paradigms shape feedback methods, alternative approaches like GenAI–based applications continue to emerge. Given these developments, the integration of diverse feedback mechanisms remains crucial for optimizing writing instruction.
Methodology
This study aims to examine the impact of feedback provided by a teacher and a chatbot on the informative texts written by fifth-grade students in Turkish language courses. Specifically, the study investigates how feedback influences students’ writing performance and analyzes the content of the feedback.
To achieve this, the study adopts a convergent parallel mixed-methods design, which involves collecting, analyzing, and integrating both quantitative and qualitative data to provide a comprehensive understanding of the research problem. This approach strengthens the study by leveraging the advantages of both data types to obtain reliable and valid results. In this mixed-methods design, quantitative and qualitative data are collected simultaneously but analyzed separately, and the findings are later compared and synthesized (Creswell, 2020).
For the quantitative component, the study employs a quasi-experimental design. Compared to traditional experimental methods, quasi-experiments are better suited for educational environments, where naturally existing classroom groups are used instead of artificially created groups (Baştürk, 2014; Eckhardt & Ermann, 1977). Since forming artificial experimental groups is not feasible in school settings, a pretest, posttest, and follow-up test design was implemented using two intact groups: One group was designated as the experimental group, which received chatbot feedback. The other was the control group, which received teacher feedback. Both groups underwent pretests to ensure comparability before the intervention. The experimental and control groups were then randomly assigned (Creswell, 2020; Patton, 2002).
Qualitative Data Collection
To analyze the qualitative aspect, a content analysis method was applied to the feedback given by both the teacher and the chatbot on students’ texts. Content analysis systematically processes written data by categorizing, coding, summarizing, and reporting it (Cohen et al., 2007; Flick, 1998; Mayring, 2004). This approach enables a deeper understanding of the nature, similarities, and differences in the feedback provided by the teacher and the chatbot.
Study Group
In experimental studies, random assignment of participants to groups is ideal. However, when random assignment is not feasible, researchers may use preexisting groups that are relatively similar in key characteristics (Büyüköztürk et al., 2023). For this study, the sample consists of fifth-grade students from a public middle school in Gaziantep, Turkey. Both groups were selected from the same school to minimize external variables. After obtaining the necessary ethical approval, a pretest was administered to both groups to determine whether there were any significant initial differences in writing skills. The pretest revealed no significant differences between the groups [t: –0.824, p: 0.414, p > 0.05]. Following the pretest, participants were randomly assigned to either the experimental (chatbot feedback) or control (teacher feedback) group. This quasi-experimental approach ensures that comparisons between the groups remain valid and reliable, maintaining the integrity of the research findings.
Data Collection Process
This study first examined research on GenAI and feedback. A literature review indicated that no chatbot had been used to provide feedback on informative texts written by fifth-grade students in Turkish language courses. Therefore, a feedback chatbot was developed using the Claude-3-Haiku foundational model within the Poe AI platform, which enables users to connect with various GenAI tools and create customized chatbots for free. (For the chatbot’s interface and access link, see Appendix 1). This chatbot was specifically trained to provide formative feedback on informative texts written by fifth-grade students in the Turkish language course. For this purpose, a clear and explicit command (prompt) was designed to guide the chatbot’s responses. Additionally, eight academic sources related to formative and high-quality feedback were uploaded to the system to enhance its knowledge base (Coşkun & Tamer, 2015; Graham et al., 2015; Hamzadayı & Dölek, 2022; Ministry of National Education [MEB], 2019; Sarıkaya, 2023; Ülper, 2019; Wisniewski et al., 2020). Based on the subsequent tests and expert opinions, the final version of the chatbot command was determined (see Appendix 2).
Following the refinement of the chatbot, an 8-week implementation was conducted in a fifth-grade class of a public middle school under the MEB. Before the implementation, both the experimental and control groups were assigned a writing task based on a theme from the MEB Turkish language textbook. Students were given two class periods to write an informative text on the selected topic. The texts were evaluated using the Informative Text Writing Achievement Scale developed by Hamzadayı and Dölek (2022). For 8 weeks, both groups were assigned the same topics, selected based on the theme and content of the textbook. Students were required to write an informative text on the given topic within 2 class hours. In the control group, the teacher provided written feedback on these texts, whereas in the experimental group, written feedback was provided by the chatbot developed by the researcher.
At the end of the 8-week period, students in both groups were once again asked to write an informative text within 2 class hours on the same topic assigned before the intervention. These texts were reassessed using the same evaluation scale applied at the beginning of the study. To assess retention, 2 months after the completion of the experimental process, students were again asked to write an informative text on the same topic within 2 class hours. These texts were also evaluated using the same scale. The list of topics assigned to students during the writing instruction process is provided in Appendix 3.
Data Collection Tools
1. Informative Text Writing Achievement Assessment Scale (for Quantitative Data Collection)
This scale was developed by Hamzadayı and Dölek (2022) to assess the informative text writing performance of middle school, high school, and university students. To determine the content validity index (CVI) of the scale items, expert opinions from 10 specialists were obtained. To establish the reliability of the scale using Kendall’s coefficient of concordance (W), informative texts from a total of 63 students (21 middle school, 21 high school, and 21 university) were collected and evaluated by three raters. The analysis revealed that the CVI values of the scale items and the inter-rater Kendall’s W coefficient were greater than 0.80. These findings indicate that the Informative Text Writing Achievement Rubric, consisting of five sub-dimensions—format, content development, content organization, word choice, and grammar—and 22 items, is both valid and reliable. For this study, the first two optional items under the format dimension were excluded, as they did not align with the research objective (see Appendix 4).
2. Written Feedback Provided by the Teacher and Chatbot (for Qualitative Data Collection)
A total of 100 written feedback instances (50 from the teacher and 50 from the chatbot) provided to fifth-grade students over 8 weeks were used as qualitative data sources for this research.
Data Analysis
Normality Test
The normality distribution of the data was first examined, using the Shapiro–Wilk test due to the sample size. The Shapiro–Wilk test is one of the most powerful normality tests, particularly for samples of less than 50 (Büyüköztürk, 2007; Razali & Wah, 2011).
As the p-values for the pretest and posttest scores of both the experimental and control groups were greater than 0.05, it was determined that the scores did not show a significant (extreme) deviation from a normal distribution. This indicates that statistical tests requiring the normality assumption can be applied. Additionally, skewness values within the range of ±1 and kurtosis values between –2 and +2 (George & Mallery, 2010) further confirmed that the pretest and posttest scores followed a normal distribution. When the data conform to a normal distribution, parametric tests—although requiring certain assumptions about distribution—are more robust in accurately representing central tendency and distribution properties (Field, 2013; Tabachnick & Fidell, 2013). However, when examining the follow-up test, the Shapiro–Wilk p-value for both groups was found to be less than 0.05. Despite the skewness and kurtosis values, the p-value of the follow-up test required the use of a nonparametric test.
Tests Used in Data Analysis
Depending on the normal distribution of the data:
– A paired samples t-test was used for within-group pretest and posttest differences.
– An independent samples t-test was used to compare pretest and posttest differences between the two groups.
– A Friedman test was used to compare repeated measures of pretest, posttest, and follow-up test scores.
Cohen’s d and r Effect Size
Since the data followed a normal distribution, Cohen’s d was used to determine the effect size of AI-supported feedback on students’ informative writing performance scores. Effect size provides a practical measure that complements statistical significance. It also helps assess how meaningful the effect is in real-world applications. Unlike p-values, which indicate whether an effect is statistically significant, Cohen’s d expresses the magnitude of the effect using a standardized measure (Cohen, 1988; Ellis, 2010; Sullivan & Feinn, 2012). The Wilcoxon signed-rank test effect size was calculated by dividing the Z statistic by the square root of the total sample size (Field, 2013), and Cohen’s classification (Cohen, 1988) was used to interpret the effect size.
Kendall’s W
Kendall’s W coefficient of concordance is a nonparametric statistical method that measures the agreement level among multiple raters by ranking their evaluations of a group. This coefficient specifically works with ranking data and ranges between 0 and 1: W = 0 indicates no agreement, while W = 1 represents perfect agreement. This method evaluates how consistently different raters rank the same sample set (Kendall, 1938). Kendall’s W is a nonparametric test, meaning that it does not require a normal distribution, making it a flexible statistical tool (Field, 2013). In this study, Kendall’s W coefficient was calculated using scores obtained from four main criteria and 20 sub-criteria for evaluating informative texts. The coefficient was found to be 0.83, indicating that the agreement among raters was sufficiently high (Howell, 2002).
Qualitative Data Analysis
AI and teacher feedback texts were analyzed using systematic sampling to ensure representativeness and comparability. Through maximum variation sampling (Teddlie & Yu, 2007), every fourth text was selected, resulting in a total of 100 texts (50 AI, 50 teacher).
Both researchers divided each text into idea units according to feedback types identified in Cho et al. (2006), Wilson and Czik (2016), and Guo and Wang (2024). To ensure coding agreement, a coding scheme was developed (see Appendix 5), and 12 texts were analyzed together by the researchers. After discussing and resolving coding disagreements, the remaining 100 texts were categorized under two main themes: feedback types and written expression structure. Sample data related to the feedback types in the coding scheme were presented in the AI-generated and teacher-generated feedback columns, as well as in the researcher notes (see Appendix 5 and Appendix 8). Students were coded as Participant 1, Participant 2, and so on. The coding scheme included five types of feedback: informative, directive, questioning, praise, and summarizing. Additionally, each feedback type was classified under one of three main categories: content, organization, or language expression. The obtained data were analyzed using frequency and percentage distributions. In qualitative data analysis, ensuring the reliability of findings and maintaining inter-coder consistency is essential. According to the Miles and Huberman model, inter-coder agreement should be at least 80% (Miles & Huberman, 1994; Patton, 2002). In this study, inter-coder agreement was calculated at 83%, confirming that the qualitative data analysis was sufficiently reliable.
Findings
This section presents the study’s findings in line with the research questions. The first two subsections report statistical results on the effects and retention of AI- and teacher-provided formative feedback on students’ informative writing. The third subsection analyzes similarities and differences in feedback content.
1. Comparing Effects of Chatbot and Teacher Feedback on Informative Text Writing Performance
The comparison of the effect of AI-supported feedback between groups and within each group is presented in Tables 1 and 2.
Table 1
Comparison between groups.
| GROUP | N | MEAN | S | df | t | p | d | |
|---|---|---|---|---|---|---|---|---|
| Post- test | Artificial intelligence | 26 | 79,1923 | 14,13370 | 50 | –1,428 | 0,160 | 0,396 |
| Teacher | 26 | 74,3846 | 9,75326 |
Table 2
In-group comparison.
| PRE-TEST-POST-TEST | N | S | df | t | p | d | ||
|---|---|---|---|---|---|---|---|---|
| Artificial intelligence | Pre-test | 26 | 45,4231 | 13,24590 | 25 | –13,712 | 0,000 | 2,50 |
| Post-test | 26 | 79,1923 | 14,13370 | |||||
| Teacher | Pre-test | 26 | 42,5000 | 12,32315 | 25 | –12,185 | 0,000 | 2,49 |
| Post- test | 26 | 74,3846 | 9,75326 |
Table 1 shows that there was no statistically significant difference between the posttest scores of the AI and teacher feedback groups [t = –1.428, p > 0.05]. Although the mean posttest score of the AI group was slightly higher than that of the teacher feedback group, the magnitude of the difference between the two groups was also small (d = 0.396). This suggests that while there is no statistical significance, there may still be a meaningful effect.
A statistically significant difference was observed between the pretest and posttest scores of the group that received AI-supported feedback [t = –13.712, p < 0.05]. Additionally, since the Cohen’s d value exceeded 1.0, it can be stated that AI-supported feedback had a very large effect (d = 2.50) on informative text writing performance. A statistically significant difference was also found between the pretest and posttest scores of the group that received teacher feedback [t = –12.185, p < 0.05], with the same level of effect size (d = 2.49). When analyzing the difference between the pretest and posttest in sub-variables, it was found that the mean of each variable increased in the posttest, and a significant difference (p = 0.000) with a large effect was observed (see Appendix 6 and Table 7).
2. Comparing the Lasting Effects of Chatbot and Teacher Feedback on Informative Text Writing
The difference in retention between the AI feedback and teacher feedback groups is another subproblem.
When examining the rank averages of the AI feedback group, an increase was observed from the pretest to the posttest, but a decrease occurred in the follow-up test (Table 3). Additionally, based on the p-value, it was determined that there was no statistically significant difference between the measurements [p = 0.168, p > 0.05]. By contrast, in the teacher feedback group, the success average increased consistently from the pretest to the follow-up test, and a statistically significant difference was found [p = 0.000, p < 0.05]. Beyond the difference itself, the effect also varied: AI-supported feedback showed a small effect on informative text writing performance over time (W = 0.0686), whereas teacher-supported feedback had a large effect (W = 0.7666).
Table 3
Change over time.
| N | MEAN RANK | CHI-SQUARE | df | p | W | ||
|---|---|---|---|---|---|---|---|
| Artificial intelligence | Pre-test | 26 | 1,90 | 3,571 | 2 | 0,168 | 0,0686 |
| Post- test | 26 | 2,29 | |||||
| Follow-up test | 26 | 1,81 | |||||
| Teacher | Pre-test | 26 | 1,40 | 39,864 | 2 | 0,000 | 0,7666 |
| Post- test | 26 | 1,60 | |||||
| Follow-up test | 26 | 3,00 |
The Wilcoxon signed-rank test conducted between the posttest and the follow-up test showed that while all participants in the teacher-supported instruction group showed a positive increase from the posttest to the follow-up test, in the AI-supported instruction group, a positive increase in informative text writing performance was observed in seven participants, whereas a decrease was observed in 18 participants, and one participant remained the same (Table 4). Therefore, although a significant difference was found between the posttest and the follow-up test in the teacher-supported group (p = 0.000), no significant difference was observed in the AI-supported experimental group (p = 0.332). Similarly, while a small effect size was found in the experimental group (r = 0.199), the effect size in the teacher-supported group was determined to be large (r = –0.875).
Table 4
Post-test-follow-up test difference.
| N | MEAN RANK | SUM OF RANKS | Z | p | r | ||
|---|---|---|---|---|---|---|---|
| Artificial intelligence | Negative Ranks | 18 | 11,03 | 198,50 | –,970 | ,332 | –0.199 |
| Positive Ranks | 7 | 18,07 | 126,50 | ||||
| Ties | 1 | – | – | ||||
| Teacher | Negative Ranks | 0 | ,00 | ,00 | –4,459 | 0,000 | –0.875 |
| Positive Ranks | 26 | 13,50 | 351,00 | ||||
| Ties | 0 | – |
A comparison of sub-variables revealed that in the group receiving AI feedback, each variable increased from the pretest to the follow-up test, while in the teacher-supported group, there was an increase in the mean rank for every variable except content creation (Table 5). In both groups, a significant difference was observed between the pretest, posttest, and follow-up test for content creation, content organization, word usage, and grammar (p = 0.000), with a large effect size (W = 0.639 to 0.871). When comparing the posttest and follow-up test (see Appendix 6 and Table 8), no significant difference was found in the AI-supported group for content creation (p = 0.119) or content organization (p = 0.283), while a significant difference was observed in the teacher-supported group (content creation p = 0.002; content organization p = 0.016), with a large effect size. While a significant difference was found in word usage in both groups, the effect size was also deemed equal. No significant difference in grammar was found in either group, and the effect size was small. In sum, differences between the posttest and follow-up test were observed between the two groups; however, these differences did not affect the values between the pretest, posttest, and follow-up test.
Table 5
Change over time in terms of sub-dimensions.
| N | MEAN RANK | CHI-SQUARE | df | p | W | |||
|---|---|---|---|---|---|---|---|---|
| Content creation | Artificial intelligence | Pre-test | 26 | 1,00 | 43,429 | 2 | 0,000 | 0,835 |
| Post- test | 26 | 2,31 | ||||||
| Follow-up test | 26 | 2,69 | ||||||
| Teacher | Pre-test | 26 | 1,10 | 34,940 | 2 | 0,000 | 0,672 | |
| Post- test | 26 | 2,63 | ||||||
| Follow-up test | 26 | 2,27 | ||||||
| Content organisation | Artificial intelligence | Pre-test | 26 | 1,00 | 42,716 | 2 | 0,000 | 0,821 |
| Post- test | 26 | 2,48 | ||||||
| Follow-up test | 26 | 2,52 | ||||||
| Teacher | Pre-test | 26 | 1,04 | 40,735 | 2 | 0,000 | 0,783 | |
| Post- test | 26 | 2,27 | ||||||
| Follow-up test | 26 | 2,69 | ||||||
| Word usage | Artificial intelligence | Pre-test | 26 | 1,0 | 45,277 | 2 | 0,000 | 0,871 |
| Post- test | 26 | 2,31 | ||||||
| Follow-up test | 26 | 2,69 | ||||||
| Teacher | Pre-test | 26 | 1,15 | 33,250 | 2 | 0,000 | 0,639 | |
| Post- test | 26 | 2,19 | ||||||
| Follow-up test | 26 | 2,65 | ||||||
| Grammar | Artificial intelligence | Pre-test | 26 | 1,00 | 43,583 | 2 | 0,000 | 0,838 |
| Post- test | 26 | 2,35 | ||||||
| Follow-up test | 26 | 2,65 | ||||||
| Teacher | Pre-test | 26 | 1,08 | 40,696 | 2 | 0,000 | 0,783 | |
| Post- test | 26 | 2,23 | ||||||
| Follow-up test | 26 | 2,69 |
3. Comparing Similarities and Differences in Chatbot and Teacher Feedback Content
Teachers and AI provide different types of feedback and vary in the components of written expression, as seen in Table 6.
Table 6
Differences in feedback rate.
| TYPE OF FEEDBACK | ARTIFICIAL INTELLIGENCE | TEACHER | COMPONENTS OF WRITTEN EXPRESSION | ARTIFICIAL INTELLIGENCE | TEACHER | ||||
|---|---|---|---|---|---|---|---|---|---|
| f | % | f | % | f | % | f | % | ||
| Router | 81 | 26,82 | 34 | 14,66 | Language/Spelling and Grammar | 54 | 21,52 | 62 | 33,16 |
| Informative | 140 | 46,36 | 126 | 54,31 | Content | 142 | 56,57 | 64 | 34,22 |
| Question | 0 | 0 | 7 | 3.01 | Organisation | 55 | 21,91 | 61 | 32,62 |
| Praise | 63 | 20,86 | 60 | 25,86 | Total | 251 | 100 | 187 | 100 |
| Summariser | 18 | 5,96 | 5 | 2,16 | |||||
| Total | 302 | 100 | 232 | 100 | |||||
| PearsonChi-Square | 25,631 | PearsonChi-Square | 3,225 | ||||||
| p | 0,000 | p | 0,199 | ||||||
| Cramer’s V | 0,219 | Cramer’s V | 0,082 | ||||||
In terms of feedback type, informative feedback was the most frequently provided type in both groups. However, in the AI group, directive feedback ranked second, followed by praise and summarizing feedback. In contrast, praise was the second most common type in the teacher group. This indicates that while AI prioritizes providing information along with suggestions for changes, teacher feedback tends to motivate participants through praise after providing information. In terms of written expression components, AI focused more on content, whereas the teacher balanced all three components. AI tended to provide content-related feedback by combining information, guidance, and partial summarization. In contrast, teacher feedback was more likely to provide information on each component, offer praise related to these components, and occasionally prompt reflection through questions.
Analysis revealed a significant difference between the groups in feedback type (p = 0.000), with a moderate effect size. However, there was no significant difference between the two groups in written expression components, meaning the effect was very low. Although the teacher primarily provided informative feedback on content creation and content organization, AI’s content-related feedback was more closely associated with praise and summarization. While this led to a significant difference in the posttest, it did not have a significant impact on the follow-up test.
Discussion
According to the quantitative findings of this study, formative feedback provided by both the teacher and the chatbot had a similar effect on the informative text writing performance of fifth-grade students. A significant difference was observed between the pretest and posttest scores of both groups, suggesting that feedback from the chatbot was as effective as that provided by the teacher. This finding is supported by several studies (Alsofyani & Barzanji, 2025; Khan et al., 2025; Kim et al., 2024). However, a study by Steiss et al. (2024) indicated that teacher feedback was more effective. While the posttest results showed differences, the fact that teacher feedback resulted in a significant difference in the retention test, whereas AI feedback did not, aligns with the findings of Steiss et al. (2024). In their study, Steiss et al. (2024) examined the quality of formative feedback provided by both human evaluators and ChatGPT-3.5 on the argument-based writing of 200 randomly selected students from grades 6 to 12 across 26 different regions in Southern California. Feedback quality was assessed according to criterion-based evaluation, clarity of revision instructions, accuracy, prioritization of essential features, and the use of a supportive tone. The results indicated that human evaluators provided higher-quality feedback than ChatGPT in all categories except criterion-based feedback. This suggests that the higher quality of teacher feedback had a greater impact on retention. Thus, the short-term performance gains observed when GenAI tools are used in educational settings—which could be described as a flash-in-the-pan effect—should not lead educators to become overly reliant on such tools. Instead, it is crucial to critically evaluate, observe, and carefully integrate these technologies into the learning process with an awareness of their long-term implications. Moreover, given that GenAI technologies and their applications are relatively new, future research with clearly defined standards and improved AI literacy may yield different results.
GenAI tools clearly hold potential for providing feedback on students’ texts while teaching writing (Tlili et al., 2023). The findings of this study show that one such potential is that GenAI chatbots can serve as an effective and innovative tool for providing feedback in Turkish writing education. However, challenges such as limited access to GenAI tools, teachers’ lack of necessary knowledge and skills to integrate these tools into the educational process, and negative attitudes toward AI adoption (Abdelhalim, 2023) hinder the realization of their full potential. This risk may lead to a widening learning gap between students who utilize AI tools and those who do not. Additionally, the pricing of GenAI tools, which varies based on their capabilities, could contribute to educational inequality between students with access to these tools and those without (McGee & Sadler, 2024). Even with equal conditions, factors such as the model of the GenAI tool, the content and quality of prompts given to these models, and how the AI processes the input may affect the outcomes.
With the increasing prevalence of GenAI tools and their growing integration into education, questions have arisen regarding whether AI can replace teachers. According to the quantitative results of this study, the feedback provided by the chatbot fulfilled the function of teacher feedback. Considering the time spent by the teacher per student for feedback (12 minutes on average) compared to the time spent by the chatbot (4 minutes on average; see Appendix 9), GenAI tools can play the role of an assistant teacher, reducing teachers’ cognitive workload in Turkish writing education. Many studies support our finding regarding AI’s potential to reduce teachers’ workload (Fan & Ma, 2022; Guo & Wang, 2024; Tossell et al., 2024). As a result, the teacher’s predominant role as a source of knowledge may shift toward the affective domain. In short, rather than replacing teachers, GenAI may serve as a transformative technology that enhances teachers’ efficiency and speed, modeling an education system where teacher–AI interaction is optimized (Karatas & Yüce, 2024). Furthermore, recent studies indicate that students tend to prefer a combined feedback approach that integrates feedback from teachers, peers, and generative AI tools (Zeevy-Solovey, 2024).
One of the most prominent qualitative findings of this study is that the chatbot was able to provide feedback similar to that of a teacher, coming close to replicating teacher feedback. Indeed, in their study comparing the quality and accuracy of teacher and ChatGPT feedback, Steiss et al. (2024) stated that what was surprising was not that the teacher provided higher-quality feedback than ChatGPT but that ChatGPT was capable of delivering teacher-like feedback. However, analysis of the feedback content in this study revealed that affective expressions were more prominent in teacher feedback. The mnemonic, emphatic, stimulating, and associative informative aspects of teacher feedback, which address the individuality of the student, were lacking in AI-generated feedback, although this may change as AI advances. The teacher autonomy observed in this study, along with the resulting emotional engagement and reactions (such as attitude, motivation, frustration, etc.) of students during the feedback process, may have influenced their writing achievement scores. This finding aligns with several studies (Han & Hyland, 2015; Jansen et al., 2025; Mahfoodh, 2017; Máñez et al., 2024; Renata et al., 2024; Zheng & Yu, 2018).
An analysis of the types of feedback provided on students’ texts showed that both the teacher and the GenAI chatbot systematically used directive, informative-stimulating, praise-encouraging, and summary feedback. One of the most critical findings was that while the teacher provided question-based feedback, the GenAI chatbot did not use this type of feedback at all. This suggests that the teacher encouraged students to think critically by asking questions. In their study comparing teacher and ChatGPT feedback, Guo and Wang (2024) found that teachers tend to not only provide informative feedback but also ask questions to stimulate students’ thinking. Additionally, in this study, when the percentage distribution of feedback types provided by the chatbot and the teacher was examined, a partially complementary pattern emerged. For example, AI provided directive feedback 26.82% of the time, while the teacher provided it 14.66% of the time. In contrast, the teacher provided praise feedback 25.86% of the time, compared to 20.86% for AI. This pattern aligns with the findings of Yao et al. (2025), which examined AI-driven chatbot applications for middle school English teachers providing feedback on student writing.
An analysis of feedback focus areas revealed that both teacher and AI feedback primarily emphasized grammar and spelling rules, content, and organization. A notable difference was that the teacher provided fewer but more balanced feedback points, whereas the GenAI chatbot focused more on content. This suggests that teachers provide more specific feedback, taking into account students’ previous mistakes and deficiencies, while the AI chatbot offers more general and inclusive feedback. This finding is supported by the results of a study conducted by Castro et al. (2025), which examined the use of CyberScholar—a GenAI assistant designed to provide writing feedback in K–12 education. Furthermore, the smaller quantity of feedback provided by the teacher may be due to the cognitive demands of assessing student writing. However, since this did not negatively affect students’ writing achievement, it may suggest that the teacher’s feedback was more effective and enduring. This is consistent with Lee’s (2019) “less is more” principle, which advocates for focusing on students’ specific weaknesses and encouraging critical thinking rather than overwhelming them with excessively detailed corrective feedback.
In conclusion, based on both the quantitative and qualitative findings of this study, GenAI tools may serve as a complementary, innovative, and alternative feedback tool that reduces teachers’ cognitive workload during the feedback stage of writing instruction, freeing time for other essential writing processes such as preparation, drafting, planning, and publishing. However, a limitation of using GenAI tools in writing education is that neither teachers nor students currently possess sufficient AI literacy. Therefore, training in AI literacy is recommended for both teachers and students (Tate et al., 2023; van Wyk, 2025).
Conclusion, Implications and Suggestions
The primary aim of this study was to compare, both quantitatively and qualitatively, the impact of teacher versus GenAI-based chatbot feedback on the development of middle school students’ writing skills. The quantitative findings showed no statistically significant difference in students’ informative writing performance scores between the AI feedback and teacher feedback groups [t = –1.428, p > 0.05]. In terms of retention, a comparison of follow-up test scores revealed a decrease in the AI group and a significant increase in the teacher group. Furthermore, beyond the observed difference, the teacher group demonstrated a greater long-term effect compared to the AI group. The qualitative findings revealed a meaningful difference in both feedback types and components of written expression. While the chatbot tended to focus more heavily on content and provided more detailed feedback, the teacher offered more balanced feedback. Additionally, the teacher utilized interrogative feedback—a type not observed in the AI responses.
These findings suggest that GenAI tools hold significant potential for providing feedback (Cope & Kalantzis, 2023; Mollick & Mollick, 2023; Sabzalieva & Valentini, 2023). The first implication is that GenAI-supported tools may be effectively utilized to provide feedback on students’ texts in the development of writing skills at the middle school level. These tools can be integrated into the writing process either directly or indirectly. Secondly, the study’s retention results indicate that AI tools should not be viewed as alternatives to teachers but rather as innovative and complementary sources of feedback. Thirdly, the fact that the teacher provided less feedback than the AI yet achieved comparable effects in the posttest and a greater effect in the follow-up test may suggest that, despite AI’s ability to fulfill the formal and functional aspects of feedback delivery, it is currently unable to replicate the affective dimension of the teacher–student relationship. Moreover, feedback should not be regarded as a simple task of automated response based on predefined criteria but rather as an essential component of the writing process involving affective dimensions (Corbin et al., 2025).
In conclusion, this study addresses a gap in the literature concerning the use of GenAI tools to provide feedback on student writing. However, further research is needed to fully understand both the potential and the limitations of these tools. Future studies could examine how feedback practices differ between teachers with AI literacy and those without, and how these differences impact students’ writing performance. Additionally, future research could compare the accuracy and content of feedback provided by general-purpose GenAI tools (e.g., ChatGPT, Claude, Gemini, DeepSeek) with feedback provided by GenAI chatbots specifically trained for feedback purposes. It would also be valuable to explore the writing development of students who use GenAI tools at home for feedback versus those who rely solely on teacher feedback. Also worth investigating are the effects of teacher and GenAI feedback across different genres of writing and various grade levels. Finally, future research could explore students’ perceptions of GenAI feedback and teachers’ experiences with these tools.
Limitations
The findings of this study may not be universally applicable due to the small sample size, the focus on fifth-grade students, the application to teaching informative texts, and the use of a limited dataset to train the AI chatbot. These limitations should be taken into consideration when analyzing the findings of the study.
Data Accessibility Statement
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
Additional File
The additional file for this article can be found as follows:
Ethics and Consent
The conduct of this research was approved on ethical grounds by the Ethics Committee of Social and Human Sciences at Gaziantep University, based on Decision No. 22 taken at Meeting No. 08 on April 3, 2024. Ethical principles of scientific research were followed at all stages of the study.
Competing Interests
The authors have no competing interests to declare.
Author Contributions (CRediT)
Mehmet Ali Baz; Conceptualization, investigation, chatbot training, validation, writing—original draft preparation, writing—review and editing. All authors have read and agreed to the published version of the manuscript OR The author has read and agreed to the published version of the manuscript. Sevil Hasırcı Aksoy: Conceptualization, methodology, formal analysis, data analysis and findings, data curation, validation, visualization, writing—review and editing, supervision.
Sustainable Development Goals (SDGs)
This study is linked to the following SDG(s): Quality education (SDG 4).
Author Notes
In line with the Academic Integrity and Transparency in AI-assisted Research and Specification Framework (Bozkurt, 2024), the authors of this study acknowledge the responsible and transparent use of AI tools at various stages of the research process. During the idea development phase, the basic free versions of Elicit AI and Consensus AI were utilized to support the literature review. The preliminary outputs generated by these tools were critically evaluated and subsequently refined solely by the human authors in accordance with ethical and academic standards. Potential biases inherent in AI-generated content were also assessed and addressed. All final conceptual frameworks and scholarly arguments are the sole responsibility of the human authors. In the language translation and localization phase, Deepl Pro and ChatGPT Plus (versions as of April 2025) were used to translate the manuscript into English. The translated text was reviewed by an expert in English academic writing instruction, then revised and finalized by the human authors. As the final stage, the manuscript underwent professional proofreading to ensure linguistic precision and academic consistency. For the data analysis, ChatGPT Plus was employed to calculate and verify Cohen’s d and r effect sizes. These analyses were reviewed, refined, and finalized by the human authors to ensure accurate representation and reporting of the research data. The authors take full responsibility for the validity and integrity of the reported findings.