Introduction
We evaluated the effectiveness of generative AI course assistants for a pilot study to determine their effectiveness within online undergraduate university courses. We utilized multiple measures to get a broad view of the effectiveness of the AI course assistants on the student experience. The measures we chose were grades (Essel et al., 2022; Wu & Zhonggen, 2003), intrinsic motivation to learn (Hartnett et al., 2011), general self-efficacy (Parsakia, 2023), and the students’ feelings of engagement, encouragement, and support (Labadze et al., 2023). We chose these measures because they directly relate to successful student outcomes and can be objectively measured using treatment and control groups.
We established a treatment group where students were granted access to a sophisticated AI course assistant powered by Nectir.io, and a control group, which did not have access to the assistant. This approach allowed us to compare the outcomes of both groups to assess the impact of AI course assistants.
The treatment group consisted of students from two classes of BIBL 230 Biblical Literature Luke/Acts and one class of the PSYC 105 Introduction to Psychology. The control group consisted of students from one BIBL 230 class and one PSYC 105 class. There were 92 participants in total: 52 in the treatment group and 40 in the control group.
The importance of this study lies in its potential to add to the current literature and measure AI course assistants’ abilities to address significant gaps in online education by leveraging AI technology to enhance student learning experiences and outcomes. The implementation of AI course assistants, such as those developed by Nectir.io, aims to provide continuous, personalized support to students, which is particularly crucial in asynchronous learning environments where students often struggle with isolation and lack of immediate feedback (Hanshaw & Miller, 2024). Research has shown that intrinsic motivation and a sense of support are key factors in student success in online courses (Hartnett et al., 2011; Labadze et al., 2023; Richardson et al., 2017). By evaluating the impact of AI course assistants on these factors, this study contributes insights into how educational institutions can utilize AI to foster a more engaging and supportive learning environment, ultimately helping to address some of the issues related to the equity gap for underserved student populations (Sublett, 2020; Williams, 2024).
The equity gap in education often stems from disparities in access to resources, personalized support, and timely feedback, which can hinder the academic success of students from underrepresented and low-income backgrounds (Smith, 2019). AI course assistants have the potential to mitigate these challenges by providing 24/7 availability, personalized learning experiences, and immediate responses to student inquiries, thus leveling the playing field for students who may not have access to traditional support systems (Chen et al., 2022). Moreover, AI-driven insights can help educators identify and address specific needs and learning gaps, offering targeted interventions that support the academic growth of underserved students (Baker & Siemens, 2014). By leveraging these capabilities, educational institutions can create a more equitable learning environment that supports the success of all students, regardless of their socioeconomic background.
It is crucial to underscore that the design of the AI course assistants did not allow the assistants to complete tasks on behalf of the students. These assistants were more advanced than basic chatbots that merely responded to queries. Nectir.io engineered the generative AI course assistants to simulate human interaction, posing questions and engaging in discussions with students. The ultimate goal of these interactions was to prompt students to reflect on their work and strive for improvement.
Literature Review
The use of AI tools is proliferating at a high rate. This is especially true for higher education. The ease of use, access, and intelligence of these tools have led to rapid adoption by students in higher education, giving higher education a unique opportunity to begin reshaping the landscape. Grassini (2023) found that the rapid adoption has sparked considerable interest in these tools due to the humanlike interaction from text generation and their potential to significantly aid students in higher education. A growing body of evidence shows the effectiveness of AI course assistants (Al-Abdullatif, 2023; Deng & Yu, 2023; Essel et al., 2022; Grassini, 2023; Hanshaw & Miller, 2024; Labadze et al., 2023; Parsakia, 2023; Williams, 2024; Wu & Zonggen, 2023).
AI course assistants in education offer promising benefits for grade outcomes and self-efficacy (Al-Abdullatif, 2023). By providing personalized, immediate assistance and fostering emotional confidence, chatbots can enhance students’ learning experiences and academic performance. However, to maximize the potential of these technologies in educational settings, it is essential to address challenges related to reliability, accuracy, and ethical considerations. Future research should continue exploring AI course assistants’ long-term impacts and develop strategies to integrate them effectively into diverse learning environments.
Bozkurt (2023) argued that properly trained iterations of generative AI can learn, unlearn, and relearn language constructs. There has been a shift from organic (human-created) to synthetic (AI-created) content. Generative AI has the potential to revolutionize pedagogical practices and represents a new frontier in the educational sphere. Maphoto et al. (2024) argued that the integration of AI-powered writing tools brings substantial challenges to higher education institutions in the areas of academic misconduct, over-reliance on technology, authenticity issues, and the promotion of passive learning. The research gap is highlighted. There is a need for further research to optimize the balance between human intelligence and generative AI capabilities within pedagogical frameworks.
Bozkurt (2024a) argued that it is critical to broaden the use of AI across all demographics to ensure inclusivity and ultimately to create responsible citizens. He stressed the importance of developing AI literacy for education. In summary, Bozkurt highlighted the increase in everyday use of AI and stressed the importance of integrating generic and generative AI into effective utilization. Bozhurt further stressed the development of educational frameworks to prioritize AI literacy in order to prepare individuals for the evolving technological and educational landscape.
Van den Berg (2024) argued that the use of generative AI tools is changing the traditional fabric of education. He further stressed that the use of generative AI will assist educators to personalize the student learning experience, make education more accessible for all, and make education more engaging and effective overall. The OpenAI language model had one million users within 5-days of being released. The rapid adoption highlights the interest across the world to utilize generative AI tools. In addition, it is prompting larger questions about the roles of human educators and generative AI.
We explored several outcomes and constructs to help further the growing body of research on the effectiveness of AI course assistants. We specifically examined the outcomes by focusing on both grade and percentage scores at the end of the course, as well as intrinsic motivation to learn, general self-efficacy, and a student’s feeling of engagement, encouragement, and support.
Grades
AI course assistants promise to substantially affect student grades. The systematic literature reviews by Deng and Yu (2023) and Labadze et al. (2023) support this assertion. AI course assistants can affect grade outcomes through several methods. They provide real-time assistance and personalized feedback, directly affecting grade outcomes (Hanshaw & Miller, 2024). AI assistants also help with knowledge retention, which refers to the ability of students to remember and apply what they have learned over time (Deng & Yu, 2023; Labadze et al., 2023). This may be the direct reason why they have a medium to high effect on grades. Deng and Yu (2023) found that the consistency and availability of AI assistants also positively affected student grades.
The ability to tailor interactions with AI course assistants to meet the needs of diverse student groups has an effect on student grades as well. Labadze (2023) found that chatbots that provide step-by-step problem-solving assistance have been particularly effective in helping students from disadvantaged backgrounds improve their grades in subjects like mathematics and science. Smith et al. (2024) found that chatbot technology helped students improve in their learning of math and science also. AI course assistants are quickly becoming an effective and necessary part of the student support ecosystem.
Intrinsic motivation to learn
Intrinsic motivation to learn refers to the internal drive to engage in learning activities for the inherent satisfaction and enjoyment derived from the process rather than for external rewards or pressures (Ryan & Deci, 2000). Intrinsic motivation is crucial in educational settings because it fosters deep learning, persistence, and a greater willingness to tackle challenging tasks (Deci et al., 1991). Intrinsic motivation is often connected to curiosity. When students are curious, they are more likely to engage deeply with the material, ask questions, and seek out additional information, all of which are indicative of intrinsic motivation (Schiefele, 1991) a natural curiosity. Without the physical presence of instructors and peers, students must rely more heavily on their internal motivation to stay engaged, complete assignments, and actively participate in the learning process (Hartnett et al., 2011).
Research has shown that intrinsic motivation can lead to better learning outcomes and higher satisfaction in online courses (Lee, 2014). In online learning environments, where students often face challenges such as isolation, lack of immediate feedback, and the temptation of numerous online distractions, fostering intrinsic motivation can be critical in ensuring student success (Richardson et al., 2017). Strategies to enhance intrinsic motivation in online courses include providing meaningful and relevant content, opportunities for self-directed learning, and interactive and engaging activities (Kearsley & Shneiderman, 1998). As educational institutions expand their online offerings, understanding and promoting intrinsic motivation among students will be essential for maintaining high levels of engagement and academic achievement.
The study of chatbots and AI course assistants is an area of research that has had mixed results and remains a rich area of focus. Fidan and Gencel (2022) found that students who had access to a course chatbot had a higher intrinsic motivation score on multiple subfactors of intrinsic motivation when compared to other students who did not utilize a course chatbot. Zhao et al. (2023) found that utilizing chatbot-assisted instructional videos did not significantly affect intrinsic motivation. Parsakia, (2023) argued that chatbots can motivate students by offering a pleasurable and rewarding learning experience, thus enhancing intrinsic motivation.
Recent research has highlighted the potential of chatbots in fostering a sense of belonging in the classroom. A study conducted by Smith et al. (2024) explored the impact of chatbots on students’ perceptions of belonging in an online learning environment. The results revealed that students who interacted with the chatbot reported feeling more connected to their peers and instructors compared to those who did not. The chatbot facilitated personalized interactions, providing timely feedback and support, which contributed to a sense of community and inclusivity among students. This study underscores the importance of leveraging chatbot technology to create a supportive and inclusive classroom environment, ultimately enhancing students’ sense of belonging and engagement in the learning process.
Recent studies have also found the potential to increase student motivation to learn through the use of chatbots. According to the findings of Al-Abdullatif (2023), “… offering a pleasurable, rewarding learning experience, chatbots can motivate students to engage more in learning activities, resulting in a better learning outcome.”
General Self-efficacy
Self-efficacy, defined as an individual’s belief in their ability to perform tasks and achieve goals, is a critical factor influencing academic success. Bandura (1997) posited that self-efficacy affects how people think, behave, and feel, directly impacting their motivation and academic performance. In educational settings, high self-efficacy is associated with greater effort, persistence, and resilience in the face of challenges, which are essential traits for successful learning, especially in online environments where students often work independently (Schunk & Pajares, 2002).
The potential of AI course assistants to enhance self-efficacy among students has been explored in several studies. Research by Schwarzer and Jerusalem (1995) introduced the General Self-Efficacy (GSE) scale, which has been widely used to measure self-efficacy in various contexts, including education. Recent studies have adapted this scale to evaluate the impact of AI interventions on students’ self-efficacy. Ilieva et al. (2023) found that students who interacted with AI chatbots reported higher self-efficacy, as the personalized feedback and support provided by the AI helped them feel more capable and confident in their academic tasks.
Furthermore, Al-Abdullatif (2023) explored the effects of AI chatbots on students’ self-efficacy and found that the interactive and responsive nature of AI tools significantly boosted students’ confidence in their ability to understand and apply course material. The study highlighted that immediate feedback and personalized assistance from AI chatbots reduced students’ anxiety and uncertainty, which are common barriers to high self-efficacy.
However, the impact of AI course assistants on self-efficacy is not uniformly positive across all studies. Labadze et al. (2023) conducted a systematic review and noted that while many studies report positive outcomes, the variability in AI implementation and the context-specific nature of self-efficacy suggest that more research is needed to understand the conditions under which AI interventions are most effective. This underscores the importance of considering the design and deployment of AI tools to maximize their benefits for enhancing self-efficacy among diverse student populations.
Feelings of Engagement, Support, and Encouragement
The integration of AI course assistants in higher education, a relatively new and promising field of research, has shown potential in enhancing various aspects of the student learning experience. This includes feelings of engagement, encouragement, and support. AI course assistants can provide continuous, personalized feedback and support, which is critical in online and asynchronous learning environments where students may feel isolated. Research by Richardson et al. (2017) emphasized the importance of social presence in online learning, noting that increased interaction and support can significantly improve student satisfaction and learning outcomes. AI course assistants, by offering immediate responses and personalized interactions, can help replicate some aspects of this social presence, thus fostering a more engaging and supportive learning environment.
In a study by Ilieva et al. (2023), the practical implications of generative chatbots in higher education were explored, highlighting their potential to enhance student engagement and provide emotional support. The study found that students who interacted with AI chatbots reported higher levels of engagement and felt more encouraged and supported compared to those who did not use such tools. This aligns with findings by Labadze et al. (2023), who conducted a systematic literature review and concluded that AI chatbots could play a significant role in reducing feelings of isolation and promoting a sense of community among students. These studies underscore the potential of AI course assistants to enhance the emotional and psychological aspects of the learning experience, which are crucial for student success.
Despite these positive findings, the impact of AI course assistants on students’ feelings of engagement, encouragement, and support remains an area requiring further research. Williams (2024) noted the ethical considerations and potential biases inherent in AI technologies, emphasizing the need for careful implementation and continuous evaluation to ensure these tools equitably benefit all students. Additionally, the work of Sublett (2020) highlighted the disparities in online learning experiences, particularly for students of color, suggesting that AI course assistants could be a valuable tool in addressing these inequities by providing personalized, round-the-clock support. This study aims to build on this body of research by specifically examining how AI course assistants affect students’ perceptions of engagement, encouragement, and support, thereby contributing to a more comprehensive understanding of their role in enhancing the online learning experience.
The rapid adoption of AI tools in higher education presents a unique opportunity to enhance various aspects of the student learning experience, including grades, intrinsic motivation, self-efficacy, and feelings of engagement, encouragement, and support. The existing literature underscores the potential benefits of AI course assistants in providing personalized, immediate assistance and fostering a supportive learning environment. However, it also highlights the need for further research to address challenges related to these technologies’ reliability, accuracy, and ethical use. By examining the impacts of AI course assistants on multiple dimensions of the student experience, this study aims to contribute to a more comprehensive understanding of their effectiveness and inform best practices for their implementation in diverse educational settings. This research is significant for identifying ways to leverage AI to support all students equitably and enhance their overall academic success and well-being.
Research Question
Does the implementation of AI course assistants in classrooms lead to improvements in student learning outcomes (measured by GPA and final percent score), self-efficacy, intrinsic motivation, and increased self-report scores of student feelings of support, engagement, encouragement?
Purpose
The purpose of this study was to investigate the potential benefits of AI course assistants in undergraduate online courses by examining their impact on a range of student grades and multiple self-reported constructs. By comparing student grades, self-efficacy, intrinsic motivation, feelings of support, engagement, and encouragement, between courses with and without AI assistants, the study will provide valuable insights into which aspects of the learning experience AI assistants enhance.
The multiple data points were selected to create a broader picture of the student experience from an academic and social perspective.
Grades: Grade point average (GPA) and the final percent score achieved in the course were two quantitative measures we used to measure the student’s academic performance. These two measures are universal in academia, and when these outcomes are compared between treatment and control groups, they can indicate if AI assistants are contributing to improved learning outcomes.
Self-Efficacy: This data will reveal if students feel more confident in their abilities after using an AI assistant, which can be a crucial factor in academic success. See Appendix A for specific questions.
Intrinsic Motivation: Understanding if AI assistants foster a love of learning in students can be a significant indicator of their long-term positive impact. See Appendix A for specific questions.
Feelings of engagement, encouragement, and support: Self-report measure taken from the end-of-course survey. The measurement is a 1–5 Likert scale with one indicating the lowest score and five indicating the highest score for the statement. Students respond to this survey during week 8 of their course. See Appendix B for specific Questions.
By combining these data points, we built a comprehensive picture of how AI course assistants influence the student experience.
Method
This study employed a mixed-method approach, combining quantitative measures to comprehensively evaluate the impact of AI course assistants on student learning outcomes. The primary research method involved a randomized controlled trial (RCT), a robust experimental design used to assess the effectiveness of an intervention.
Quantitative data were collected from a diverse range of sources, providing a comprehensive view of the AI course assistants’ effectiveness. These sources included:
Grades:: Overall course grades and percentage scores were recorded to measure academic performance.
Surveys:: Standardized surveys were administered to assess intrinsic motivation, self-efficacy, and students’ perceptions of engagement, encouragement, and support.
The Motivated Strategies for Learning Questionnaire (MSLQ) was used to measure intrinsic motivation.
The General Self-Efficacy Scale (GSE) was adapted to evaluate self-efficacy.
Custom end-of-course surveys assessed students’ feelings of engagement, encouragement, and support.
Statistical analyses, including independent samples t-tests and permutation tests, were conducted to determine the significance of the differences between the treatment and control groups. Effect sizes were measured using Cohen’s d for normally distributed data and Cliff’s delta for non-normally distributed data. This mixed-method approach allowed for a thorough evaluation of the AI course assistants’ impact on student learning experiences, providing both broad quantitative insights and detailed statistical analyses.
Research Design
This study received approval from the Institutional Review Board (IRB) at Los Angeles Pacific University prior to data collection and analysis. The research protocol was reviewed to ensure that all ethical considerations, including participant privacy and data security, were adequately addressed, and the study was deemed to meet the necessary ethical standards for research involving human subjects.
This study utilized a randomized controlled trial (RCT) design to evaluate the impact of AI course assistants on the student learning experience. The participants comprised 92 undergraduate students enrolled in either BIBL 230 Biblical Literature Luke/Acts or PSYC 105 Introduction to Psychology courses at Los Angeles Pacific University. The students were randomly assigned to either the treatment or control groups. The treatment group, comprising 52 students, had access to AI course assistants developed by Nectir.io, while the control group, consisting of 40 students, did not have access to these assistants.
The random assignment ensured that any differences in outcomes between the two groups could be attributed to the presence of the AI course assistants. Multiple measures were used to assess the effectiveness of the AI assistants, including grades, intrinsic motivation to learn, general self-efficacy, and students’ feelings of engagement, encouragement, and support.
Data was collected through end-of-course surveys and course performance records. Statistical analyses, including independent samples t-tests and permutation tests, were conducted to determine the significance of the results. Effect sizes were measured using Cohen’s d and Cliff’s delta, as appropriate. This rigorous RCT design allowed for a robust evaluation of the AI course assistants’ impact on various dimensions of the student learning experience.
Participants
The study involved 92 undergraduate students enrolled in either BIBL 230 Biblical Literature Luke/Acts or PSYC 105 Introduction to Psychology as part of their degree programs. The course enrollment software randomly assigned participants to either a treatment group with the AI course assistant (n = 52) or a control group without the assistant (n = 40). These courses were chosen as they typically include students in the early stages of their degrees, offering a clear perspective on the potential benefits of AI course assistants. The participants’ ages ranged from 19 to 64 years (M = 36.38, SD = 12.48). The gender distribution was 85% female and 15% male, reflecting the broader demographics of the LAPU undergraduate population. Ethnicity distribution was as follows: 53.2% Hispanic, 15.3% White, 15.3% Black or African American, 6.3% Asian, 6.3% two or more races, and 1.8% American Indian or Alaska Native.The participants were randomly selected through the course enrollment software process, with no considerations for technical abilities, familiarity with AI, or potential bias against AI. It was assumed that, given the widespread integration of AI across many industries and everyday applications, a random selection process without factoring in familiarity, bias, or technical skills would provide a representative sample of the general student population.
Procedure
Students and instructors participated asynchronously in their online courses. The AI course assistants were integrated into three sections from two courses: two sections of BIBL 230 and one section of PSYC 105 served as treatment groups, while the remaining sections functioned as control groups. The treatment groups had access to the AI course assistant throughout their online course. All course content was identical across sections, with the only difference being the inclusion of the AI course assistant in the treatment groups (see Appendix C for LMS directions).
Minimal training on the AI assistant was provided via a five-minute video and an infographic explaining the AI course assistant’s functions and access methods (see Appendix D for the infographic). The AI course assistants’ personalities were crafted using a publicly available Microsoft prompt, focusing on encouraging critical thinking and decision-making. Students were not required to use the AI assistant, but it was made easily accessible.
Instruments and Tools
AI Course Assistants
Developed by Nectir.io, the AI course assistants were designed to offer intelligent, 24/7 responses to student inquiries about course content and assignments. This feature is particularly beneficial for students working outside traditional hours. The assistants’ personalities were designed to foster a Socratic method of learning, promoting student engagement and critical thinking without performing tasks for them (see Appendix E for an example prompt). The AI assistants complement existing student support structures, such as success coaches and faculty members, aiming to increase intrinsic motivation and address inequities in online learning.
End-of-Course Survey
During the final week, a 41-question end-of-course survey was administered, combining Likert scale ratings and free responses to assess students’ feelings of engagement, encouragement, and support (see Appendix B for questions). This survey allowed for direct comparison between treatment and control groups, using standard T-tests and Cohen’s d to measure effect sizes.
Motivated Strategies for Learning Questionnaire (MSLQ)
A modified version of the MSLQ (Pintrich et al., 1991) was used to assess intrinsic motivation to learn. Twelve questions from the motivation section were included in a separate study survey administered in week seven (see Appendix A for modified statements). The instrument’s modifications ensured relevance and accuracy in this context (Artino, 2005).
Scoring and interpretation
Responses on the modified MSLQ are scored by averaging the self-report likert responses. Higher scores indicate stronger presence of the intrinsic motivation to learn.
Reliability and Validity
The MSLQ has demonstrated high internal consistency, with Cronbach’s alpha coefficients ranging from 0.52 to 0.93 for the different subscales (Pintrich et al., 1993). The Cronbach’s alpha for intrinsic goal orientation is 0.74 (Pintrich et al., 1993). The validity of the MSLQ has been established through various studies that have confirmed its factor structure and its ability to predict academic performance (Duncan & McKeachie, 2005).
General Self-Efficacy Survey
The GSE scale (Schwarzer & Jerusalem, 1995) was modified for this study, reducing and adapting the questions to fit the specific research context. The modified GSE demonstrated high reliability (Cronbach’s alpha) and validity through expert reviews and pilot testing. This survey was essential for exploring the impact of AI course assistants on students’ self-efficacy.
Measures
We utilized a multi step process to compare the treatment and control group in each category. The steps were as follows.
Data Cleaning
Test for normal distribution using Shapiro-Wilk test.
If data is found to follow a normal distribution:
Perform an independent samples t-test.
Perform Cohen’s D for effect size.
Perform permutation testing to determine if the results from the independent samples T-test are supported. This is due to eliminate any errors due to sample size. Permutation testing gives us a more exact p-value as well.
If data is found to not follow a normal distribution and to verify findings from the t-test for data that is normally distributed:
Perform permutation testing to determine the precise p-value. Permutation tests do not assume any specific distribution, making them suitable for your data.
Utilize Cliff’s d to measure effect size.
This process was applied to the outcomes of the grades, overall percentage scores, self-efficacy, intrinsic motivation to learn and the feelings of engagement, support, and encouragement.
Data Cleaning
Before analysis, data were validated and cleaned through the following steps:
Data Validation: Cross-referenced raw data with original sources to check for entry errors.
Handling Missing Values: Excluded cases with missing values to avoid bias.
Identifying Inconsistencies: Reviewed for uniform extreme responses and logical coherence.
Consistency Checks: Ensured demographic variables and GPA scores were within valid ranges and no overlaps between treatment and control groups.
Test for Normality
In our study on AI course assistants, we utilized the Shapiro-Wilk test to assess the normality of the data distribution for the measured variables. The Shapiro-Wilk test is appropriate for this purpose because it is a powerful test for normality, even with small sample sizes, providing a robust evaluation of whether the data conforms to a normal distribution (Razali & Wah, 2011). Establishing the normality of the data is critical because the independent samples t-test we chose to utilize assumes a normal distribution. If the data is found not to have a normal distribution the t-test is not reliable. Ensuring normality allows for the appropriate application of these parametric tests, thereby enhancing the validity and reliability of the study’s conclusions (Field, 2018).
T-test
Independent samples T-tests compared mean scores of dependent variables (grades, self-efficacy, intrinsic motivation, engagement, support, and encouragement) between groups. Cohen’s d measured effect sizes, and Cronbach’s alpha assessed internal consistency of scales.
Cohen’s D
We utilized Cohen’s d to measure the effect size of the AI course assistants on the variables we measured if the data was normally distributed. Cohen’s d is a widely recognized measure of effect size that quantifies the difference between two group means in terms of standard deviation units, offering a clear interpretation of the practical significance of the results (Cohen, 1988).
Permutation Testing
In addition to the above analyses, we employed permutation testing to validate our findings. Permutation testing involves repeatedly shuffling group labels and recalculating the test statistic to build a distribution of the statistic under the null hypothesis. This non-parametric method provides robust significance testing, particularly for small sample sizes or non-normal data distributions. The permutation testing results corroborated the traditional T-test findings, reinforcing our conclusions’ reliability.
Ciiff’s d
We utilized Cliff’s delta (Cliff’s d) to calculate the effect size when there was non-normal distribution of the data. Several of the data sets did not meet the normality standards. Cliff’s delta is a non-parametric effect size measure that evaluates the degree of overlap between two groups, making it particularly suitable for data that violate normality assumptions (Cliff, 1993). This measure is robust against skewed distributions and outliers, providing a more accurate reflection of the actual differences between groups. By using Cliff’s delta, we aimed to preserve the integrity of our data and offer a valid, reliable estimation of the AI course assistants’ impact on student engagement, satisfaction, and performance. This approach ensured that our findings were not biased by the non-normality of the data, thereby enhancing the credibility and interpretability of the study’s results (Romano et al., 2006).
Results
Grades
The treatment group consisted of 52 participants and the control group consisted of 40 students. Grades were measured as an overall course grade point average (GPA) on a typical four point scale and as a percentage earned outcome. The GPA scores and percentage earned were tested for normality with the Shapiro-Wilk test. Both were found to be normally distributed. The normal distribution allowed us to perform the independent t-test and Cohens D.
There was a significant difference found in each of the measures (p = 0.0292 for GPA and 0.0143 for percent score). Each showed a moderate effect on students’ grades with an effect size of 0.4580 for GPA and 0.4981 for percent scored within the course. See Table 1 for GPA and Table 2 for percentage scored.
Table 1
GPA Results.
| GROUP | N | MEAN GPA | SD | P-VALUE | EFFECT SIZE |
|---|---|---|---|---|---|
| Treatment | 52 | 3.340 | 1.093 | 0.0292 | 0.4580 |
| Control | 40 | 2.765 | 1.399 |
Table 2
Percentage Earned.
| GROUP | N | PERCENT SCORE | SD | P-VALUE | EFFECT SIZE |
|---|---|---|---|---|---|
| Treatment (AI course assistant) | 52 | 88.269 | 14.735 | 0.0143 | 0.4981 |
| Control | 40 | 77.893 | 25.512 |
The significant differences found in both GPA and percentage score show a moderate positive effect on the use of AI course assistants on student grades. Specifically, students in the treatment group, who had access to an AI course assistant, had a higher mean GPA and percentage score than those in the control group. These findings suggest that AI course assistants could be an effective tool for improving academic outcomes in online courses.
Permutation Testing
To verify the robustness of the initial t-test results and to ensure that the sample size did not bias the findings, we conducted a permutation.
GPA
The observed difference in mean GPA between students with AI course assistants and those without was 0.575. The permutation test, conducted with 10,000 permutations, yielded a p-value of 0.0278 which is similar to the results from the initial t-test. This indicates that the observed difference in GPA is statistically significant at the 0.05 significance level, suggesting that the use of AI course assistants is associated with a higher GPA. The histogram in Figure 1 below displays the distributions of the difference in means for both GPA and percent scores under the null hypothesis. The dashed lines represent the observed differences. The observed differences in both GPA and percent scores lie in the extreme tails of the respective distributions, reinforcing the statistical significance of the results.
Figure 1
Permutation Test for GPA.
Percent score
The observed difference in mean percent scores between students with AI course assistants and those without was 10.577. The permutation test, conducted with 10,000 permutations, resulted in a p-value of 0.0121 which is similar to the results from the independent samples t-test. This result is statistically significant at the 0.05 significance level, indicating that the use of AI course assistants is associated with higher percent scores.
The histogram in Figure 2 shows the distribution of the difference in means (Percent Score) under the null hypothesis. The dashed line represents the observed difference in means.
Figure 2
Permutation Test for Percent Score.
The permutation tests provide strong evidence that the use of AI course assistants has a positive effect on both GPA and percent scores. These findings suggest that AI course assistants can enhance student performance, offering a valuable tool for improving educational outcomes.
Self-efficacy
The results for the self efficacy scores were tested for normal distribution using the Shapiro-Wilk test. The data was found to not be normally distributed (W = 0.784). The mean self-efficacy score for students with access to the AI course assistant was 4.3 while the mean self-efficacy score for students without access to the AI course assistant was 3.77. This indicates a higher average self-efficacy among students who had access to the AI course assistant.
Given that the self-efficacy scores did not follow a normal distribution, as confirmed by the Shapiro-Wilk test, a permutation test was conducted to compare the self-efficacy scores between the two groups.
The observed mean difference in self-efficacy scores was ΔM = 0.53\Delta M = 0.53 ΔM = 0.53. To assess the significance of this difference, a permutation test with 10,000 iterations was performed. The resulting p-value was p = 0.0004 p = 0.0004 p = 0.0004, indicating that the observed difference in self-efficacy scores is statistically significant.
The results of the permutation test suggest that access to an AI course assistant has a significant positive impact on students’ self-efficacy. This finding is supported by both the descriptive statistics and the statistical test, indicating that the use of AI course assistants can effectively enhance students’ confidence in their ability to succeed in their studies.
The calculated Cliff’s delta for self-efficacy between the treatment group with access to AI course assistants and the control group without access to AI course assistants is approximately 0.228. This value indicates a small to moderate effect size, suggesting that the introduction of AI course assistants has a noticeable but not large impact on self-efficacy among students.
Intrinsic motivation to learn
The results from the modified MSLQ were tested using the Shapiro-Wilk test. The data was found to not be normally distributed. Permutation testing was then conducted. The observed difference in means between the two groups was 0.6845. To assess the statistical significance of this observed difference, 10,000 permutations of the data were performed. In each permutation, the labels indicating whether the AI course assistant was used were shuffled, and the difference in means was recalculated.
The distribution of the differences in means from the 10,000 permutations is shown in Figure 3. The observed difference of 0.6845 is indicated by the vertical dotted line.
Figure 3
Permutation Test Intrinsic Motivation to Learn.
The p-value, calculated as the proportion of permuted differences that were greater than or equal to the observed difference, was found to be 0.0002. This p-value is less than the common significance level of 0.05, leading us to reject the null hypothesis. Therefore, we conclude that the use of AI course assistants has a statistically significant positive effect on students’ motivation to learn.
The calculated Cliff’s delta for the treatment group with AI course assistants compared to the control group without AI course assistants is approximately 0.356. This value indicates a moderate effect size, suggesting that the introduction of AI course assistants has a noticeable impact on intrinsic motivation to learn among students. In general, a Cliff’s delta value between 0.33 and 0.474 indicates a moderate effect, meaning that there is a meaningful difference in the distributions of intrinsic motivation scores between the two groups.
Feelings of engagement, support, and encouragement
The feelings of engagement, encouragement, and support were tested for normal distribution using the Shapiro-Wilk test. None of the three were normally distributed. The permutation tests revealed no statistically significant differences were found between the use of AI course assistants and a students’ feeling of engagement, encouragement, or supported. The lack of statistical significance suggests that within the context of this study, the AI course assistants neither increased or decreased these feelings in a meaningful way.
Discussion
While this study focused on courses in biblical literature and psychology, the findings suggest that the benefits of AI course assistants—such as enhanced academic performance, increased intrinsic motivation, and improved self-efficacy—can likely be applied across a wide range of academic disciplines. The core functions of the AI course assistants, including providing personalized feedback, fostering engagement, and offering immediate support, are not discipline-specific. These features can be just as beneficial in fields such as the natural sciences, social sciences, and humanities.
For example, in fields like mathematics or engineering, AI course assistants could help students work through complex problem-solving exercises by offering step-by-step guidance and immediate feedback. In creative disciplines like art or design, AI assistants could provide real-time feedback on creative projects, supporting iterative improvements and fostering creativity, as Zhao (2023) suggests in his work on AI in creative industries. Furthermore, in language and literature courses, AI could assist with grammar, writing structure, and language learning, providing a valuable supplement to instructor feedback.
The potential of AI course assistants extends beyond specific content areas and can be applied broadly in higher education to support diverse student populations and enhance the learning experience in various educational contexts.
Grades, motivation, and self-efficacy
The study’s results indicate that implementing AI course assistants significantly and positively impacted students’ GPA, self-efficacy, and intrinsic motivation to learn. This was found using the independent samples t-test and permutation testing. Specifically, students in the treatment group with access to the AI course assistants demonstrated higher GPA achievement than their peers in the control group, 3.34 to 2.77. This finding is reinforced by the moderate effect size observed, suggesting that the AI course assistants played a meaningful role in enhancing academic performance. The improvement in GPA highlights the potential of AI-driven support tools to provide effective, personalized assistance that can positively influence student outcomes (Hanshaw & Miller, 2024).
Similarly, there was a significant difference in intrinsic motivation to learn between the treatment and control groups. This finding underscores the practical significance of the AI course assistants in fostering a more engaging and motivating learning environment. Intrinsic motivation is critical to student success, as it drives deeper engagement with course material and a greater willingness to tackle challenging tasks (Ryan & Deci, 2000). The use of AI course assistants appears to promote an educational experience that is both enjoyable and rewarding, thereby enhancing students’ intrinsic motivation Parsakia, (2023).
The results for general self-efficacy found a statistically significant difference between the treatment and control groups.These findings suggest that the AI course assistants have a meaningful impact on students’ self-efficacy. The combination of support being available at all times and the immediate feedback provided by AI course assistants suggest a positive outcome of building students’ efficacy in their ability to manage course-related tasks (Bandura, 1997).
These findings suggest that AI course assistants have a significant role in creating a positive learning experience for students. We are not suggesting that AI course assistants are a substitute for other types of content and human interactions. The findings show that the AI course assistants are part of the larger student support ecosystem that works in unison to create positive student learning experiences. These results underscore the potential of AI technologies to enhance the educational experience by providing continuous, personalized support that addresses individual student needs (Richardson et al., 2017; Sublett, 2020.; Williams, 2024).
Feelings of engagement, support, and encouragement
Interestingly, our study found no statistically significant differences in students’ feelings of engagement, encouragement, and support between the treatment and control groups. This is different from, but not necessarily in opposition to, findings of Ilieva et al. (2023). We posit that this lack of significant difference does not necessarily indicate the AI course assistant’s ineffectiveness in these areas. Instead, we attribute this to Los Angeles Pacific University’s (LAPU) high-touch model, which utilizes student success coaches to provide ongoing personalized support on a weekly basis and maintain high levels of student engagement, encouragement, and support. LAPU designed the success coach model to ensure that students receive personalized attention and guidance throughout their educational journey, effectively addressing their needs and concerns (Richardson et al., 2017).
The effectiveness of LAPU’s success coach model may render additional support from the AI course assistant negligible in terms of further enhancing students’ feelings of engagement, encouragement, and support. This suggests that while the AI course assistants are beneficial for improving academic performance and intrinsic motivation, the existing support structures at LAPU may already be highly effective in maintaining student engagement and support. Future research could explore this further by examining the interaction between human and AI support systems to determine the optimal balance for enhancing various aspects of the student learning experience.
It is important to note that the increased feelings of encouragement, engagement, and support may be directly influenced by other factors, such as the type of model used by the institution. Different AI models and their implementation strategies can significantly impact how students perceive and interact with these technologies. Williams (2024) highlighted the ethical considerations and potential biases inherent in AI technologies, emphasizing the need for careful implementation and continuous evaluation to ensure these tools equitably benefit all students.
Equitably benefiting all students means that AI technologies should be implemented in a manner that considers these disparities and actively works to mitigate them. This could include ensuring that AI tools are accessible to students with disabilities, culturally responsive, and free from biases that could disadvantage certain groups of students. Continuous monitoring and evaluation of AI tools are crucial to identify and rectify any issues that may arise, ensuring that these technologies support the learning needs of every student fairly and inclusively.
For example, research by West and Bleiberg (2013) discusses the importance of designing educational interventions that cater to the diverse needs of students, emphasizing the need for inclusivity and fairness in educational technologies. Similarly, Williams (2024) underscores the ethical considerations in AI implementation, advocating for a careful approach to ensure that these tools do not perpetuate existing inequities but instead promote equal opportunities for all students.
Limitations
While this study offers valuable insights into the impact of AI course assistants on student learning outcomes, several limitations and potential biases must be acknowledged. First, the study did not account for students’ prior experience with AI tools or their attitudes toward generative AI, which could influence their willingness to engage with the AI assistant and affect learning outcomes. This introduces a potential bias, as students with more familiarity or positive views on AI might have been more inclined to use the assistant effectively, thereby skewing the results in favor of the treatment group.
Additionally, while random assignment was employed to create the treatment and control groups, there remains the possibility of selection bias due to unmeasured variables, such as students’ baseline technology skills or academic motivation, which could also influence outcomes. Although the sample size was sufficient for detecting significant differences, it was relatively small and limited to students from specific courses at one institution, which may affect the generalizability of the findings.
Finally, as with any study involving self-reported measures, there is the risk of response bias, where students in the treatment group may have responded more positively due to their awareness of using a novel tool. This could potentially inflate perceptions of engagement, motivation, and support. Future research should consider these limitations and strive for larger, more diverse samples while controlling for pre-existing familiarity with AI tools.
Future research
Repeating and expanding this research to more constructs is recommended for further research. In order to fully understand the “whole student experience” from a holistic perspective, a qualitative research study is also recommended to delve deeper into the nuanced interactions between students and AI-driven support systems. Moreover, expanding the research with larger and more diverse samples would allow for broader generalizations and insights into optimizing the balance between human and AI support mechanisms for maximizing student success.
Future research should explore additional constructs to gain a holistic understanding of the student experience, including qualitative studies to delve deeper into the nuanced interactions between students and AI-driven support systems. Moreover, expanding the research with larger and more diverse samples would allow for broader generalizations and insights into optimizing the balance between human and AI support mechanisms for maximizing student success.
In future research, it would be beneficial to conduct regression analyses to determine if we can develop reliable predictive models for student outcomes and well-being based on the usage of AI course assistants and other support variables that make up the student support ecosystem. Regression analysis is a powerful statistical tool that can help identify the relationships between various independent variables, such as the extent of AI course assistant usage, student demographics, and study habits, and dependent variables like GPA, self-efficacy, intrinsic motivation, and feelings of engagement, encouragement, and support.
By employing regression models, researchers can not only assess the direct impact of AI course assistants on these outcomes but also control for potential confounding factors. This approach allows for a more nuanced understanding of how different variables interact and contribute to student success. For instance, a multiple regression analysis could reveal how much of the variance in GPA or self-efficacy can be explained by the frequency of AI course assistant interactions, while controlling for other factors such as prior academic performance or socio-economic status.
The use of regression analysis can help in identifying key predictors of student well-being, which can inform the design and implementation of AI tools in educational settings. For example, if the analysis indicates that certain features of the AI course assistant are particularly effective in enhancing self-efficacy or intrinsic motivation, educational institutions can focus on optimizing these features to better support student needs.
Future studies should also consider longitudinal designs to examine the long-term effects of AI course assistants on student outcomes. By tracking students over multiple semesters, researchers can determine whether the benefits of AI support are sustained over time and how these tools influence students’ academic trajectories and overall well-being. Integrating qualitative data from student feedback can further enrich the findings, providing deeper insights into the experiences and perceptions of students using AI course assistants.
Conclusion
The findings of this study suggest that AI course assistants have a significant and positive impact on students’ academic performance and intrinsic motivation to learn. However, it is important to view AI not as a replacement for human-driven learning experiences, but as a complement to them. Zhao’s (2023) work on the integration of artificial intelligence in creative industries provides a valuable framework for understanding the role of AI in education. Just as AI can assist artists and designers by automating repetitive tasks and offering new forms of inspiration, AI course assistants can enhance the educational process by providing immediate feedback, personalized support, and innovative learning opportunities.
Zhao emphasizes that the most effective use of AI in creative industries occurs within collaborative frameworks where humans guide AI systems. This perspective aligns with the results of our study, which suggest that AI course assistants are most beneficial when they complement, rather than replace, the existing human support structures in education. As AI continues to evolve, future advancements in educational AI systems may come from integrating better models of human cognition, emotion, and perspective, as Zhao argues for in creative contexts. By leveraging these advances, educational institutions can create even more impactful, personalized learning environments that foster student success.
Data Accessibility Statement
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
Additional File
The additional file for this article can be found as follows:
Ethics and Consent
Ethical approval was obtained through the Institutional Review Board (IRB) at Los Angeles Pacific University.
Acknowledgements
We acknowledge the assistance and insight from the following organization and people.
Nectir provided the AI course assistants and trained us on how to implement the assistants.
Ethan Wicker provided statistical analysis expertise to help us define the types of tests we should use.
Jacob Hanshaw wrote much of the code for the different analyses that we utilized within the process.
Competing Interests
The authors have no competing interests to declare.
Author Contributions (CRediT)
George Hanshaw: Conceptualization, Data curation, Formal Analysis, Investigation, Methodology, Supervision, Validation, Visualization. Writing – original draft, Writing – review & editing. Joanna Vance: Conceptualization, Data curation, Formal Analysis, Investigation, Writing – original draft, Writing – review & editing. Craig Brewer: Conceptualization, Data curation, Formal Analysis, Investigation, Writing – original draft, Writing – review & editing. All authors have read and agreed to the published version of the manuscript.
Author Notes
Based on Academic Integrity and Transparency in AI-assisted Research and Specification Framework (Bozkurt, 2024b), the authors of this paper acknowledge that this paper was partially reviewed and edited by NectirAI Writing assistant (June 2024) and Grammarly (June 2024), complementing the human editorial process. The human authors critically assessed and validated the content to maintain academic rigor. The authors also assessed and addressed potential biases inherent in the AI-generated content. The final version of the paper is the sole responsibility of the human authors.