Introduction
Increasing the number of college graduates in science, technology, engineering and mathematics (STEM) is a priority for the United States (National Academies of Sciences, 2016). Yet only about 40% of U.S. undergraduate students who express interest in a science major at the start of college will complete one within six years (Eagan et al., 2014; National Academies of Sciences, 2016). Completion rates for underrepresented or marginalized (URM) students, such as African American, Hispanic, Native American, low-income, first-generation, or female students are even lower (Chang et al., 2014; Estrada et al., 2016). Thus, there is a need for evidenced-based tools to improve the success and retention of undergraduate STEM majors.
Most, if not all, institutions of higher education offer a tutoring or academic support program of some form to students in introductory STEM courses. A traditional drop-in tutoring program (TT) usually consists of open office hours staffed by undergraduate or graduate students who answer questions and help students work through course material. Supplemental Instruction (SI, also known as Peer-Assisted Learning or Peer-Assisted Study Sessions) is an academic support program initially developed at the University of Missouri, Kansas City (Martin & Arendale, 1992) that is widely used both in the United States and internationally (Arendale, 2002). SI uses “Leaders,” students who have previously succeeded in the course, to model student success in the classroom and run collaborative peer learning sessions that help students master course content. Past research on SI has documented improvements in course grades (Burkholder et al., 2021; Martin & Arendale, 1992; Rath et al., 2012), retention (Martin & Arendale, 1992), and study skills (Stanich et al., 2018). SI can also build students’ sense of community, self-confidence, and self-efficacy (Malm et al., 2015; Mkonto, 2018; Spedding et al., 2017). Benefits to SI have also been demonstrated for underrepresented and marginalized students (Bowman et al., 2021; Rath et al., 2012; Yue et al., 2018). However, McCarthy et al. (1997) and Dawson et al. (2014) identified multiple flaws in experimental design and/or reporting in many of these previous studies. While recent studies account for many of these criticisms (e.g., Bowman et al., 2021; Burkholder et al., 2021; Guarcello et al., 2017), there is a need for more studies that demonstrate the benefits of SI with rigorous experimental designs. Two areas of particular concern are properly controlling for self-selection or motivation bias and the treatment of attendance as a binary variable.
Assessing the effectiveness of SI
The methodological flaws identified by McCarthy et al. (1997) and Dawson et al. (2014) include failing to report p-values or effect sizes, failing to describe how closely the local implementation of SI matched program principles, treating SI participation as a binary variable regardless of the number of sessions attended, and failing to control for differences in academic ability or motivation to succeed between students who used SI and those who didn’t. For example, when McCarthy et al. (1997) analyzed their results using the methods of Martin and Arendale (1992), they found a significantly higher final mean grade for undergraduate engineering students who attended SI compared to those who didn’t, but they found no significant differences in initial academic abilities as measured by admissions scores. When the same authors used a multivariate analysis that included academic abilities as a separate variable, SI no longer significantly improved course grades. Dawson et al. (2014) examined 29 studies of SI published from 2001 to 2010 and found that while there were many suggested benefits to SI, none were “supported by a gold standard study involving random assignment to groups and sufficient detail about methodology, participants, and the SI intervention in practice” (pp. 634–635).
While more recent studies have addressed many of these criticisms, self-selection bias remains a challenge. Self-selection bias arises because voluntary attendance is a fundamental tenet of the SI program. Because students choose to attend SI sessions, there is a possibility that the students who attend SI differ from non-attenders in ways that also influence their success in the course. SI attendees could be better prepared academically or more motivated to succeed in the course than non-attenders, and these differences could cause grade differences between attendees and non-attenders that are unrelated to the content of the SI program.
Random assignment to treatments is the best way to control for self-selection bias (Dawson et al., 2014); however, this approach undermines the voluntary nature of SI. Moreover, true randomized controlled trials would face ethical considerations of denying a student an intervention that might improve their success. In cases where there are a limited number of seats in an SI program, researchers have randomly created SI and control groups from a pool of volunteers (Parkinson, 2009; Stanich et al., 2018). Alternately, individual student attributes have been used to control for student differences in a multivariate analysis (Buchanan et al., 2019; Burkholder et al., 2021) or to refine a control group of non-attenders to better match the group of SI attenders (Bowman et al., 2021; Guarcello et al., 2017). Important variables to include are demographic traits, measures of academic ability, and measures of student motivation.
A third approach to control for self-selection bias is to compare groups of students offered two different academic support programs. This design treats the number of sessions attended as a covariate in the analysis. If, after controlling for attendance, one program has significantly better outcomes, that would indicate the program itself is driving those outcomes. Conversely, if students with similar levels of attendance in each program show similar performance or grades it suggests that motivation or other student attributes, rather than the academic support program, is driving student success. A secondary advantage of this approach is that it directly compares two different academic support programs to each other. This may provide more relevant information for an institution that is choosing among programs. Direct comparisons are also important because SI can be more expensive to implement than traditional tutoring. SI Leaders must be paid for time that is spent attending the class and preparing sessions. It is important to know if these additional costs produce additional benefits for students. Surprisingly few studies have compared the success of students in an SI program directly to other forms of academic support. One exception is Hodges and White (2001), who found greater grade improvement among students attending SI sessions than those attending traditional tutoring. However, SI and tutoring were not compared within the same course but across courses in different disciplines.
Attendance as an explanation for the success of SI
A secondary goal of this study was to compare attendance rates between two different support programs. Voluntary attendance is a foundational aspect of both SI and traditional tutoring programs, but a voluntary program can only help the students who attend. All else being equal, an academic support program that attracts more students will benefit more students. Yet, few studies have directly compared attendance rates for SI to other academic support programs. Hodges and White (2001) reported a greater percentage of students attending tutoring than SI (68% vs. 51%) but did not test for statistical significance. SI participation rates are often below 50% of the students enrolled in a course (Martin & Arendale, 1992) and are frequently much lower (Allen et al., 2019; Hodges et al., 2001).
Supplemental Instruction and URM students
Several aspects of the SI program may be particularly beneficial to URM students. The retention of URM science majors depends on both a student’s individual characteristics and the institutional environment they experience (Chang et al., 2011; Chang et al., 2014). SI can contribute positively to both pieces. First, SI is founded in a constructivist model of learning and emphasizes the development of abstract reasoning (Martin & Arendale, 1992). Student success in SI is attributed not just to increased knowledge of course content but also to improved study skills and better metacognition of the student’s own learning processes (Burkholder et al., 2021). These skills are rarely addressed directly in introductory science courses (Bowman et al., 2021). URM students frequently enter college with poorer academic preparation than non-URM students, so addressing those challenges can increase retention (Chang et al., 2014). Second, the collaborative, peer-learning approach of SI may provide a positive institutional environment that is particularly beneficial to URM students who often experience marginalization in STEM classrooms (Bowman et al., 2021). SI sessions may foster a sense of community and belonging (Bowman et al., 2021) that has been demonstrated to increase URM persistence (Chang et al., 2014). This combination of academic and environmental benefits may explain why URM students often benefit disproportionately from SI when compared to other demographic groups (Bowman et al., 2021; Burkholder et al., 2021; Rath et al., 2012; Rath et al., 2007; Stanich et al., 2018; Yue et al., 2018).
Current Study
In this study, the effectiveness of SI and TT sessions are compared for students attending introductory chemistry courses offered by three small liberal arts colleges in the southwestern United States. Importantly, it takes into consideration the methodological flaws of previous studies. In each of two years, two sections of the course taught by the same instructor were compared: one section used SI as the academic support, while the other used TT. Although this is not a truly randomized trial, students were unaware of the different support programs when they enrolled in the courses. This experimental design separates motivation to attend support sessions from the benefits of the sessions themselves and also allows for a direct comparison of the attendance rates and exam grades for each program. Secondary goals of this study were to compare attendance rates between the two programs and to separately examine the program differences for URM students, as these groups may experience, use, and/or benefit from these programs differently.
In sum, this study investigated four major questions:
Does attending more support sessions improve final exam grades, and does the pattern differ between SI and TT?
Do URM students (Hispanic, African American, or first-generation) benefit differently from attending support sessions than non-URM students? Does one program benefit them more?
Do students attend SI and TT sessions at similar rates?
Can student attributes, such as demographic data or motivation, predict attendance frequency?
Methods
Participants
Participants were 159 undergraduates (see Table 1 for details). Students were enrolled in the first semester of a two-semester introductory chemistry course (“Chem 14”) offered in Fall semester 2017 and 2018 from a joint science program for three small liberal arts colleges in the southwestern United States; two of the colleges are coeducational and one is a women’s college.
Table 1
Participant Flow and Demographics of Participating Students.
| ATTRIBUTE | 2017 | 2018 | ||
|---|---|---|---|---|
| CONTROL | SI | CONTROL | SI | |
| Enrolled Students | 42 | 38 | 38 | 41 |
| Did not consent | 3 | 2 | 1 | 5 |
| Dropped course | 6 | 2 | 1 | 0 |
| Transferred sections | 0 | 0 | 1 | 0 |
| Participating Students | 33 | 34 | 35 | 36 |
| Gender | ||||
| % Women | 83.3 | 73.7 | 88.6 | 72.2 |
| IPEDS Race and Ethnicity | ||||
| % American Indian or Alaska Native | 0 | 0 | 0 | 0 |
| % Asian | 12.1 | 17.6 | 17.1 | 5.6 |
| % Black or African American | 6.1 | 2.9 | 8.6 | 8.3 |
| % Hispanic | 15.2 | 8.8 | 5.7 | 11.1 |
| % Two or more races | 9.1 | 2.9 | 0 | 2.8 |
| % Native Hawaiian or Other Pacific Islander | 0 | 0 | 0 | 0 |
| % Nonresident | 0 | 2.9 | 2.8 | 2.8 |
| % White | 54.5 | 47.1 | 62.9 | 61.1 |
| % Unknown race | 3.1 | 17.7 | 2.8 | 2.8 |
| Years Enrolled | ||||
| % First year of college | 75.8 | 79.4 | 82.9 | 72.2 |
Of the original 158, 138 students met the inclusion criteria for the study: enrolled in the standard chemistry course, completed all exams, and provided written consent (Table 1). The majority of participants were white, female, and in their first year of college (Table 2). Although first-year students at these institutions have not declared a major, the chemistry course is primarily taken by students planning to complete a science major or pursue a medical career. Students did not provide information about their ages, but the three participating colleges report an average age of 18 for first-time first-year students.
Table 2
Descriptive Statistics of Experimental Variables in Each Year of the Experiment.
| ATTRIBUTE | 2017 | 2018 | ||
|---|---|---|---|---|
| CONTROL n = 33 | SI n = 34 | CONTROL n = 35 | SI n = 36 | |
| URM Status | ||||
| URM | 9 | 5 | 7 | 8 |
| Non-URM | 24 | 29 | 28 | 28 |
| Motivation Score | ||||
| N | 31 | 29 | 30 | 32 |
| Mean | 15.0 | 14.9 | 15.9 | 15.4 |
| SD | 2.00 | 2.12 | 1.78 | 2.39 |
| Neighborhood Income | ||||
| N | 33 | 33 | 35 | 36 |
| Mean | $118,234 | $164,418 | $164,145 | $160,503 |
| SD | $74,263 | $125,541 | $215,720 | $154,113 |
| Math Placement | ||||
| Pre-calculus | 0 | 1 | 7 | 2 |
| Calculus 1 | 17 | 14 | 16 | 17 |
| Calculus 2 or 3 | 15 | 18 | 9 | 16 |
| No information | 1 | 1 | 3 | 1 |
| Previous Science Courses | ||||
| 0 | 26 | 28 | 32 | 29 |
| 1 | 4 | 2 | 2 | 4 |
| 2 | 2 | 4 | 1 | 2 |
| 3 | 1 | 0 | 0 | 1 |
| Years since entering college | ||||
| 0 | 25 | 27 | 29 | 26 |
| 1 | 7 | 5 | 4 | 9 |
| 2 | 1 | 0 | 0 | 1 |
| 3 | 0 | 1 | 0 | 0 |
| No data | 0 | 1 | 2 | 0 |
| Attendance | ||||
| % ≥ 1 session | 51.5% | 82.5% | 28.6% | 58.3% |
| % ≥ 3 sessions | 21.1% | 32.4% | 14.3% | 38.9% |
| Exam 1 score (±SD) | 78.1 (± 14.6) | 78.5 (± 12.2) | 77.5 (± 14.5) | 78.7 (± 13.0) |
| URM | 72.8 (±20.4) | 72.7 (±19.4) | 63.1(±9.6) | 74.1 (± 12.5) |
| Non-URM | 80.1 (±11.7) | 79.5 (±10.7) | 81.1 (±13.3) | 80.0 (±13.1) |
| Final Exam Score (±SD) | 71.1 (± 9.90) | 75.5 (± 10.8) | 79.7 (± 13.6) | 78.3 (± 11.5) |
| URM | 73.22 (±11.6) | 70.9 (±13.4) | 73.0 (±8.2) | 80.4 (±8.1) |
| Non-URM | 79.9 (±8.7) | 76.3 (±10.3) | 81.4 (±14.2) | 77.6 (±12.3) |
Academic support treatments
Supplemental Instruction
The SI condition was part of a pilot Supplemental Instruction program that was established in the fall of 2017 and continued through the 2018–2019 academic year. The pilot program comprised 2–4 courses per semester in biology and chemistry. The implementation of the Supplemental Instruction program followed the UMKC model (Martin & Arendale, 1992) as closely as possible. SI Leaders were recruited from a pool of students who had successfully completed the course, with the same instructor when possible. Leaders attended the class, met regularly with the course instructor to pick topics for SI sessions, and ran two 1-hour SI sessions weekly. Each session comprised 2–4 short activities developed from SI materials. To ensure adherence to the SI program, all SI Leaders attended eight hours of training, based on UMKC training materials (The Curators of the University of Missouri, 2014), at the start of each semester. Additionally, SI Leaders attended biweekly training sessions and submitted lesson plans weekly. A trained supervisor observed at least two individual SI sessions per Leader per semester. Leaders also observed and gave feedback on each other’s sessions.
Traditional Tutoring
The traditional tutoring (TT) sections were offered drop-in tutoring provided through the department’s chemistry program. As with SI, the drop-in tutoring program was staffed by trained peer educators who were undergraduate students who had successfully completed the general chemistry course. Unlike SI Leaders, the TT tutors did not plan activities or attend classes. They were available during office hours to answer questions and help students work through course material. The format of the tutoring differed slightly between the two years of this experiment. In 2017, one or more peer educators held 2–3 evening office hours five days per week (Sunday–Thursday evenings). Students who attended tutoring could ask questions about course material but may have had to wait for a tutor to finish working with another student before they could get help. In 2018, the TT program made several changes intended to improve its effectiveness. The hours were expanded to 12–8 pm, four days a week, and the program switched to an online reservation system that allowed students to sign up, either individually or in groups of up to six, for a 30-minute tutoring appointment. Drop-in visits were still allowed. Each student was limited to 60 minutes of individual tutoring per week, but students could exceed this limit if they signed up in groups. Alongside these changes, the training program for peer educators was revised and strengthened.
Procedure
In each year of the study, two sections of the introductory chemistry course taught by the same instructor were selected. The two sections covered the same content and used the same exams. One section was assigned to SI and the other to TT. The section assignment was made based on the availability of an SI Leader to attend the classes in the SI section. To encourage attending sessions, students were offered extra credit to attend three sessions of the support program assigned to their section (SI or traditional tutoring) before the first exam. Students were unaware of the experiment or the differing support programs when they enrolled in the course; however, during the first week of the course, I explained the study to students in both sections and solicited written consent. The SI Leader involved in the experimental sections was the same individual both years. This experiment was deemed “exempt” by the Institutional Review Boards of the three colleges.
Measures
Exam Scores
Each student’s final exam grade, calculated as a percentage (Table 2), was used as the response variable for Questions 1 and 2. Final exam scores were chosen over course grade because course grades included a separate laboratory component. The student’s percentage score on the first exam in the course was used as a covariate for Questions 1 and 2 to control for differences in initial scientific and mathematical ability.
Attendance
The number of support sessions attended (“Total Visits”) was used as the response variable for Questions 3 and 4 and as an independent variable for Questions 1 and 2. Session attendance for SI was collected on paper sign-in sheets. Session attendance for drop-in tutoring was collected electronically by the chemistry program and provided as a .csv file.
Individual Student Traits
Individual student information, including race, ethnicity, first-generation status, initial college math placement, and number of prior science courses completed were provided by the three college registrars (Table 2). The race and ethnicity categories were defined by the U.S. Integrated Postsecondary Education Data System (National Center for Education Statistics). Students were identified as “first-generation” if their parents had not completed a bachelor’s degree. Because of the small numbers of first-generation, Hispanic, and Black students (Table 1), I pooled all students in those groups into a single category of “underrepresented or marginalized” (URM). The non-URM category included Asian, White, multiracial, noncitizens, and students for whom racial information was unknown.
The number of science courses previously taken by each student was calculated from transcript information provided for each student by the three college registrars. Mathematics placement, used as an independent variable for Questions 1 and 2, was a qualitative ranking of math skills on entering college (pre-Calculus, Calculus I, or Calculus II and higher). Where available, the student’s first math course taken was used to determine mathematics placement. Alternately, placement was determined by an AP or IB calculus test result or an institutional math placement test result. Other estimates of quantitative skills, such as standardized test scores or high school GPA, were not available from all three institutions.
Science Motivation
Motivation was assessed using the Science Motivation Questionnaire II (Glynn et al., 2011), which consists of 25 statements with Likert-scale responses that measure five components of motivation: intrinsic motivation, self-determination, self-efficacy, career motivation, and grade motivation. The survey was administered in class in the first and last week of each semester in a paper format. Subsection scores were summed to create an overall score for motivation.
Estimated Neighborhood Income
Because financial aid status was not available, an estimated neighborhood income was calculated as the average household income of the student’s home ZIP code. The student’s high school ZIP code was substituted if the home ZIP code was unavailable. Average household income by ZIP code was calculated from the United States Internal Revenue Service’s Individual Income Tax ZIP Code Data 2015 (Internal Revenue Service, 2015). However, neighborhood income was highly correlated with URM status and so was not included in any analyses.
Analyses
All statistical analyses were conducted in R version 4.0.2 (R Core Team, 2020). Only students with values for all explanatory variables were included in analyses. The model selection approach for Questions 1, 2, and 4 followed a backwards stepwise approach (Stanich et al., 2018; Zuur et al., 2009), beginning with a full model that included all explanatory variables of interest and their interactions. Non-significant model terms were dropped starting with the highest-order interactions in the model. Terms that occurred in a significant interaction term were also retained. All parametric models were tested for assumptions of normality and homoskedasticity as described below. Where transformations were used to meet assumptions, results are presented after back-transformation.
Results
Questions 1 and 2: Effect of Attendance and Treatment on Final Exam Score
This analysis used Exam1 score, URM status, total number of sessions attended, year, and treatment as the explanatory variables. Motivation scores were included in preliminary analyses but showed no correlation with final exam score. Math placement and family income were also excluded because they were highly correlated with Exam 1 score and URM status, respectively. The prior number of science courses was also excluded as it was zero for most students. Initial analyses revealed a significant four-way interaction among year, URM status, program type, and total visits. To simplify the interpretation of the results, I analyzed each year separately starting from a model that included all two- and three-way interactions.
2017
The 2017 data were analyzed with a linear model. The final model included five terms: Exam 1 score, Total Visits, URM status, treatment, and the interaction between treatment and Total Visits (Table 3). There was a trend of decreasing error variance with increasing number of visits. However, including a separate error term for Total Visits did not significantly improve the overall model fit (LRT = 2.781348, p = 0.0954), although it did decrease p-values for some terms in the model.
Table 3
ANOVA Table for Final Exam Grade in 2017.
| MODEL TERM | df | ESTIMATE | STANDARD ERROR | F | P |
|---|---|---|---|---|---|
| Exam1 | 1 | 0.426 | 0.076 | 31.2564 | <0.0001† |
| Total Visits | 1 | 0.949 | 0.375 | 5.2959 | 0.0248† |
| URM | 1 | 1.028 | 1.284 | 0.6410 | 0.4265 |
| Treatment | 1 | 3.343 | 1.276 | 2.9827 | 0.0892 |
| Visits * Treatment | 1 | –0.750 | 0.368 | 4.1534 | 0.0459† |
| Terms dropped from the final model | |||||
| Ex1 * Visits | 1 | 0.24285 | 0.6241 | ||
| Ex1 * URM | 1 | 0.00024 | 0.9893 | ||
| Ex1 * Trt | 1 | 0.35636 | 0.5529 | ||
| Visits * URM | 1 | 1.00129 | 0.3140 | ||
| URM * Trt | 1 | 0.56178 | 0.4566 | ||
| Ex1 * TV * URM | 1 | 0.18102 | 0.6722 | ||
| Ex1 * TV * Trt | 1 | 0.00421 | 0.9488 | ||
| Ex1 * URM * Trt | 1 | 0.12803 | 0.7219 | ||
| Visits * URM * Trt | 1 | 0.61707 | 0.4353 | ||
[i] 1–9 Order of terms dropped from initial model, †p < 0.05.
Note. n = 67 students.
There were significant effects of Exam 1 score, sessions attended, and a significant interaction between the treatment and attendance in the final model. A student’s Exam 1 score was strongly correlated with their final exam score, with each 1% increase in Exam 1 increasing the final exam score by 0.46%, 95% CI[0.30,0.63%]. Attending support sessions also increased final exam scores regardless of treatment, as evidenced by the significant main effect of Total Visits (Table 3). But there was also a significant interaction between treatment and Total Visits, with a greater benefit of each additional visit for SI students than control students (Figure 1).
Figure 1
The Effect of Session Attendance on Final Exam Scores in 2017.
Note. TT = traditional tutoring, SI = Supplemental Instruction. Points represent individual student scores. Lines represent predicted final exams scores from the final statistical model at three different initial exam one scores (50%, 70%, and 90%). Degree of shading indicates first exam score.
Specifically, the SI treatment saw an average 1.70%, 95% CI[0.59,2.81] increase in final exam score with each additional session they attended, a statistically significant slope (t = 3.07, df = 61, p = 0.0033). Students in the TT treatment only saw a 0.2%, 95% CI[–1.02,1.41] increase in final exam grade for each session attended, which was not statistically different from 0 (t = 0.40, df = 61, p = 0.69). There were no significant effects of URM status on final exam grade, nor were there significant interactions between URM status and other model terms (Table 3).
2018
The 2018 data required a generalized linear model to account for heterogeneous variances. Three outliers violated assumptions of normality and homoskedasticity. All analyses were run with and without the outliers to determine their effect on the final results. In both models, significant heterogeneity of variances was observed among both URM status and Total Visits. For the model with outliers, including a separate variance term for URM and non-URM students significantly improved model fit (LRT = 10.7465, p = 0.001). For the model with outliers omitted, including separate error variance terms for Total Visits fit the data significantly better than one with no error heterogeneity terms (LRT = 7.444728, p = 0.0064). Excluding the outliers had no effect on the statistical tests of the fixed effects in the model but did change some parameter estimates. For simplicity, I primarily report the model with outliers and include information from the model without outliers only where it led to a different interpretation of the data.
The final models for 2018 included two statistically significant three-way interactions: Exam 1 × URM status × Treatment and Total Visits × URM status × Treatment (Table 4). These terms were significant in models with and without outliers. These interactions were driven primarily by the differences in success of the URM students in the TT treatment (Figure 2). Those students were the only group that did not show a positive relationship between Final Exam score and either Exam 1 score or Total Visits (Table 5, Figure 2).
Table 4
ANOVA Table for Final Exam Grade in 2018.
| MODEL TERM | df | ESTIMATE | STANDARD ERROR | F | P |
|---|---|---|---|---|---|
| Exam1 | 1 | 0.346 | 0.1735 | 3.978† | 0.0507 |
| Total Visits | 1 | 3.037 | 1.472 | 4.254† | 0.0436† |
| URM | 1 | 69.988 | 21.216 | 10.882 | 0.0016† |
| Treatment | 1 | –10.376 | 20.512 | 0.256† | 0.6148 |
| Ex1 * URM | 1 | –0.832 | 0.272 | 9.360† | 0.0033† |
| Ex1 * Trt | 1 | 0.131 | 0.248 | 0.280† | 0.5982 |
| TV * URM | 1 | –10.695 | 2.216 | 23.302† | <0.0001† |
| TV * Trt | 1 | –2.412 | 1.716 | 1.976† | 0.1651 |
| URM * Trt | 1 | –72.525 | 27.097 | 7.164† | 0.0096† |
| Ex1 * URM * Trt | 1 | 0.942 | 0.346 | 7.408† | 0.0085† |
| Visits * URM * Trt | 1 | 10.505 | 2.463 | 18.184† | <0.0001† |
| Terms dropped from the final model | χ2 | ||||
| Ex1 * Trt * URM | 1 | 1.9701 | 0.1605 | ||
| Ex1 * TV* Section | 1 | 2.2562 | 0.1331 | ||
| Ex1 * TV | 1 | 0.5233 | 0.4696 | ||
[i] 1–3Order of terms dropped from initial model; †p < 0.05.
Note. Outliers included. n = 71 students.
Figure 2
The Effect of Session Attendance on Final Exam Scores in 2018 for Each Combination of Treatment and URM Status.
Note. TT = traditional tutoring, SI = Supplemental Instruction. Points represent individual student scores. Lines represent predicted final exams scores from the final statistical model at three different initial exam one scores (50%, 70%, and 90%). The solid and dashed lines are predicted from the models with and without the three outlier points (circled in black), respectively. Shading indicates first exam score.
Table 5
Predicted Slopes (With 95% Confidence Intervals) for the Relationship Between Exam 1 Score or Number of Tutoring Sessions Attended and Final Exam Score for the 2018 Experiment.
| SECTION | URM STATUS | EXAM 1 | SESSIONS ATTENDED |
|---|---|---|---|
| Model with outliers included | |||
| Control | Non-URM | 0.346 (–0.001,0.693)a | 3.04 (0.09,5.98)a* |
| SI | Non-URM | 0.477 (0.123,0.830)a* | 0.62 (–1.14, 2.39)a |
| Control | URM | –0.486 (–0.906, –0.067)b* | –7.66 (–10.97, –4.34)b* |
| SI | URM | 0.587 (0.345,0.829)a* | 0.43 (–0.80,1.67)a |
| Model with 3 outliers excluded | |||
| Control | Non-URM | 0.288 (0.091,0.484)a,b* | 1.94 (0.68,3.19)a* |
| SI | Non-URM | 0.292 (0.117,0.467)a,b* | 1.18 (0.59, 1.77)a* |
| Control | URM | –0.587 (–1.410, 0.237)b | –7.97 (–14.66,–1.28)b* |
| SI | URM | 0.663 (0.383,0.943)a* | 0.49 (–1.12,2.10)a,b |
[i] a, bGroups that share letters within a model and variable are not statistically different from each other, based on a Tukey multiple comparison test (p > 0.05).
*Slope is significantly different from zero, based on a t-test (p < 0.05).
In the other three combinations of URM status and treatment, attending support sessions increased final exam scores. Among non-URM students, when the outliers were included, TT students saw more than triple the benefit of attending as SI students (Table 5, Figure 2, solid lines). However, this difference was not statistically significant (t = 1.40, df = 57, p = 0.50) and the effect was reduced by half when the three outliers were removed (Table 5, Figure 2 dashed lines). Within the SI treatment, the benefit of attending sessions was similar for URM and non-URM students (Table 5, Figure 2). However, the slope was only statistically different from zero for the non-URM group, and only after excluding the outliers (Table 5). These three groups also showed a positive association between initial exam score and final exam score, both with and without the outliers present (Table 5).
In contrast to the other three subgroups, the URM/TT students showed reduced final exam scores at higher levels of both Exam 1 score and attendance (Table 5). However, there were only seven students in this subgroup and no student attended more than three sessions.
Question 3: Overall Attendance
To determine if support session attendance differed by treatment, a Χ2 contingency test was used to compare the proportion of students attending sessions between the two programs (SI and tutoring) in each year. Two different response variables were examined: the proportion of students attending at least one session and the proportion attending at least three sessions.
In both years, the SI sessions were better attended than TT. Approximately 30% more students attended ≥ 1 SI session than attended ≥ 1 TT session (2017: χ2 = 5.89, df = 1, p = 0.0152; 2018: χ2 = 5.24, df = 1, p = 0.0221). The same trend was apparent among students who attended ≥ 3 sessions, but the effect was smaller, on the order of 10–14% (2017: χ2 = 0.5668, df = 1, p = 0.4515; 2018: χ2 = 4.30, df = 1, p = 0.0382). However, the majority of students attended fewer than three sessions in all treatments and years (Table 2), suggesting that neither treatment attracted large numbers of students.
Question 4: Predictors of Student Attendance
A multiple linear regression was conducted to determine whether the number of sessions attended was predicted by either treatment or individual student traits. The dependent variable was the number of sessions attended, and the explanatory variables were URM status, motivation score, math placement, number of previous science courses, treatment, and year. To determine whether attendance varied by treatment or year, the interactions of these two terms were included with all other terms in the model. An inverse transformation (1/(Total Visits + 1)) was used to reduce non-normality of the residuals.
Twenty-one students were excluded from the analysis because of missing data for motivation score, math placement, or prior number of science courses (Table 2). Missing data for the last two variables can be considered “missing completely at random,” as they arose from clerical errors. A failure to complete the motivation survey could reflect a lower overall motivation than students who did complete the survey. However, a Fisher’s Exact Test did not reveal a significant difference in attendance frequencies between those who did and did not complete the survey (p = 0.2878), so the students without survey data were excluded from the analysis.
There were no significant associations between attendance and math placement, number of prior science courses, motivation score, or URM status. Only treatment, year, and an interaction between URM status and year were significant (Table 6, Figure 3). When pooled across year and URM status, students attended SI sessions (MSI = 0.994 sessions/student, CI:0.665–1.483) at roughly twice the frequency of TT students (MTT = 0.416 sessions/student, CI:0.244–0.644), although attendance averaged less than one session per student in all treatments. Attendance was also significantly higher overall in 2017 (M2017 = 0.742 sessions/student, 95% CI[0.461,1.155]) than 2018 (M2018 = 0.578 sessions/student, 95% CI[0.352,0.896]). While there was a tendency for non-URM students to attend more sessions than URM students when all data was pooled together, this result was not statistically significant. This pattern reversed in the second year, as indicated by the significant interaction between URM status and year (Figure 3).
Table 6
ANOVA Table for the Number of Sessions Attended.
| MODEL TERM | df | ESTIMATE | STANDARD ERROR | F | P |
|---|---|---|---|---|---|
| URM status | 1 | 0.193 | 0.111 | 3.0027 | 0.0859 |
| Motivation Score | 1 | –0.018 | 0.153 | 1.4330 | 0.2338 |
| Previous Science Courses | 1 | 0.081 | 0.051 | 2.4911 | 0.1174 |
| Treatment | 1 | –0.204 | 0.063 | 10.5364 | 0.0155† |
| Year | 1 | 0.231 | 0.072 | 10.3126 | 0.0017† |
| URM * Year | 1 | –0.343 | 0.155 | 4.9104 | 0.0288† |
| Terms dropped from the final model | |||||
| Math Placement | 1 | 0.0951 | 0.7584 | ||
| URM * Trt | 1 | 0.781511 | 0.3787 | ||
| Mot*Trt | 1 | 0.13928 | 0.7098 | ||
| Math * Trt | 1 | 0.329010 | 0.5674 | ||
| Sci * Trt | 1 | 0.07756 | 0.7813 | ||
| Year * Trt | 1 | 1.454512 | 0.2304 | ||
| Mot* Year | 1 | 0.01105 | 0.9167 | ||
| Math * Year | 1 | 0.69849 | 0.4052 | ||
| Sci * Year | 1 | 0.09907 | 0.7536 | ||
| URM * Year * Trt | 1 | 1.22354 | 0.2713 | ||
| Mo * Year * Trt | 1 | 0.53461 | 0.4664 | ||
| Math * Year * Trt | 1 | 0.73902 | 0.3921 | ||
| Sci * Year * Trt | 1 | 1.22043 | 0.2720 | ||
[i] 1–12Order of terms dropped from initial model; †p < 0.05.
Note. n = 117 students.
Figure 3
Number of Academic Support Sessions Attended by Year, Treatment, and URM Status.
Note. Marginal means (±95CI) from the statistical model are presented in the outlined bars. Points represent individual students. Points are jittered slightly along the x-axis for clarity.
Discussion
Supplemental Instruction is recognized as an important tool for undergraduate student success in STEM courses (Arendale, 2002; Burkholder et al., 2021; Chang et al., 2014; Rath et al., 2012). However, much of the past research into SI has been criticized as flawed (Dawson et al., 2014; McCarthy et al., 1997). This study improved on previous research by using an experimental design that compared two different academic support programs in two sections of a course taught by the same instructor. The results indicate that, on a per session basis, attending SI improved student final exam scores significantly more than TT for all students in the first year of the experiment and for URM students in both years. SI also consistently attracted more students than TT, suggesting that, even where per-session benefits were comparable, the SI program provided a greater aggregate benefit. Overall, these results are consistent with past studies (e.g., Bowman et al., 2021; Guarcello et al., 2017; Stanich et al., 2018) that have documented grade improvement for students attending SI relative to those who did not. However, this study extends beyond previous work to demonstrate that SI can improve student grades over other forms of student academic support and also attracts greater participation by students.
There are two benefits of comparing SI to another academic support program, rather than comparing students who attended SI sessions to those who did not. First it avoids confounding the effects of individual student motivation with the effects of the SI program itself. Because SI attendance is voluntary, it is possible that the students who attended SI would have received higher grades than non-attenders even without SI because they are more motivated to succeed overall (Dawson et al., 2014). The direct comparison of two different tutoring programs avoids this problem. While there may still be differences in motivation between those who attended many support sessions and those who did not, the use of two different types of support sessions made it possible to separate out the effects of motivation from the effects of the sessions themselves. Second, the direct comparison of two different academic support programs should be of more practical use to institutions. Most institutions of higher education in the United States offer some form of academic support or tutoring in introductory sciences. Thus, they are likely more interested in gauging the benefits of SI relative to an existing academic support program rather than relative to no program at all.
The results suggest that SI had greater benefits for students than TT. Students who attended SI sessions had greater final exam scores than those attending the same number of TT session. The greater benefits of SI were most apparent in 2017, where students saw roughly 8 times the benefit of attending an SI session as a drop-in tutoring session. This benefit was also evident for URM students in 2018, but there was not a clear difference for non-URM students in 2018. The 2018 results were complicated by changes to the TT program that year. These included expanded hours and greater emphasis on training tutors, which may have changed students’ experiences in this program. The 2018 analysis was also influenced by three outliers, non-URM students who scored very low on the final exam because they did not complete all exam questions. With the outliers included, the redesigned drop-in tutoring program appeared to benefit non-URM students more than SI; however, upon excluding the outliers, the two treatments had comparable positive effects on final exam grades. Thus, on a per-session basis, attending SI improved final exam grades as much or more than attending drop-in tutoring.
However, the benefits of the SI program increase when also considering the differences in student attendance between the two treatments. SI attracted students at double the rate of TT. This pattern was consistent across both years of the study and did not differ by URM status. It is not known how general this result is, as only one other study (Hodges & White, 2001) has directly compared attendance rates between SI and another academic support program. They found the opposite pattern of greater attendance in the traditional tutoring program, but no statistics were presented, and the two programs were not offered in the same courses. If the greater SI attendance reported here were to be confirmed in other studies, it would suggest an additional mechanism for the success of SI. By attracting greater student attendance, SI can benefit more students, even when the benefits per session attended are comparable to other programs.
However, attendance was low in both support programs. While a majority of the students participated, defined as attending at least one academic support session, less than 40% of the SI treatment and less then 25% of students of the TT treatment attended 3 or more sessions, despite the offer of extra credit. Additionally, attendance frequency was not predicted by motivation, academic preparation, or URM status. Other studies have reported SI participation of as few as one-third of enrolled students (Allen et al., 2019; Hodges et al., 2001; Reittinger et al., 1996) with even fewer attending regularly (Allen et al., 2019; Guarcello et al., 2017; McCarthy et al., 1997). While this study did not directly ask students why they chose not to participate in support sessions, other researchers have identified a range of explanations including scheduling conflicts, motivation, misperception of their own academic needs, and resistance to peer collaboration (reviewed in Allen et al., 2019).
One additional reason for the low attendance in this study may be the nature of the student population involved. The introductory chemistry course is offered in three different versions, two of which require an application: an accelerated section that combines two semesters of content into a single course and a remedial section that includes additional class time. These two sections each constitute roughly 10% or less of the total student enrollment in this course. Importantly, because the two sections require applications, they may have disproportionately attracted highly motivated students. While this could explain part of the low attendance observed, the effect should be small. More likely, attendance was low for the same reasons identified in other studies: scheduling conflicts or misperception of students’ own need for academic support.
Attracting students to support programs remains a fundamental challenge for all such programs. While SI is intended to be a voluntary program, researchers have experimented with incentives to increase attendance, including prize raffles (Paloyo, 2015) or extra credit (Reittinger et al., 1996). Attendance is much higher (>90% of sessions) when SI is offered as a for-credit course linked to the study course (e.g., Hall et al., 2014; Stanich et al., 2018). But this approach excludes students who might not realize they need support until later in the semester. Only rarely has SI been made mandatory. Hodges et al. (2001) replaced 50 minutes of lecture time each week with mandatory SI sessions in groups of 10–15 students. Those students did as well or better than a group of voluntary SI attendees in a different year of the same course. Using class time for SI sessions is essentially another way of incorporating active learning into the classroom, which has frequently been shown to improve student learning and narrow achievement gaps (Freeman et al., 2014; Theobald et al., 2020).
An additional goal of this study was to examine the benefits of different support programs specifically for underrepresented or marginalized students. It is critical to identify academic support programs that benefit URM students, since in the United States, they leave STEM majors at higher rates than non-URM students (Chang et al., 2014; Estrada et al., 2016). I found that URM students, like their non-URM peers, were more likely to attend SI than TT support sessions. URM students also showed greater benefits of attending SI over TT on a per-session basis in both years of the experiment. Thus, it appears that SI is a better program for URM students than traditional tutoring. However, this conclusion should be tempered by small numbers (<10 per treatment per year) of URM students involved in the study. Such small sample sizes can lead to spurious patterns in the results. This issue could explain the negative relationship between session attendance and final exam grade for URM students in the 2018 TT treatment. Other studies have found that SI benefits underrepresented and marginalized students at comparable or higher rates than non-URM students (Rath et al., 2012; Stanich et al., 2018; Yue et al., 2018), although these studies have generally compared attenders to nonattenders rather than comparing across types of academic support programs. Further studies are necessary to determine whether URM students benefit more from SI than they do from other academic support programs.
In summary, this study presented a novel method for testing the effectiveness of Supplemental Instruction that controlled for self-selection bias. The results suggest that SI programs can significantly improve student exam grades through both increased attendance and greater grade improvement per session attended. The SI program also appeared to benefit URM students similarly or more so than non-URM students. However, few students consistently attended sessions of either program, suggesting that encouraging attendance remains a central challenge of any academic support program.
Acknowledgements
D. Drew provided important advice and feedback. I thank S. Kleigman and B. Caulkins for promoting this research project in their courses. I also thank C. Suh, E. Choi, E. CruzGrave, M. Ishii, and D. Ngyen for their work running the SI sessions.
Competing Interests
The author has no competing interests to declare.