Table 1
Description of columns.
| COLUMN | DATA TYPE | DESCRIPTION |
|---|---|---|
| User ID | string | Unique ID for each student |
| User email | string | Student email |
| Assistant name | string | AI assistant name (specific to a single academic course) |
| Conversation ID | string | Unique conversation ID |
| Conversation Created At | timestamp | Timestamp indicating when first message of conversation was created (UTC) |
| Message ID | string | Unique message ID |
| Message Created At (UTC) | timestamp | Timestamp indicating when message was created (UTC) |
| Message Created At (PT) | timestamp | Timestamp indicating when message was created (Pacific Time) |
Table 2
Summary of Statistical Methods.
| STATISTICAL METHOD | PURPOSE |
|---|---|
| Mann-Whitney U-Test | Compares GPA distributions between groups |
| Permutation Test (using all data) | Validates GPA differences |
| Propensity Score Matching | Controls for confounders |
| Wilcoxon Signed-Rank Test | Examines GPA changes in matched pairs |
| Permutation Test (on PSM matched pairs) | Validates GPA differences |
Figure 1
GPA Distributions for Control (Did Not Use Spark) and Treatment (Did Use Spark) Groups.
Figure 2
Distribution of Permuted GPA Differences for Matched Student-Course Pairs.
| ANALYSIS | TEST STATISTIC | p-VALUE | EFFECT SIZE | MEAN DIFFERENCE |
|---|---|---|---|---|
| Mann-Whitney U-Test (Before PSM) | U = 307562.5 | 0.00185 | 0.553 | – |
| Permutation Test (Before PSM) | – | 0.001 | – | 0.226 |
| Wilcoxon Signed-Rank Test (After PSM) | W = 10027.5 | 0.0002583 | 0.262 | – |
| Permutation Test (After PSM) | – | 0.0005 | – | 0.377 |
Table 3
Standardized Mean Differences at Different Stages of Propensity Score Matching.
| COVARIATE | SMD BEFORE MATCHING | SMD AFTER MATCHING | SMD AFTER AGE DIFFERENCE FILTERING |
|---|---|---|---|
| Age | 0.3886 | 0.4333 | 0.0749 |
| Gender (Male) | –0.1332 | 0.0193 | –0.041 |
| Ethnicity (Asian) | 0.0214 | –0.0125 | –0.0185 |
| Ethnicity (Black or African American) | –0.0532 | 0.0453 | 0.0125 |
| Ethnicity (Hispanics of any race) | 0.0868 | 0.0195 | –0.044 |
| Ethnicity (Native Hawaiian or Other Pacific Islander) | 0.0137 | –0.0271 | –0.0315 |
| Ethnicity (Race and Ethnicity unknown) | –0.0786 | –0.0514 | 0.0939 |
| Ethnicity (Two or more races) | –0.108 | –0.0969 | 0.1426 |
| Ethnicity (White) | 0.0171 | 0.0326 | 0 |
Figure 3
Absolute Standardized Mean Differences at Different Stages of Propensity Score Matching.