Data from the Effects of Congruent and Incongruent Perceptual Cues on Middle Schoolers’ Mathematical Performance, Learning, and Retention Study

1.1. Background

Solving order-of-operations problems, which involve mathematical expressions with multiple operations (e.g., 4 – 7 × 2 + 9), is a crucial part of pre-algebra and algebra in middle school (Norton & Cooper, 2001), laying the groundwork for students’ future math learning in high school. However, many middle school students struggle with these types of problems, particularly when the order-of-operations rules preclude solving the problem by performing calculations from left to right (e.g., 3 + 4 × 2; Bye et al., 2024). Building on Gibson’s (1969) perceptual learning theory and Bjork’s (1994) desirable difficulties, we developed an online intervention incorporating perceptual cues that highlighted the order of operations within worked examples and practice problems. The present study investigates the impact of this intervention on students’ math performance and learning outcomes in regular classroom settings, as well as the role of student-level characteristics in the intervention’s effects.

1.2. Aims of the project

This research project investigated the individual and combined effects of two perceptual cues—spacing and color—under varying conditions of congruence with the order of operations. In our previous work, we examined whether and how congruent perceptual cues (i.e., spacing and color) affected students’ performance on order-of-operations problems. We previously found that students with a higher perception of equivalence benefited more from congruent spacing cues than those with a lower perception of equivalence (Hornburg et al., 2025). Extending our prior research on the benefits of perceptual cues on mathematics performance, we integrated the perceptual cues into worked examples to examine their influence on students’ immediate problem-solving performance, short-term learning, and long-term retention on order-of-operation problems (e.g., 4 – 7 × 2 + 9). The overall goals of the project were to:

1. Explore whether incongruent perceptual cues could provide desirable difficulties leading to improved learning and retention on order-of-operations problems

2. Identify whether and how combining multiple perceptual cues affects problem-solving performance, learning, and retention on order-of-operations problems within online contexts

3. Examine whether the effects of these perceptual cues on performance, learning, and retention on order-of-operations problems varied based on individual student characteristics

Using a 3×3 factorial design, we tested the effects of perceptual cues (spacing, color, or no cues) and cue congruence modes (congruent, neutral, or incongruent with the order of operations) while also exploring potential moderation effects of individual student characteristics. Based on prior literature, we predicted that (1) all students would demonstrate learning gains from pretest to posttest after the intervention, (2) congruent cues would support students’ intervention performance and short-term learning, whereas incongruent cues would present challenges that promote deeper understanding, leading to greater long-term retention, and (3) student characteristics would interact with perceptual cue conditions to moderate the effects on students’ performance, learning, and retention.

To address these study aims and hypotheses, the data reported here were collected through a randomized controlled trial (RCT) conducted in 2024 in a suburban school district in the southeastern U.S. The datasets contain information about 1,100 6th-grade school students, including their demographics, pre- and post-assessments, and performance during the intervention. Note that the full study description and findings are reported in another paper (Closser et al., under review). All intervention data are fully de-identified and stored on a project page on Open Science Framework (OSF) at https://osf.io/d3tzq. Additional data that include student demographics and assessments are available through a data sharing agreement (DSA).

2.1 Study design

Students participated in four 30-minute sessions (see Table 1) over 9.63 weeks on average (SD = 1.42 weeks, Max = 11.90 weeks, Min = 6.43 weeks). Each session occurred approximately 1–2 weeks after the previous session. To accommodate varying classroom needs, some flexibility in this timeframe was permitted, resulting in small differences in the duration between sessions (Session 1-Session 2 (2.49 ± 0.94 weeks), Session 2-Session 3 (4.41 ± 1.30 weeks), and Session 3-Session 4 (2.73 ± 1.15 weeks).

During Session 1, participants’ baseline knowledge of the order of operations was assessed, as well as student-level characteristics, including postdiction of performance, perceptual processing skills, and math attitudes (math anxiety and math value).

In Session 2, students were randomly assigned at the student level to one of nine experimental conditions based on the 3x3 research design that varied the combinations of color and spacing-based cues (congruent, neutral, or incongruent for each) (see Table 2) and completed an experimental math intervention. As shown in Table 2, problems with congruent color cues used an orange font to highlight the higher-order operator (multiplication or division) and its operands, aligning with the order of operations. Neutral spacing cues used uniform spacing throughout. Congruent spacing cues removed spacing between the higher-order operator and its operands, while maintaining uniform spacing elsewhere to align with the order of operations. In contrast, incongruent spacing cues removed spacing between the lower-order operators (addition and subtraction), while keeping uniform spacing elsewhere. During the intervention, participants solved 21 order-of-operations problems (one warm-up and 20 intervention problems) in their assigned condition. After answering each problem, participants received feedback on their accuracy and viewed a correct worked example of the problem.

Session 3 consisted of nine problems completed in the same experimental intervention format as Session 2, followed by an immediate posttest consisting of ten order-of-operations problems with no perceptual cues or correctness feedback provided, as well as two postdiction of performance items.

In Session 4, participants completed a delayed posttest, consisting of 10 order-of-operations problems, two postdiction of performance items, and 12 perceptual processing items. The 10 order-of-operations problems completed in the immediate and delayed posttests were identical to those completed during the pretest. Details for all intervention tasks, conditions, and assessments can be found in the Materials/Survey Instruments section.

2.2 Time of data collection

Data were collected between August 2024 and November 2024.

2.3 Location of data collection

Data were collected from one large suburban school district in the southeastern U.S.

2.4 Sampling, sample, and data collection

The student-level dataset contains 1,110 sixth graders (typically 11–12 years old) from 16 teachers in nine middle schools, within one school district. Our team had an established relationship and memorandum of understanding set up with the specific school district. We recruited sixth-grade teachers via an email sent from the middle school math specialist including a flier describing the study and a link to an interest form for teachers to complete. All schools with at least one teacher interested in participating were included in recruitment. Students in the math classrooms of participating teachers were included in the study. Parents had the opportunity to opt their child out of the study and also participating students had the opportunity to not assent for our team to use their data in this research study.

Sixth-grade students in classes of participating teachers participated in this experimental study online during their math class time. Participants accessed study tasks on their school-provided devices via links provided to them by their teachers. Teachers who opted to participate in the study received email instructions and the links before each session on how to assist students with the session assignment. The school district provided demographic information for the 1,011 participating students. Table 3 shows the demographic information for the students.

2.5 Materials/Survey Instruments

All intervention materials and assessments were built using custom jsPsych plugins (De Leeuw et al., 2023) and hosted on Pavlovia (https://pavlovia.org), which supports online experiments and was used to provide intervention materials to participants remotely. Sample study materials and items from the research study can be found in the materials folder on OSF at https://osf.io/d3tzq.

2.5.2. Experimental materials

Practice Problems and Worked Examples (Sessions 2 and 3). Students completed 21 experimental practice problems (including one warm-up problem) in Session 2 and nine problems in Session 3. Each problem was presented using the perceptual cues for a given participant’s assigned condition (see Table 2). These problems were constructed as order-of-operation problems with three operators. Each problem included one addition and one subtraction operator. The third operator was either multiplication (Session 2: 18 problems; Session 3: 6 problems) or division (Session 2: 3 problems; Session 3: 3 problems). The third operator (multiplication or division) was counterbalanced between the left, middle, and right positions. Parentheses and exponents were not included in any problems in this study.

Problems were presented one at a time via a screen progression of: Screen 1: the problem is presented and the student enters their response; Screen 2: student is shown their response alongside the correct response and asked to evaluate their answer (“How did you do?”; immediate correctness feedback); Screen 3: student is shown a step-by-step worked example of the problem. See Figure 1 for examples of these screen displays.

Figure 1

Students completed the first problem in Session 2 prior to any instruction on the order of operations. This problem was presented to students as a “warmup” problem with the instructions: “In this section, you will solve a series of math problems. Try your best to simplify each expression. Answers could be positive or negative numbers.” After entering their response, students were shown the correct response and worked example screens as described above. Session 3 followed the same format but did not include a warmup problem.

2.6 Quality Control

Our datasets meet data quality standards in terms of quantity, quality, and accuracy. They include a large amount of student data (N = 1,110) collected through an RCT conducted in a large school district. To ensure data accuracy, students were required to enter only numeric responses to move on to the next screen during problem-solving tasks. To maintain data quality, our research team de-identified and pre-processed the dataset. We first handled the entry errors in students’ IDs (PID) and teachers’ names. For example, some students incorrectly completed the fields for their PIDs (e.g., 123456) or teachers’ names (e.g., “I don’t want to tell you”). Additionally, a student may input the PID correctly in sessions 1, 3, and 4 (e.g., 137579) but incorrectly in session 2 (e.g., mistype 137579 as 137570). To handle such cases (n = 10), our research team manually 1) checked with the school district on whether 137570 is a valid PID, 2) confirmed that the PID 137570 and PID 137579 are associated with the same teacher, and 3) confirmed that except for 137570, no other similar PID (e.g., 137578) can compensate for the missing session associated with the PID 137579. Then, we replaced student IDs used in the school district with randomly generated IDs consisting of EHR followed by four-digit numbers (e.g., EHR0001). Quality control efforts also included making design decisions that were informed by lessons learned from a prior study (Hornburg et al., 2025), where our research team collected student performance and teacher feedback for a sample of similar order-of-operations problems. Changes made based on the prior study included changing the online platform used for the study and adjusting verbiage to improve the clarity of instructions.

2.7 Data anonymisation and ethical issues

This study was approved by the Institutional Review Board (IRB) at each affiliated university, and the primary institution oversees the study under the “research conducted in established educational settings” exemption. Teachers provided informed consent to participate in the study. Prior to giving their informed consent, teachers were given an overview of the study requirements and time commitment. A letter was sent home to parents providing information about the study and the contact information for the district administrator, who was our primary contact for this project. The letter also informed parents that to opt their child out of participating, they needed only to email the district administrator. A study overview with screenshots of the types of questions their child would be responding to was provided to the district to share with parents who preferred additional information about the study before choosing whether to opt their child out. On the first day of the study, students were provided with a written description of the study activities and purpose, and they gave their assent to participate. Students who did not provide assent were given a non-research math activity by their teacher. At the beginning of the study, each student was assigned a de-identified participant identification number. Students’ names were never recorded.

2.8 Existing use of data

Closser, A. H., Bye, J. K., Zhang, P., Lee, J., Egorova, A., Colbert, M. A., Yu, S., Hornburg, C. B., & Ottmar, E. (under review). The effects of congruent and incongruent perceptual cues on middle schoolers’ mathematical performance, learning, and retention.

Hornburg, C. B., Zhang, P., Closser, A. H., Bye, J. K., Lee, J.-E., Egorova, A., Colbert, M. A., Yu, S., & Ottmar, E. (2025, June). Understanding the effects of perceptual cues on middle schoolers’ mathematical performance, learning, and retention. In A. H. Closser (Chair), Leveraging cognitive principles in low-cost interventions to improve mathematics problem solving and learning. Paper to be presented at the 8th Annual Meeting of the Mathematical Cognition and Learning Society (MCLS), Hong Kong, China.

In these papers, we addressed the project aims outlined in Section 1.2 and tested the unique and combined effects of spacing and color cues across three modes of congruence with the order of operations, as well as the moderating effects of student characteristic variables. Specifically, we used student-level assessment data (i.e., pretest, perception of equivalence skills, posttest, delayed posttest, and math anxiety), performance data from the intervention, and demographic data to address our research questions. For the data analysis, we re-ran the models using the Neutral Spacing and Neutral Color (NSNC) condition as the reference group, comparing performance, learning, and retention across eight dummy-coded experimental groups to make the results easier to interpret and to enable clearer comparisons across conditions. See Closser et al. (under review) for more details on the data analyses and results.

3.1 Repository location

Information and files for the study description, assessment materials, and data dictionary describing the data are located on the OSF website (DOI 10.17605/OSF.IO/D3TZQ, URL: https://osf.io/d3tzq/) and can be accessed on the project repository. A subset of the data, including the experimental intervention items, is also available on this OSF page. The full dataset that includes the students’ assessments and demographic information requires the data sharing agreement and is stored on a second OSF repository and is available with a separate link that is emailed to the researcher once the DSA is received. For access to the full dataset, data sharing agreement forms and instructions are also available on this page. To access and download the full data files, including student demographic and assessment data, researchers must complete the Data Sharing Agreement (DSA) form and email their signed agreements to the project team (described in more detail in section 3.7 below). The additional step is required by the IRB in order to ensure that the de-identified data files can be available for sharing while meeting ethical requirements. Once the request process is completed, a link to the actual data files will be sent to the researcher by the project team.

3.2 Object/file name

As described above, all intervention data are openly available on the OSF repository (https://osf.io/d3tzq/). This open dataset consists of two separate files: problem-level data from intervention problems in Session 2 (training_1.csv) and problem-level data from Session 3 (training_2.csv). Table 4 provides a description of each variable included in the open intervention dataset.

The second dataset that requires the DSA includes students’ assessment and demographic information. It is organized into two student-level datasets and 11 problem-level datasets, totalling 13 separate files. The contents within each file are described in Table 5.

The student-level datasets consist of two files: (1) data on student pretest, intervention, posttest, and delayed posttest (“student_all” in Table 5), and (2) data on their demographics (“student_demo” in Table 5). Student data (student_all) provide overall measures of how well each student performed on each task or how they self-reported their perceptions. Demographic data (student_demo) were provided by the school district and include information on students’ biological sex, race/ethnicity, English as a Second Language (ESL) status, their 5th-grade state assessment math score, and other relevant variables. Their 5th-grade state assessment scores include overall numeric scores as well as seven performance levels that were provided by the district (i.e., below target, approaching target, met target) for subdomains of mathematics, including numerical reasoning, pattern and algebraic reasoning, measurement and data reasoning, and geometric and spatial reasoning.

The problem-level dataset is structured by the data collection time points (e.g., pre, post) as well as the task type (e.g., knowledge of the order of operations, perception of equivalence skills, math anxiety). Hence, these data provide detailed information on how well each student performed on each problem and how they self-reported their perceptions of each item. Details about other measures included in each file are documented in the data descriptions on OSF (https://osf.io/d3tzq/).

3.3 Data type

These datasets contain primary data from the randomized controlled trial. They include various types of data, including students’ demographics, assessment, and performance during the intervention, collected from multiple sources. The datasets comprise four types of data: raw clickstream data, pre-processed data, students’ self-reported data, and data provided by the school district.

The problem-level data files are derived from the raw clickstream data extracted from the Pavlovia platform, where the study was hosted and conducted. Student data on pretest, intervention, posttest, and delayed posttest were pre-processed and aggregated from the problem-level log data. Assessment data on math anxiety, math value, helpfulness ratings, and postdiction of performance were self-reported by students on the platform. All demographic data were provided by the students’ school district.

3.4 Format names and versions

All 13 data files are in the comma-separated value (CSV) file format. Each file contains unique anonymous identifiers of students (e.g., EHR0001), sessions (e.g., S1, S2), or problems. Researchers can merge multiple layers of data using these identifiers for specific data analyses. For example, to investigate demographic differences in student performance on intervention problems, a researcher could merge the “student_all” and “student_demo” files using the student identifier (i.e., stu_ID). Alternatively, to examine how different types of problems affect student performance on intervention problems, they would merge the “problem_meta”, “training_1”, and “training_2” files using the student identifier as well as the problem identifier (i.e., stim_num).

3.5 Language

The data are stored in American English.

3.6 License

The data descriptions, study description, a subset of intervention data, and request procedures and forms to access the full study data are available on OSF and are available under a CC0 license. However, due to restrictions imposed by our institution’s IRB and the Memorandum of Understanding (MOU) with the participating district, the full data cannot be made completely open and are not under a CC0 license (i.e., no copyright reserved). However, access to the full deidentified data from this study is publicly available on OSF with a signed DSA. The details of the DSA process are described on the project OSF page, and the project Principal Investigator (PI) at the primary institution manages the approval process.

3.7 Limits to sharing

As mentioned above, a subset of the data that includes problem-level responses and condition assignment for each student is openly available for researchers to download on OSF (https://osf.io/d3tzq/). However, the full dataset containing assessment and demographic data is only available to researchers who have completed the DSA process with the PI’s university. The DSA is required by the PI’s institution’s IRB as well as the signed MOU with the school district. As student-level data are protected under the Family Educational Rights and Privacy Act (FERPA) in the U.S., the PI’s institution requires researchers to sign an agreement stating that they will not share the data with others.

To access the full dataset, researchers must complete a DSA form (i.e., data sharing agreement) available on our OSF page (https://osf.io/d3tzq/) and submit it to the project PI along with evidence of an ethics training certificate. One commonly used example of such a training is the Collaborative Institutional Training Initiative (CITI) training (Human Subjects in Social & Behavioral Research course) (https://about.citiprogram.org). CITI training on the protection of human subjects in research is required by university IRBs for most faculty, staff, and students in the United States before conducting any research, is available to anyone aged 18 and older, and takes approximately 2–3 hours to complete. After completion, CITI provides a certificate that can be downloaded and sent to the IRB as evidence of ethics training. For researchers in institutions or countries where the CITI program is unavailable or not required, any equivalent ethics certificate or evidence of human subjects training is accepted. Once the signed DSA forms and ethics certificates are submitted to the PI, the researchers will receive an emailed link to access the full de-identified datasets. The estimated processing time for the DSA is 2 to 3 business days. In the case that the PI changes institutions, the OSF link will be updated with the new contact information and forms.

3.8 Publication date

The open intervention datasets and study materials were added on 7/15/2025. Public DSA Request processes and forms for the full datasets were added on 5/22/2025. The full DSA-approved dataset was also added to the second OSF page on 5/22/2025.

3.9 FAIR data/Codebook

Our dataset meets FAIR data standards, ensuring it is findable, accessible, interoperable, and reusable (Wilkinson et al., 2016).

Findability: The data are easily findable on OSF and include unique identifiers of students, sessions, or problems, which allows researchers to merge multiple layers of data using these identifiers for specific data analyses. The content and description of each column in all data files are clearly documented in our data dictionary, which is available on a searchable webpage on OSF (https://osf.io/d3tzq).

Accessibility: The intervention data and study materials are easily accessible through a link on OSF, and the protocol for accessing the full data is free and universally open-sourced through the authorization of our university required DSA protocol. Metadata (i.e., data dictionary) will remain available on OSF even after the intervention study is no longer accessible due to MOU agreements and the termination of the IRB approval.

Interoperable: The datasets are saved in CSV format with multiple layers and can be effectively analyzed using various statistical or data analysis packages. Our metadata (i.e., data dictionary) provides column names, descriptions, data types (e.g., character, categorical, continuous), categorical values with corresponding codes (e.g., No = 0, Not really = 1, Kind of = 2, Yes = 3) as well as data sources, ensuring clarity and interpretability for other researchers.

Reusable: The dataset is readily reusable, given its large sample size, comprehensive measurements, and systematic procedures for the manipulation of perceptual cues. Please see the section below titled “Reuse potential” for additional details regarding reusability.

4.1 Strengths of the Dataset

This dataset possesses great potential for reuse in educational research, given its large sample size, comprehensive measurements, and systematic manipulation of perceptual cues. First, this dataset includes over 1,000 middle school students with comprehensive measurements, such as demographic information provided by the school district, math problem-solving measures, as well as self-ratings of math anxiety and math value. Second, this dataset records students’ performance during pretest, intervention with perceptual cues, posttest, and delayed posttest, which allows researchers to investigate the impact of perceptual cues on immediate problem-solving performance, math learning, and long-term retention. Moreover, two visual attributes (i.e., color and spacing) of math expressions were manipulated, resulting in a 3×3 factorial design, i.e., three levels of color cue (congruent, neutral, incongruent) × three levels of spacing cue (congruent, neutral, incongruent). This 3×3 factorial design allows the investigation of how different combinations of perceptual cues impact problem-solving, learning, and retention.

4.2 Secondary Analysis

This dataset can be used to conduct multiple secondary analyses focusing on math learning and perceptual cues. For example, by analyzing student-level data, researchers can explore how individual differences (e.g., 5th-grade numerical reasoning, pattern and algebra reasoning, measurement and data reasoning, geometric and spatial reasoning scores) influence the learning of operational rules under different perceptual cues. Additionally, as the position of the high-order operator in the order-of-operations problems is manipulated, the problem-level dataset can be used to examine how different locations of higher-order operators (e.g., left, middle, or right) impact problem-solving performance and how these interact with different perceptual cues (e.g., Closser et al., 2024). In terms of data analysis methods, further research can apply more sophisticated educational data mining techniques. For example, students’ intervention performance can be used to train knowledge-tracing models to further understand how perceptual cues impact math learning (i.e., hidden knowledge states) and immediate problem-solving performance (i.e., guessing or slipping rate) as intervention proceeds. Finally, after the perceptual cue-involved intervention, students rated the helpfulness of worked examples. Researchers can also explore how perceptual cues and students’ characteristics (e.g., prior knowledge, math anxiety, or in-class math performance) influence their rating scores on the helpfulness of worked examples. In sum, through these potential secondary analyses, this dataset can provide insight into the application of perceptual scaffolding in math education, as well as the way of tailoring perceptual scaffolding based on individual characteristics.

4.3 Theoretical Implications

The current dataset can promote the understanding of the perceptual learning theory in math (Closser et al., 2022; Goldstone et al., 2017; Kellman et al., 2010). Previous literature has commonly manipulated spatial (Braithwaite, 2016; Jiang et al., 2014; Landy et al., 2008; Landy & Goldstone, 2010) or color proximity (Alibali et al., 2018) of mathematical terms to create perceptual cues during math problem-solving or math intervention. Our dataset includes the manipulation of color and spacing cues, which are two different visual attributes and may influence math reasoning differently. For example, unbalanced spatial proximity can spontaneously appear in the handwriting of math expressions (Landy & Goldstone, 2007), while the color of mathematical terms within one math expression is relatively less likely to differ. Moreover, spatial processing and math processing are highly overlapped (for a review, see Hawes & Ansari, 2020). Thus, the effect of spacing cues may reflect students’ perceptual learning driven by daily math problem-solving experience apart from bottom-up perceptual grouping, while the effect of color cues may be relatively limited to the perceptual grouping mechanism. Studying the difference between spatial and color cues, as well as the individual differences in the reliance on spatial and color cues, illuminates the associations between low-level perceptual pathways and high-level math reasoning.

4.4 Potential Collaboration and Validation

Our datasets can serve as a means for new collaborations between researchers and the project team for replication research or follow-up research. For example, future studies focusing on perceptual cues can refer to this study. Since the perceptual cue condition was manipulated between subjects, researchers can select subgroups of students from certain cue conditions (e.g., CSNC versus NSNC versus ISNC) to examine the effect size of one type of perceptual cue (e.g., spacing). In addition, this dataset can serve as a control condition for future studies that aim to improve perceptual cue-based intervention, such as providing personalized perceptual cues adapted to students’ learning progress. Moreover, collaborators who are interested in designing theory-based educational tools can use this dataset to understand the effects of different perceptual scaffolding (e.g., congruent spacing or congruent color) on problem-solving and math learning.

4.5 Limitations

Despite containing rich assessments and a large sample size, our dataset still exhibits several limitations. One major limitation is the presence of missing data. Of the 1,110 students, only 537 (48.38%) of them completed all assessments across four sessions without any missing data points. The low completion rate may be partly due to the extended data collection period, which spanned on average of 9.62 weeks (SD = 1.43 weeks). Another critical issue resulting in missing values is the entry errors in students’ PIDs and teachers’ names. Since the PID was used as the identifier of each student, an error in the PID led to the failure of matching students across sessions. A second drawback concerns the low inter-item reliability of the perceptual processing skills assessment (i.e., PMET), which may be due to the limited number (n = 12) of items included in the test or their varied formats. Additionally, we employed the same set of order-of-operations problems in the pretest, posttest, and delayed posttest, which may potentially introduce the memory effect on the posttest performance.