Interviewing Peer Leaders: Inter-Reliability of a Growth-Oriented Selection Rubric

Jillian P. Rager; Jaime S. Miller; Joseph M. Kush; Deidre Leslie; Ashley C. Carr

doi:10.21061/jopl.151

Interviewing Peer Leaders: Inter-Reliability of a Growth-Oriented Selection Rubric

Journal of Peer Learning

Volume 17 (2026): Issue 1

By: Jillian P. Rager, Jaime S. Miller , Joseph M. Kush , Deidre Leslie and Ashley C. Carr

Open Access

|Aug 2026

Abstract

Peer leaders are student employees selected by program administrators in higher education to support their fellow classmates and peers in college. This study evaluated the inter-reliability of a selection rubric used by raters during group interviews for a peer leader program, Supplemental Instruction (SI), that maintains a team of over 60 peer leaders each semester. Results confirm that the rubric possesses a substantial degree of reliability, with an overall average agreement percentage of 88%. Researchers discuss reasons for this statistically significant result, including effective and theory-based processes, thorough training, and openness to change and growth. Implications include more emphasis on educators’ process in creating assessment tools, including rubrics, and validation of other selection tools presently used in the field of higher education to determine students’ eligibility for peer leadership roles.

References

Ansley, S., Miller, J. S., Rager, J., Brode, C., & Chaplin, C. (2025). Dignity by design. In J. Tippets & E. Kirby (Eds.), ASCEND to higher retention rates: A collection of best practices and practical strategies from higher education experts for increasing student retention, completion, and student success (Vol. 3). Tippets & Kirby Consulting.
Search in Google Scholar Back to article
Ashenafi, M. M. (2017). Peer-assessment in higher education: Twenty-first century practices, challenges, and the way forward. Assessment & Evaluation in Higher Education, 42(2), 226–251. 10.1080/02602938.2015.1100711
Open DOI Search in Google Scholar Back to article
Berg, S. A., & Stoner, J. C. (2016). Selecting resident assistants: The relationship between candidate assessment and job performance. The Journal of College and University Student Housing, 42, 26–36.
Search in Google Scholar Back to article
Bowman, N. A., Preschel, S., & Martinez, D. (2023). Does Supplemental Instruction improve grades and retention? A propensity score analysis approach. Journal of Experimental Education, 91(2), 205–229. 10.1080/00220973.2021.1891010
Open DOI Search in Google Scholar Back to article
Breslin, J. D., Kope, M. H., O’Hatnick, J. L., & Sharpe, A. G. (2018). Students as colleagues: A paradigm for understanding student leaders in academic support. The Learning Assistance Review, 23(2).
Search in Google Scholar Back to article
Carr, A. C. (2018). Supplemental Instruction Interview Rubric (SIIR). [Unpublished work].
Search in Google Scholar Back to article
Chan, Z., & Ho, S. (2019). Good and bad practices in rubrics: The perspectives of students and educators. Assessment & Evaluation in Higher Education, 44(4), 533–545. 10.1080/02602938.2018.1522528
Open DOI Search in Google Scholar Back to article
Cheung-Blunden, V., & Khan, S. R. (2018). A modified peer rating system to recognise rating skill as a learning outcome. Assessment & Evaluation in Higher Education, 43(1), 58–67. 10.1080/02602938.2017.1280721
Open DOI Search in Google Scholar Back to article
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. 10.1177/001316446002000104
Open DOI Search in Google Scholar Back to article
Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220. 10.1037/h0026256
Open DOI Search in Google Scholar Back to article
Colvin, J. W. & Ashman, M. (2010). Roles, risks, and benefits of peer mentoring relationships in higher education. Mentoring & Tutoring: Partnership in Learning, 18(2), 121–134. 10.1080/13611261003678879
Open DOI Search in Google Scholar Back to article
Culpepper, D., White-Lewis, D., O’Meara, K., Templeton, L., & Anderson, J. (2023). Do rubrics live up to their promise? Examining how rubrics mitigate bias in faculty hiring. The Journal of Higher Education, 94(7), 823–850. 10.1080/00221546.2023.2168411
Open DOI Search in Google Scholar Back to article
Cuseo, J. (2010a). Peer leadership: Definition, description, and classification. E-Source for College Transitions, 7(5), 3–5. https://sc.edu/about/offices_and_divisions/national_resource_center/publications/search/details.php?id=1545
Search in Google Scholar Back to article
Cuseo, J. (2010b). Peer leadership: Situation-specific support roles. E-Source for College Transitions, 7(6), 4–5. https://sc.edu/about/offices_and_divisions/national_resource_center/publications/search/details.php?id=1541
Search in Google Scholar Back to article
Cuseo, J. (2010c). Peer power: Empirical evidence for the positive impact of peer interaction, support, and leadership. E-Source for College Transitions, 7(4), 4–6. https://sc.edu/about/offices_and_divisions/national_resource_center/publications/search/details.php?id=1552
Search in Google Scholar Back to article
Ender, S. C., & Kay, K. (2001). Peer leadership programs: A rationale and review of the literature. In S. L. Hamid (Ed.), Peer leadership: A primer on program essentials (Monograph Series No. 32, pp. 1–11). The National Resource Center for The First-Year Experience and Students in Transition.
Search in Google Scholar Back to article
Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research, 70(3), 287–322. https://www.jstor.org/stable/1170785?seq=1
Search in Google Scholar Back to article
Gannouni, K., & Ramboarison-Lalao, L. (2018). Leadership and students’ academic success: Mediating effects of self-efficacy and self-determination. International Journal of Leadership in Education, 21(1), 66–79, 10.1080/13603124.2015.1123300
Open DOI Search in Google Scholar Back to article
Ganser, S. R., & Kennedy, T. L. (2012). Where it all began: Peer education and leadership in student services. New Directions for Higher Education, 157, 17–29. 10.1002/he.20003
Open DOI Search in Google Scholar Back to article
Gladding, S. (2012). Transactional analysis, reality therapy, Adlerian, and person-centered groups. In S. Gladding (Ed.), Groups: A counseling specialty (6th ed., pp. 321–351). Pearson.
Search in Google Scholar Back to article
Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Advanced Analytics.
Search in Google Scholar Back to article
Hamid, S. L., & VanHook, J. (2001). First-year seminar peer leaders, programs, and profiles. In S. L. Hamid (Ed.), Peer leadership: A primer on program essentials (Monograph No. 32, pp. 37–52). University of South Carolina, The National Resource Center for The First-Year Experience and Students in Transition.
Search in Google Scholar Back to article
Harper, S. R. (2010). An anti-deficit achievement framework for research on students of color in STEM. New Directions for Institutional Research, 2010(148), 63–74. 10.1002/ir.362
Open DOI Search in Google Scholar Back to article
Harper, S. R. (2012). Black male student success in higher education: A report from the National Black Male College Achievement Study. Center for the Study of Race and Equity in Education, University of Pennsylvania. https://rossierapps.usc.edu/facultydirectory/publications/231/Harper%20(2012)%20Black%20Male%20Success.pdf
Search in Google Scholar Back to article
Hicks, D. (2021). Dignity: Its essential role in resolving conflict. Yale University Press. 10.12987/9780300263503
Open DOI Search in Google Scholar Back to article
Hunter, M. S., & Heath, M. M. (2001). The building blocks of the peer leader programs: Recruitment, selection, and training. In S. L. Hamid (Ed.), Peer leadership: A primer on program essentials (Monograph No. 32, pp. 37–52). The National Resource Center for The First-Year Experience and Students in Transition.
Search in Google Scholar Back to article
Irwin, L. N., Miller, J. S., Morgan, K., & Linley, J. (2023). A critical ecological exploration of peer socialization agents’ sense of belonging. Journal of College Student Development, 64(5), 509–523. 10.1353/csd.2023.a911787
Open DOI Search in Google Scholar Back to article
Janson, H., & Olsson, U. (2001). A measure of agreement for interval or nominal multivariate observations. Educational and Psychological Measurement, 61(2), 277–289. 10.1177/00131640121971239
Open DOI Search in Google Scholar Back to article
Janson, H., & Olsson, U. (2004). A measure of agreement for interval or nominal multivariate observations by different sets of judges. Educational and Psychological Measurement, 64(1), 62–70. 10.1177/0013164403260195
Open DOI Search in Google Scholar Back to article
Jarratt, L., Martin-Stanley, C. R. II, Linley, J. L., Mollet, A. L., Anderson, D. L., & Mohebali, M. (2024). Underlife: Peer socialization agents’ resistance to higher education diversity narratives. Journal of Diversity in Higher Education, 17(3), 430–442. 10.1037/dhe0000431
Open DOI Search in Google Scholar Back to article
Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144. 10.1016/j.edurev.2007.05.002
Open DOI Search in Google Scholar Back to article
Keller, P., & Porter, H. D. (2020). A terminological study of peer education in higher education [White paper]. College Reading and Learning Association. https://cdn.ymaws.com/crla.net/resource/resmgr/white_papers/crla_2020_whitepaper_peer_ed.pdf
Search in Google Scholar Back to article
Keup, J., & Skipper, T. (2010, March 21–24). Findings from the 2009 National Survey of Peer Leaders. [Conference session]. American College Personnel Association Annual Convention, Boston, MA, United States.
Search in Google Scholar Back to article
Klein, D. (2018). Implementing a general framework for assessing interrater agreement in Stata. The Stata Journal: Promoting Communications on Statistics and Stata, 18(4), 871–901. 10.1177/1536867X1801800408
Open DOI Search in Google Scholar Back to article
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. 10.2307/2529310
Open DOI Search in Google Scholar Back to article
Leyzberg, D., Lumbroso, J., & Moretti, C. (2017). Nailing the TA interview: Using a rubric to hire teaching assistants. Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education, 128–133. 10.1145/3059009.3059057
Open DOI Search in Google Scholar Back to article
Linley, J. L. (2015). We are (not) all bulldogs: Minoritized peer socialization agents’ critical sensemaking about collegiate contexts [Doctoral dissertation, Michigan State University]. MSU Libraries Digital Repository. https://d.lib.msu.edu/etd/3464
Search in Google Scholar Back to article
Linley, J. L. (2017). We are (not) all bulldogs: Minoritized peer socialization agents’ meaning-making about collegiate contexts. Journal of College Student Development, 58(5), 643–656. 10.1353/csd.2017.0051
Open DOI Search in Google Scholar Back to article
Linley, J. L. (2018). Racism here, racism there, racism everywhere: The racial realities of minoritized peer socialization agents at a historically white institution. Journal of College Student Development, 59(1), 21–36. 10.1353/csd.2018.0002
Open DOI Search in Google Scholar Back to article
McDaniel, A. (2008). Recruiting and training Supplemental Instruction Leaders. In M.E. Stone & G. Jacobs (Eds.), Supplemental Instruction: Improving First-Year Student Success in High-Risk Courses (pp. 39–56). University of South Carolina.
Search in Google Scholar Back to article
Moore, C., & Teather, S. (2013). Engaging students in peer review: Feedback as learning. Issues in Educational Research, 23(2), 196–211.
Search in Google Scholar Back to article
Panadero, E., & Romero, M. (2014). To Rubric or Not to Rubric? The Effects of Self-Assessment on Self-Regulation, Performance and Self-Efficacy. Assessment in Education: Principles, Policy & Practice, 21(2), 133–148. 10.1080/0969594X.2013.877872
Open DOI Search in Google Scholar Back to article
Perez, D., Ashlee, K., Do, V., Karikari, S., & Sim, C. (2017). Re-conceptualizing student success in higher education: Reflections from graduate student affairs educators using anti-deficit achievement framework. Journal on Excellence in College Teaching, 28(3), 5–28.
Search in Google Scholar Back to article
Popham, W. J. (1997). What’s wrong – and what’s right – with rubrics. Educational Leadership, 55(2), 72–75. https://skidmore.edu/assessment/handbook/Popham_1997_Whats-Wrong_and-Whats-Right_With-Rubrics.pdf
Search in Google Scholar Back to article
Rager, J. P., & Leslie, D. (2023). Adapted Supplemental Instruction Interview Rubric (SIIR). [Unpublished work].
Search in Google Scholar Back to article
Reddy, Y. M., & Andrade, H. (2010). A review of rubric use in higher education. Assessment & Evaluation in Higher Education, 35(4), 435–448. 10.1080/02602930902862859
Open DOI Search in Google Scholar Back to article
Reyes, H. L., & Duran, A. (2021). Higher education scholars challenging deficit thinking: Ananalysis of research informed by community cultural wealth. Journal of Critical Scholarship on Higher Education and Student Affairs, 6(1), 7–20.
Search in Google Scholar Back to article
Rogers, C. R. (1961). On becoming a person: A therapist’s view of psychotherapy. Houghton Mifflin.
Search in Google Scholar Back to article
Sambamurthy, N., & Cox, M. F. (2016, June), Developing a Behavior-Based Rubric for Assessing Professional Skills During Hiring. [Paper presentation]. 2016 ASEE Annual Conference & Exposition, New Orleans, LA, United States. 10.18260/p.27288
Open DOI Search in Google Scholar Back to article
Shook, J., & Keup, J. (2012). The benefits of peer leader programs: An overview from the literature. New Directions for Higher Education, 2012(157), 5–16. 10.1002/he.20002
Open DOI Search in Google Scholar Back to article
Skalicky, J., Speed, H., van der Meer, J., & Young, D. G. (2024). An international snapshot of peer leadership in higher education. Journal of Leadership Education, 24(1), 84–101. 10.1108/JOLE-02-2024-0031
Open DOI Search in Google Scholar Back to article
Taylor, B., Kisby, F., & Reedy, A. (2024). Rubrics in higher education: An exploration of undergraduate students’ understanding and perspectives. Assessment & Evaluation in Higher Education, 49(6), 799–809. 10.1080/02602938.2023.2299330
Open DOI Search in Google Scholar Back to article
The International Center for Supplemental Instruction. (2019). SI Supervisor manual.
Search in Google Scholar Back to article
The International Center for Supplemental Instruction. (2022). What is Supplemental Instruction? https://info.umkc.edu/si/
Search in Google Scholar Back to article
van der Meer, J., Skalicky, J., Speed, H., & Young, D. G. (2022). Focusing on the development of the whole student: An international comparative study of the perceived benefits of peer leadership in higher education. Open Journal of Social Sciences, 10, 14–35. 10.4236/jss.2022.103002
Open DOI Search in Google Scholar Back to article
Witty, M. C., & Adomaitis, R. (2014). Carl Rogers and client-centered counseling. In R. D. Parsons & N. Zhang (Eds.), Counseling theory: Guiding reflective practice (pp. 171–199). Sage. 10.4135/9781483399621.n7
Open DOI Search in Google Scholar Back to article
Zhang, B., Johnston, L., & Kilic, G. B. (2008). Assessing the reliability of self- and peer rating in student group work. Assessment & Evaluation in Higher Education, 33(3), 329–340. 10.1080/02602930701293181
Open DOI Search in Google Scholar Back to article

Articles in this issue

DOI: https://doi.org/10.21061/jopl.151 | Journal eISSN: 2200-2359

Journal RSS Feed

Language: English

Page range: 4 - 4

Submitted on: Jul 31, 2025

Accepted on: Jan 5, 2026

Published on: Aug 11, 2026

Published by: Virginia Tech Publishing

In partnership with: Paradigm Publishing Services

© 2026 Jillian P. Rager, Jaime S. Miller, Joseph M. Kush, Deidre Leslie, Ashley C. Carr, published by Virginia Tech Publishing
This work is licensed under the Creative Commons Attribution 4.0 License.

Volume 17 (2026): Issue 1