Interviewing Peer Leaders: Inter-Reliability of a Growth-Oriented Selection Rubric
References
- Ansley, S., Miller, J. S., Rager, J., Brode, C., & Chaplin, C. (2025).
Dignity by design . In J. Tippets & E. Kirby (Eds.), ASCEND to higher retention rates: A collection of best practices and practical strategies from higher education experts for increasing student retention, completion, and student success (Vol. 3). Tippets & Kirby Consulting. - Ashenafi, M. M. (2017). Peer-assessment in higher education: Twenty-first century practices, challenges, and the way forward. Assessment & Evaluation in Higher Education, 42(2), 226–251. 10.1080/02602938.2015.1100711
- Berg, S. A., & Stoner, J. C. (2016). Selecting resident assistants: The relationship between candidate assessment and job performance. The Journal of College and University Student Housing, 42, 26–36.
- Bowman, N. A., Preschel, S., & Martinez, D. (2023). Does Supplemental Instruction improve grades and retention? A propensity score analysis approach. Journal of Experimental Education, 91(2), 205–229. 10.1080/00220973.2021.1891010
- Breslin, J. D., Kope, M. H., O’Hatnick, J. L., & Sharpe, A. G. (2018). Students as colleagues: A paradigm for understanding student leaders in academic support. The Learning Assistance Review, 23(2).
- Carr, A. C. (2018). Supplemental Instruction Interview Rubric (SIIR). [Unpublished work].
- Chan, Z., & Ho, S. (2019). Good and bad practices in rubrics: The perspectives of students and educators. Assessment & Evaluation in Higher Education, 44(4), 533–545. 10.1080/02602938.2018.1522528
- Cheung-Blunden, V., & Khan, S. R. (2018). A modified peer rating system to recognise rating skill as a learning outcome. Assessment & Evaluation in Higher Education, 43(1), 58–67. 10.1080/02602938.2017.1280721
- Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. 10.1177/001316446002000104
- Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220. 10.1037/h0026256
- Colvin, J. W. & Ashman, M. (2010). Roles, risks, and benefits of peer mentoring relationships in higher education. Mentoring & Tutoring: Partnership in Learning, 18(2), 121–134. 10.1080/13611261003678879
- Culpepper, D., White-Lewis, D., O’Meara, K., Templeton, L., & Anderson, J. (2023). Do rubrics live up to their promise? Examining how rubrics mitigate bias in faculty hiring. The Journal of Higher Education, 94(7), 823–850. 10.1080/00221546.2023.2168411
- Cuseo, J. (2010a). Peer leadership: Definition, description, and classification. E-Source for College Transitions, 7(5), 3–5.
https://sc.edu/about/offices_and_divisions/national_resource_center/publications/search/details.php?id=1545 - Cuseo, J. (2010b). Peer leadership: Situation-specific support roles. E-Source for College Transitions, 7(6), 4–5.
https://sc.edu/about/offices_and_divisions/national_resource_center/publications/search/details.php?id=1541 - Cuseo, J. (2010c). Peer power: Empirical evidence for the positive impact of peer interaction, support, and leadership. E-Source for College Transitions, 7(4), 4–6.
https://sc.edu/about/offices_and_divisions/national_resource_center/publications/search/details.php?id=1552 - Ender, S. C., & Kay, K. (2001).
Peer leadership programs: A rationale and review of the literature . In S. L. Hamid (Ed.), Peer leadership: A primer on program essentials (Monograph Series No. 32, pp. 1–11). The National Resource Center for The First-Year Experience and Students in Transition. - Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research, 70(3), 287–322.
https://www.jstor.org/stable/1170785?seq=1 - Gannouni, K., & Ramboarison-Lalao, L. (2018). Leadership and students’ academic success: Mediating effects of self-efficacy and self-determination. International Journal of Leadership in Education, 21(1), 66–79, 10.1080/13603124.2015.1123300
- Ganser, S. R., & Kennedy, T. L. (2012). Where it all began: Peer education and leadership in student services. New Directions for Higher Education, 157, 17–29. 10.1002/he.20003
- Gladding, S. (2012).
Transactional analysis, reality therapy, Adlerian, and person-centered groups . In S. Gladding (Ed.), Groups: A counseling specialty (6th ed., pp. 321–351). Pearson. - Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Advanced Analytics.
- Hamid, S. L., & VanHook, J. (2001).
First-year seminar peer leaders, programs, and profiles . In S. L. Hamid (Ed.), Peer leadership: A primer on program essentials (Monograph No. 32, pp. 37–52). University of South Carolina, The National Resource Center for The First-Year Experience and Students in Transition. - Harper, S. R. (2010). An anti-deficit achievement framework for research on students of color in STEM. New Directions for Institutional Research, 2010(148), 63–74. 10.1002/ir.362
- Harper, S. R. (2012). Black male student success in higher education: A report from the National Black Male College Achievement Study. Center for the Study of Race and Equity in Education, University of Pennsylvania.
https://rossierapps.usc.edu/facultydirectory/publications/231/Harper%20(2012)%20Black%20Male%20Success.pdf - Hicks, D. (2021). Dignity: Its essential role in resolving conflict. Yale University Press. 10.12987/9780300263503
- Hunter, M. S., & Heath, M. M. (2001).
The building blocks of the peer leader programs: Recruitment, selection, and training . In S. L. Hamid (Ed.), Peer leadership: A primer on program essentials (Monograph No. 32, pp. 37–52). The National Resource Center for The First-Year Experience and Students in Transition. - Irwin, L. N., Miller, J. S., Morgan, K., & Linley, J. (2023). A critical ecological exploration of peer socialization agents’ sense of belonging. Journal of College Student Development, 64(5), 509–523. 10.1353/csd.2023.a911787
- Janson, H., & Olsson, U. (2001). A measure of agreement for interval or nominal multivariate observations. Educational and Psychological Measurement, 61(2), 277–289. 10.1177/00131640121971239
- Janson, H., & Olsson, U. (2004). A measure of agreement for interval or nominal multivariate observations by different sets of judges. Educational and Psychological Measurement, 64(1), 62–70. 10.1177/0013164403260195
- Jarratt, L., Martin-Stanley, C. R. II, Linley, J. L., Mollet, A. L., Anderson, D. L., & Mohebali, M. (2024). Underlife: Peer socialization agents’ resistance to higher education diversity narratives. Journal of Diversity in Higher Education, 17(3), 430–442. 10.1037/dhe0000431
- Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144. 10.1016/j.edurev.2007.05.002
- Keller, P., & Porter, H. D. (2020). A terminological study of peer education in higher education [White paper]. College Reading and Learning Association.
https://cdn.ymaws.com/crla.net/resource/resmgr/white_papers/crla_2020_whitepaper_peer_ed.pdf - Keup, J., & Skipper, T. (2010, March 21–24). Findings from the 2009 National Survey of Peer Leaders. [Conference session]. American College Personnel Association Annual Convention, Boston, MA, United States.
- Klein, D. (2018). Implementing a general framework for assessing interrater agreement in Stata. The Stata Journal: Promoting Communications on Statistics and Stata, 18(4), 871–901. 10.1177/1536867X1801800408
- Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. 10.2307/2529310
- Leyzberg, D., Lumbroso, J., & Moretti, C. (2017). Nailing the TA interview: Using a rubric to hire teaching assistants. Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education, 128–133. 10.1145/3059009.3059057
- Linley, J. L. (2015). We are (not) all bulldogs: Minoritized peer socialization agents’ critical sensemaking about collegiate contexts [Doctoral dissertation, Michigan State University]. MSU Libraries Digital Repository.
https://d.lib.msu.edu/etd/3464 - Linley, J. L. (2017). We are (not) all bulldogs: Minoritized peer socialization agents’ meaning-making about collegiate contexts. Journal of College Student Development, 58(5), 643–656. 10.1353/csd.2017.0051
- Linley, J. L. (2018). Racism here, racism there, racism everywhere: The racial realities of minoritized peer socialization agents at a historically white institution. Journal of College Student Development, 59(1), 21–36. 10.1353/csd.2018.0002
- McDaniel, A. (2008).
Recruiting and training Supplemental Instruction Leaders . In M.E. Stone & G. Jacobs (Eds.), Supplemental Instruction: Improving First-Year Student Success in High-Risk Courses (pp. 39–56). University of South Carolina. - Moore, C., & Teather, S. (2013). Engaging students in peer review: Feedback as learning. Issues in Educational Research, 23(2), 196–211.
- Panadero, E., & Romero, M. (2014). To Rubric or Not to Rubric? The Effects of Self-Assessment on Self-Regulation, Performance and Self-Efficacy. Assessment in Education: Principles, Policy & Practice, 21(2), 133–148. 10.1080/0969594X.2013.877872
- Perez, D., Ashlee, K., Do, V., Karikari, S., & Sim, C. (2017). Re-conceptualizing student success in higher education: Reflections from graduate student affairs educators using anti-deficit achievement framework. Journal on Excellence in College Teaching, 28(3), 5–28.
- Popham, W. J. (1997). What’s wrong – and what’s right – with rubrics. Educational Leadership, 55(2), 72–75.
https://skidmore.edu/assessment/handbook/Popham_1997_Whats-Wrong_and-Whats-Right_With-Rubrics.pdf - Rager, J. P., & Leslie, D. (2023). Adapted Supplemental Instruction Interview Rubric (SIIR). [Unpublished work].
- Reddy, Y. M., & Andrade, H. (2010). A review of rubric use in higher education. Assessment & Evaluation in Higher Education, 35(4), 435–448. 10.1080/02602930902862859
- Reyes, H. L., & Duran, A. (2021). Higher education scholars challenging deficit thinking: Ananalysis of research informed by community cultural wealth. Journal of Critical Scholarship on Higher Education and Student Affairs, 6(1), 7–20.
- Rogers, C. R. (1961). On becoming a person: A therapist’s view of psychotherapy. Houghton Mifflin.
- Sambamurthy, N., & Cox, M. F. (2016, June), Developing a Behavior-Based Rubric for Assessing Professional Skills During Hiring. [Paper presentation]. 2016 ASEE Annual Conference & Exposition, New Orleans, LA, United States. 10.18260/p.27288
- Shook, J., & Keup, J. (2012). The benefits of peer leader programs: An overview from the literature. New Directions for Higher Education, 2012(157), 5–16. 10.1002/he.20002
- Skalicky, J., Speed, H., van der Meer, J., & Young, D. G. (2024). An international snapshot of peer leadership in higher education. Journal of Leadership Education, 24(1), 84–101. 10.1108/JOLE-02-2024-0031
- Taylor, B., Kisby, F., & Reedy, A. (2024). Rubrics in higher education: An exploration of undergraduate students’ understanding and perspectives. Assessment & Evaluation in Higher Education, 49(6), 799–809. 10.1080/02602938.2023.2299330
- The International Center for Supplemental Instruction. (2019). SI Supervisor manual.
- The International Center for Supplemental Instruction. (2022). What is Supplemental Instruction?
https://info.umkc.edu/si/ - van der Meer, J., Skalicky, J., Speed, H., & Young, D. G. (2022). Focusing on the development of the whole student: An international comparative study of the perceived benefits of peer leadership in higher education. Open Journal of Social Sciences, 10, 14–35. 10.4236/jss.2022.103002
- Witty, M. C., & Adomaitis, R. (2014).
Carl Rogers and client-centered counseling . In R. D. Parsons & N. Zhang (Eds.), Counseling theory: Guiding reflective practice (pp. 171–199). Sage. 10.4135/9781483399621.n7 - Zhang, B., Johnston, L., & Kilic, G. B. (2008). Assessing the reliability of self- and peer rating in student group work. Assessment & Evaluation in Higher Education, 33(3), 329–340. 10.1080/02602930701293181
DOI: https://doi.org/10.21061/jopl.151 | Journal eISSN: 2200-2359
Language: English
Page range: 4 - 4
Submitted on: Jul 31, 2025
Accepted on: Jan 5, 2026
Published on: Aug 11, 2026
Published by: Virginia Tech Publishing
In partnership with: Paradigm Publishing Services
© 2026 Jillian P. Rager, Jaime S. Miller, Joseph M. Kush, Deidre Leslie, Ashley C. Carr, published by Virginia Tech Publishing
This work is licensed under the Creative Commons Attribution 4.0 License.