An Open-Source Adaptive Comparative Judgement App for Technology Education Research and Practice: Alpha Version
By: Jeffery Buckley
References
- Bartholomew, S., & Jones, M. (2021). A systematized review of research with adaptive comparative judgment (ACJ) in higher education. International Journal of Technology and Design Education. 10.1007/s10798-020-09642-6
- Bartholomew, S. R., Mentzer, N., Jones, M., Sherman, D., & Baniya, S. (2022). Learning by evaluating (LbE) through adaptive comparative judgment. International Journal of Technology and Design Education, 32(2), 1191–1205. 10.1007/s10798-020-09639-1
- Bartholomew, S., Strimel, G., & Jackson, A. (2018). A comparison of traditional and adaptive comparative judgment assessment techniques for freshmen engineering design projects. International Journal of Engineering Education, 34(1), 20–33.
- Bartholomew, S., Strimel, G., & Zhang, L. (2018). Examining the potential of adaptive comparative judgment for elementary STEM design assessment. The Journal of Technology Studies, 44(2), 58–75. 10.2307/26730731
- Bartholomew, S., Yoshikawa, E., Hartell, E., & Strimel, G. (2020). Identifying design values across countries through adaptive comparative judgment. International Journal of Technology and Design Education, 30(2), 321–347. 10.1007/s10798-019-09506-8
- Bartholomew, S., & Yoshikawa-Ruesch, E. (2018).
A systematic review of research around adaptive comparative judgement (ACJ) in K-16 education . In J. Wells (Ed.), CTETE - Research Monograph Series (Vol. 1, pp. 6–28). Council on Technology and Engineering Teacher Education. - Bramley, T. (2015). Investigating the reliability of adaptive comparative judgment [Cambridge Assessment Research Report]. Cambridge Assessment.
- Bramley, T., & Vitello, S. (2019). The effect of adaptivity on the reliability coefficient in adaptive comparative judgement. Assessment in Education: Principles, Policy & Practice, 26(1), 43–58. 10.1080/0969594X.2017.1418734
- Bramley, T., & Wheadon, C. (2015). The reliability of Adaptive Comparative Judgment. AEA-Europe Annual Conference,
March, 7–9 . - Buckley, J. (2024). Adaptive comparative judgement shiny app: Supplementary material.
https://osf.io/y4aht/ - Buckley, J., & Canty, D. (2022). Assessing performance: Addressing the technical challenge of comparing novel portfolios to the ‘ACJ-Steady State’. PATT39: PATT on the Edge - Technology, Innovation and Education, 523–537.
- Buckley, J., Canty, D., & Seery, N. (2022). An exploration into the criteria used in assessing design activities with adaptive comparative judgment in technology education. Irish Educational Studies, 41(2), 313–331. 10.1080/03323315.2020.1814838
- Buckley, J., Seery, N., Gumaelius, L., Canty, D., Doyle, A., & Pears, A. (2021). Framing the constructive alignment of design within technology subjects in general education. International Journal of Technology and Design Education, 31(5), 867–883. 10.1007/s10798-020-09585-y
- Buckley, J., Seery, N., & Kimbell, R. (2022). A review of the valid methodological use of adaptive comparative judgment in technology education research. Frontiers in Education, 7(787926), 1–6. 10.3389/feduc.2022.787926
- Buckley, J., Seery, N., & Kimbell, R. (2023). Modelling approaches to combining and comparing independent adaptive comparative judgement ranks. The 40th International Pupils’ Attitudes Towards Technology Conference Proceedings 2023, 1(October), Article October.
https://openjournals.ljmu.ac.uk/PATT40/article/view/1570 - Hartell, E., & Buckley, J. (2021).
Comparative judgement: An overview . In A. Marcus Quinn & T. Hourigan (Eds.), Handbook for Online Learning Contexts: Digital, Mobile and Open (pp. 289–307). Springer International Publishing. 10.1007/978-3-030-67349-9_20 - Hunter, D. R. (2004). MM algorithms for generalized Bradley-Terry models. The Annals of Statistics, 32(1), 384–406. 10.1214/aos/1079120141
- Kimbell, R. (2022). Examining the reliability of Adaptive Comparative Judgement (ACJ) as an assessment tool in educational settings. International Journal of Technology and Design Education, 32(3), 1515–1529. 10.1007/s10798-021-09654-w
- Kimbell, R., Wheeler, T., Stables, K., Shepard, T., Martin, F., Davies, D., Pollitt, A., & Whitehouse, G. (2009). E-scape portfolio assessment: Phase 3 report. Goldsmiths, University of London.
- Newhouse, C. P. (2014). Using digital representations of practical production work for summative assessment. Assessment in Education: Principles, Policy and Practice, 21(2), 205–220. 10.1080/0969594X.2013.868341
- Pollitt, A. (2012). Comparative judgement for assessment. International Journal of Technology and Design Education, 22(2), 157–170. 10.1007/s10798-011-9189-x
- Pollitt, A. (2015). On ‘reliability’ bias in ACJ: Valid simulation of adaptive comparative judgement [Occasional Research Paper]. Cambridge Exam Research.
- Robitzsch, A. (2021). sirt: Supplementary Item Response Theory Models (Version R package version 3.10-118) [R].
https://CRAN.R-project.org/package=sirt - Sadler, D. R. (2009).
Transforming holistic assessment and grading into a vehicle for complex learning . In G. Joughin (Ed.), Assessment, Learning and Judgement in Higher Education (pp. 45–63). Springer. - Seery, N., Kimbell, R., & Buckley, J. (2022). Using Teachers’ Judgments of Quality to Establish Performance Standards in Technology Education Across Schools, Communities, and Nations. Frontiers in Education, 7.
https://www.frontiersin.org/article/10.3389/feduc.2022.806894 - Stables, K. (2020).
Signature pedagogies for designing: A speculative framework for supporting learning and teaching in design and technology education . In P. J. Williams & D. Barlex (Eds.), Pedagogy for Technology Education in Secondary Schools (pp. 99–120). Springer International Publishing. 10.1007/978-3-030-41548-8_6 - The R Foundation for Statistical Computing. (2022). R: A language and environment for statistical computing (Version Version 4.2.2 ‘Innocent and Trusting’) [Computer software]. R Foundation for Statistical Computing.
https://www.R-project.org/ - Thurstone, L. L. (1927). A law of comparative judgement. Psychological Review, 34(4), 273–286. 10.1037/h0070288
- Verhavert, S. (2018). Beyond a mere rank order: The method, the reliability and the efficiency of comparative judgment [Doctoral thesis, Universiteit Antwerpen].
https://repository.uantwerpen.be/desktop/irua - Verhavert, S., Bouwer, R., Donche, V., & De Maeyer, S. (2019). A meta-analysis on the reliability of comparative judgement. Assessment in Education: Principles, Policy & Practice, 26(5), 541–562. 10.1080/0969594X.2019.1602027
- Verhavert, S., De Maeyer, S., Donche, V., & Coertjens, L. (2018). Scale Separation Reliability: What Does It Mean in the Context of Comparative Judgment? Applied Psychological Measurement, 42(6), 428–445. 10.1177/0146621617748321
- Verhavert, S., Furlong, A., & Bouwer, R. (2022). The accuracy and efficiency of a reference-based adaptive selection algorithm for comparative judgment. Frontiers in Education, 6.
https://www.frontiersin.org/article/10.3389/feduc.2021.785919 - Whitehouse, C., & Pollitt, A. (2012). Using adaptive comparative judgement to obtain a highly reliable rank order in summative assessment. Centre for Education Research and Policy.
- Williams, P. J., & Kimbell, R. (Eds.). (2012). Special issue on e-scape [Special issue]. International Journal of Technology and Design Education, 22(2).
DOI: https://doi.org/10.21061/jte.v36i1.a.4 | Journal eISSN: 1045-1064
Language: English
Page range: 58 - 82
Submitted on: Mar 28, 2024
Accepted on: Oct 22, 2024
Published on: Nov 6, 2024
Published by: Virginia Tech
In partnership with: Paradigm Publishing Services
© 2024 Jeffery Buckley, published by Virginia Tech
This work is licensed under the Creative Commons Attribution 4.0 License.