Learning by Evaluating: An Exploration of Optimizing Design-Based Instruction
References
- Aitken, A., & Thompson, D. G. (2018). Using software to engage design students in academic writing. International Journal of Technology and Design Education, 28(3), 885–898. 10.1007/s10798-017-9413-4
- Baniya, S., Mentzer, N., Bartholomew, S., Chesley, A., Moon, C., & Sherman, D. (2019). Using Adaptive Comparative Judgment in writing assessment: An investigation of reliability among interdisciplinary evaluators. The Journal of Technology Studies, 45(1), 24–35.
- Barab, S., & Squire, K. (2004). Design-Based Research: Putting a Stake in the Ground. Journal of the Learning Sciences, 13(1), 1–14. 10.1207/s15327809jls1301_1
- Bartholomew, S. (2017). Assessing open-ended design problems. Technology and Engineering Teacher, 76(6), 13–17.
- Bartholomew, S., & Connolly, P. E. (2017, June 24). Adaptive Comparative Judgment in Graphics Applications and Education. 2017 ASEE Annual Conference & Exposition.
https://strategy.asee.org/adaptive-comparative-judgment-in-graphics-applications-and-education - Bartholomew, S., & Jones, M. D. (2022). A systematized review of research with adaptive comparative judgment (ACJ) in higher education. International Journal of Technology and Design Education, 32(2), 1159–1190. 10.1007/s10798-020-09642-6
- Bartholomew, S., Mentzer, N., & Jackson, A. (2022). Lessons from Dilbert: Clarifying design expectations. Technology & Engineering Teacher.
- Bartholomew, S., Mentzer, N., Jones, M., Sherman, D., & Baniya, S. (2022). Learning by evaluating (LbE) through adaptive comparative judgment. International Journal of Technology and Design Education, 32(2), 1191–1205. 10.1007/s10798-020-09639-1
- Bartholomew, S., Yauney, J., Mentzer, N., & Thorne, S. (2021). Investigating the impacts of differentiated stimulus materials in a Learning by Evaluating activity. Mississippi Valley Technology Teacher Education Conference (MVTTEC), Nashville, TN, USA.
- Bell, P. (2004). On the theoretical breadth of design-based research in education. Educational Psychologist, 39(4), 243–253. 10.1207/s15326985ep3904_6
- Bouwer, R., Lesterhuis, M., Bonne, P., & De Maeyer, S. (2018). Applying criteria to examples or learning by comparison: Effects on students’ evaluative judgment and performance in writing. Frontiers in Education, 3.
https://www.frontiersin.org/articles/10.3389/feduc.2018.00086 - Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32–42. 10.3102/0013189X018001032
- Canty, D., Seery, N., Hartell, E., & Doyle, A. (2017). Integrating peer assessment in technology education through adaptive comparative judgment.
- Carless, D., & Boud, D. (2018). The development of student feedback literacy: Enabling uptake of feedback. Assessment & Evaluation in Higher Education, 43(8), 1315–1325. 10.1080/02602938.2018.1463354
- Carless, D., & Chan, K. K. H. (2017). Managing dialogic use of exemplars. Assessment & Evaluation in Higher Education, 42(6), 930–941. 10.1080/02602938.2016.1211246
- Chong, S. W. (2019). The use of exemplars in English writing classrooms: From theory to practice. Assessment & Evaluation in Higher Education, 44(5), 748–763. 10.1080/02602938.2018.1535051
- Collins, A., Collins, A., Brown, J. S., & Holum, A. (1991). Cognitive apprenticeship: Making thinking visible. AMERICAN EDUCATOR, 6, 38–46.
- Creswell, J., & Poth, C. (2017). Qualitative inquiry and research design (4th ed.). SAGE Publications Inc.
https://us.sagepub.com/en-us/nam/qualitative-inquiry-and-research-design/book246896 - Crismond, D. P., & Adams, R. S. (2012). The Informed Design Teaching and Learning Matrix. Journal of Engineering Education, 101(4), 738–797. 10.1002/j.2168-9830.2012.tb01127.x
- Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5–8. 10.3102/0013189X032001005
- Dixon, H., Hawe, E., & Hamilton, R. (2020). The case for using exemplars to develop academic self-efficacy. Assessment & Evaluation in Higher Education, 45(3), 460–471. 10.1080/02602938.2019.1666084
- Douglas, E. P., Koro-Ljungberg, M., McNeill, N. J., Malcolm, Z. T., & Therriault, D. J. (2012). Moving beyond formulas and fixations: Solving open-ended engineering problems. European Journal of Engineering Education, 37(6), 627–651. 10.1080/03043797.2012.738358
- Ge, X., Chen, C.-H., & Davis, K. A. (2005). Scaffolding novice instructional designers’ problem-solving processes using question prompts in a web-based learning environment. Journal of Educational Computing Research, 33, 219–248. 10.2190/5F6J-HHVF-2U2B-8T3G
- Ghefaili, A. (2003). Cognitive apprenticeship, technology, and the contextualization of learning environments. Journal of Educational Computing, Design & Online Learning,
4 . - Grainger, P. R., Heck, D., & Carey, M. D. (2018). Are Assessment Exemplars Perceived to Support Self-Regulated Learning in Teacher Education? Frontiers in Education,
3 .https://www.frontiersin.org/articles/10.3389/feduc.2018.00060 - Hawe, E., & Dixon, H. (2017). Assessment for learning: A catalyst for student self-regulation. Assessment & Evaluation in Higher Education, 42(8), 1181–1192. 10.1080/02602938.2016.1236360
- Headley, J., & Pittson, H. (2020). Bottom Up! Establishing a University Wide Community of Practice for Exemplars and Exploration of Its Affect in Enhancing the Quality of Student Assessment. Practitioner Research in Higher Education, 13(1), 18–26. eric.
- Hendry, G. D., & Tomitsch, M. (2014). Implementing an exemplar-based approach in an interaction design subject: Enhancing students’ awareness of the need to be creative. International Journal of Technology and Design Education, 24(3), 337–348. 10.1007/s10798-013-9256-6
- Hsieh, H.-F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288. 10.1177/1049732305276687
- Huber, S., Bosman, L., & Bartholomew, S. (2021). Library instruction and adaptive comparative judgment to foster visual literacy skills. Portal: Libraries and the Academy, 21(1), 149–169. 10.1353/pla.2021.0001
- ITEEA. (2011). Engineering Design Process (EDP) grades 6-12. Retrieved August 10, 2023, from
https://www.iteea.org/engineering-bydesign - Jansson, D. G., & Smith, S. M. (1991). Design fixation. Design Studies, 12(1), 3–11. 10.1016/0142-694X(91)90003-F
- Jonassen, D. H. (1997). Instructional design models for well-structured and III-structured problem-solving learning outcomes. Educational Technology Research and Development, 45(1), 65–94. 10.1007/BF02299613
- Katehi, L., Pearson, G., & Feder, M. (2009). The status and nature of K-12 engineering education in the United States. The Bridge on K-12 Engineering Education, 3, 5–10.
- Kimbell, R. (2007). E-assessment in project e-scape. Design & Technology Education: An International Journal, 12(2).
- Kimbell, R. (2012). The origins and underpinning principles of e-scape. International Journal of Technology and Design Education, 22(2), 123–134. 10.1007/s10798-011-9197-x
- Kimbell, R. (2018). Constructs of quality and the power of holism. Perspectives on Human Capacity and Development, 181–186.
- Knight, S., Leigh, A., Davila, Y. C., Martin, L. J., & Krix, D. W. (2019). Calibrating assessment literacy through benchmarking tasks. Assessment & Evaluation in Higher Education, 44(8), 1121–1132. 10.1080/02602938.2019.1570483
- Mentzer, N., Lee, W., & Bartholomew, S. (2021). Examining the validity of adaptive comparative judgment for peer evaluation in a design thinking course. Frontiers in Education,
6 .https://www.frontiersin.org/articles/10.3389/feduc.2021.772832 - Newlyn, D. (2013). Providing Exemplars in the Learning Environment: The Case For and Against. Universal Journal of Educational Research, 1(1), 26–32.
- Poitras, G., & Poitras, E. (2011). A cognitive apprenticeship approach to engineering education: The role of learning styles. Engineering Education, 6(1), 62–72. 10.11120/ened.2011.06010062
- Pollitt, A. (2012). The method of Adaptive Comparative Judgement. Assessment in Education: Principles, Policy & Practice, 19(3), 281–300. 10.1080/0969594X.2012.665354
- Potter, T., Englund, L., Charbonneau, J., MacLean, M. T., Newell, J., & Roll, I. (2017). ComPAIR: A new online tool using adaptive comparative judgement to support learning with peer feedback. Teaching and Learning Inquiry, 5(2),
Article 2 . 10.20343/teachlearninqu.5.2.8 - Rust, C., Price, M., & O’Donovan, B. (2003). Improving Students’ Learning by Developing their Understanding of Assessment Criteria and Processes. Assessment & Evaluation in Higher Education, 28(2), 147–164. 10.1080/02602930301671
- Sadler, D. R. (1987). Specifying and Promulgating Achievement Standards. Oxford Review of Education, 13(2), 191–209. 10.1080/0305498870130207
- Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18(2), 119–144. 10.1007/BF00117714
- Sadler, D. R. (2002).
Ah!… so that’s ‘quality’ . In Assessment. Routledge. - Saldaña, J. (2013). The Coding Manual for Qualitative Researchers (Second). SAGE Publications Inc.
https://us.sagepub.com/en-us/nam/the-coding-manual-for-qualitative-researchers/book243616 - Seery, N., Buckley, J., Delahunty, T., & Canty, D. (2019). Integrating learners into the assessment process using adaptive comparative judgement with an ipsative approach to identifying competence based gains relative to student ability levels. International Journal of Technology and Design Education, 29(4), 701–715. 10.1007/s10798-018-9468-x
- Strimel, G. J., Bartholomew, S., Purzer, S., Zhang, L., & Ruesch, E. Y. (2021). Informing engineering design through adaptive comparative judgment. European Journal of Engineering Education, 46(2), 227–246. 10.1080/03043797.2020.1718614
- Tam, A. C. F. (2021). Undergraduate students’ perceptions of and responses to exemplar-based dialogic feedback. Assessment & Evaluation in Higher Education, 46(2), 269–285. 10.1080/02602938.2020.1772957
- Vygotsky, L. S. (1987). The Collected Works of L.S. Vygotsky: The Fundamentals of Defectology. Springer Science & Business Media.
- Yucel, R., Bird, F. L., Young, J., & Blanksby, T. (2014). The road to self-assessment: Exemplar marking before peer review develops first-year students’ capacity to judge the quality of a scientific report. Assessment & Evaluation in Higher Education, 39(8), 971–986. 10.1080/02602938.2014.880400
DOI: https://doi.org/10.21061/jte.v35i2.a.4 | Journal eISSN: 1045-1064
Language: English
Page range: 53 - 80
Submitted on: Sep 9, 2023
Accepted on: Mar 29, 2024
Published on: Jun 25, 2024
Published by: Virginia Tech
In partnership with: Paradigm Publishing Services
Keywords:
© 2024 Scott Thorne, Nathan Mentzer, Greg J. Strimel, Scott Bartholomew, Jason Ware, published by Virginia Tech
This work is licensed under the Creative Commons Attribution 4.0 License.