Computer Testing, Formative or Summative, and Proctoring: Does it Matter? Lessons Learned From the Dutch Interuniversity Progress Test of Medicine During the Corona Pandemic

Jan Hindrik Ravesloot; Jeroen Donkers; Ariadne A. Meiboom; Alexandra M. J. Langers; Bram Jacobs; Rene Tio; Cees van der Vleuten; Andreas J. A. Bremers

doi:10.5334/pme.1771

Abstract

Introduction: The COVID-19 pandemic prompted significant changes in the administration of the Dutch Interuniversity Progress Test of Medicine, offering a unique opportunity to investigate the effects of various test formats. This study explores the impact of transitioning from paper-based to computer-based testing, the shift from summative to formative testing, and the effectiveness of remote proctoring compared to live supervision.

Methods: Data from over 10,000 participants across five medical schools were analyzed.

Results: Results showed no significant difference in student performance between paper-based and computer-based tests. Additionally, remote proctoring proved to be as effective as live supervision in preventing dishonest behavior. Formative testing yielded slightly better results than summative testing in most schools, although the effect varied between institutions.

Discussion: Overall, our study concludes that computer-based testing is a viable alternative to paper-based formats, and remote proctoring can effectively replace live invigilation, at least in our setting and under COVID-19–related circumstances.

References

Tio RA, Schutte B, Meiboom AA, Greidanus J, Dubois EA, Bremers AJA. The progress test of medicine: the Dutch experience. Perspect Med Educ. 2016;5(1):51–5. DOI: 10.1007/S40037-015-0237-1
Open DOI Search in Google Scholar Back to article
Aristovnik A, Keržič D, Ravšelj D, Tomaževič N, Umek L. Impacts of the COVID-19 pandemic on life of higher education students: A global perspective. Sustainability. 2020;12(20):8438. DOI: 10.3390/su12208438
Open DOI Search in Google Scholar Back to article
Nurunnabi M, Almusharraf N. Social distancing and reopening universities after the COVID-19 pandemic: policy complexity in G20 countries. J Public Health Res. 2020;9(4):1957. DOI: 10.4081/jphr.2020.1957
Open DOI Search in Google Scholar Back to article
R Core Team. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2022.
Search in Google Scholar Back to article
Bates D, Mächler M, Bolker B, Walker S. Fitting linear mixed-effects models using lme4. J Stat Softw. 2015;67(1):1–48. DOI: 10.18637/jss.v067.i01
Open DOI Search in Google Scholar Back to article
Fox J, Weisberg S. An R companion to applied regression. 3rd ed. Thousand Oaks (CA): Sage; 2019.
Search in Google Scholar Back to article
Cecilio-Fernandes D, Kerdijk W, Jaarsma D, Tio RA. Development of cognitive processing and judgments of knowledge in medical students: analysis of progress test results. Med Teach. 2016;38(11):1125–9. DOI: 10.3109/0142159X.2016.1170781
Open DOI Search in Google Scholar Back to article
Karay Y, Schauber SK, Stosch C, Schüttpelz-Brauns K. Computer versus paper—does it make any difference in test performance? Teach Learn Med. 2015;27(1):57–62. DOI: 10.1080/10401334.2014.979175
Open DOI Search in Google Scholar Back to article
Backes B, Cowan J. Is the pen mightier than the keyboard? The effect of online testing on measured student achievement. Econ Educ Rev. 2019;68:89–103. DOI: 10.1016/j.econedurev.2018.12.007
Open DOI Search in Google Scholar Back to article
Gu L, Ling G, Liu OL, Yang Z, Li G, Kardanova E, et al. Examining mode effects for an adapted Chinese critical thinking assessment. Assess Eval High Educ. 2021;46(6):879–93. DOI: 10.1080/02602938.2020.1836121
Open DOI Search in Google Scholar Back to article
Yu W, Iwashita N. Comparison of test performance on paper-based testing and computer-based testing by English-majored undergraduate students in China. Lang Test Asia. 2021;11(1):32. DOI: 10.1186/s40468-021-00147-0
Open DOI Search in Google Scholar Back to article
Toroujeni SMH. Computerized testing in reading comprehension skill. Educ Inf Technol. 2022;27(2):1771–810. DOI: 10.1007/s10639-021-10584-2
Open DOI Search in Google Scholar Back to article
Anakwe B. Comparison of student performance in paper-based versus computer-based testing. J Educ Bus. 2008;84(1):13–7. DOI: 10.3200/JOEB.84.1.13-17
Open DOI Search in Google Scholar Back to article
Gordanier J, Ozturk O, Zhan C. Pencils down? Computerized testing and student achievement. Educ Finance Policy. 2023;18(2):232–52. DOI: 10.1162/edfp_a_00373
Open DOI Search in Google Scholar Back to article
Dillon A. Reading from paper versus screens: a critical review. Ergonomics. 1992;35(10):1297–326. DOI: 10.1080/00140139208967394
Open DOI Search in Google Scholar Back to article
Couto LB, Durand MT, Wolff ACD. Formative assessment scores correlate with OSCE and progress testing. Med Teach. 2019;41(2):190–5.
Search in Google Scholar Back to article
Devine OP, Harborne AC, McManus IC. Assessment at UK medical schools correlates with postgraduate attainment. BMC Med Educ. 2015;15:146. DOI: 10.1186/s12909-015-0428-9
Open DOI Search in Google Scholar Back to article
McManus IC, Elder AT, de Champlain A, et al. Performance differences on MRCP(UK). BMC Med. 2008;6:5. DOI: 10.1186/1741-7015-6-5
Open DOI Search in Google Scholar Back to article
Seagrave MP, Foster-Johnson L, Waits JB. Enhancing examination success. Med Sci Educ. 2022;32(3):451–9. DOI: 10.1007/s40670-022-01568-z
Open DOI Search in Google Scholar Back to article
Alessio HM, Malay N, Maurer K, et al. Effect of proctoring on online test scores. Online Learn. 2017;21(1). DOI: 10.24059/olj.v21i1.885
Open DOI Search in Google Scholar Back to article
Bergmans L, Bouali N, Luttikhuis M, Rensink A. Efficacy of online proctoring using Proctorio. CSEDU. 2021:279–90. DOI: 10.5220/0010399602790290
Open DOI Search in Google Scholar Back to article
Coghlan S, Miller T, Paterson J. Ethics of online exam supervision technologies. Philos Technol. 2021;34:1581–606. DOI: 10.1007/s13347-021-00476-1
Open DOI Search in Google Scholar Back to article

Computer Testing, Formative or Summative, and Proctoring: Does it Matter? Lessons Learned From the Dutch Interuniversity Progress Test of Medicine During the Corona Pandemic

Abstract

Paradigm

My account