Have a personal or library account? Click to login
The Effect of the Question Mark Option in Progress Testing: A Large-Scale Longitudinal Study Cover

The Effect of the Question Mark Option in Progress Testing: A Large-Scale Longitudinal Study

Perspectives on Medical Education
Volume 14 (2025): Issue 1
By: Elise V. van Wijk,  Jeroen Donkers,  Peter C. J. de Laat,  Ariadne A. Meiboom,  Bram Jacobs,  Jan Hindrik Ravesloot,  René A. Tio,  Frederike M. M. Oud,  Jeroen P. Kooman,  André J. A. Bremers and  Alexandra M. J. Langers  
Open Access
|Dec 2025

References

  1. 1Cecilio-Fernandes D, Medema H, Collares CF, Schuwirth L, Cohen-Schotanus J, Tio RA. Comparison of formula and number-right scoring in undergraduate medical training: a Rasch model analysis. BMC Medical Education. 2017;17:192. DOI: 10.1186/s12909-017-1051-8
    Back to article
  2. 2Lord FM. Formula scoring and number-right scoring. Journal of Educational Measurement. 1975;12(1):7–11. DOI: 10.1111/j.1745-3984.1975.tb01003.x
    Back to article
  3. 3Wrigley W, Van Der Vleuten CP, Freeman A, Muijtjens A. A systemic framework for the progress test: Strengths, constraints and issues: AMEE Guide No. 71. Medical Teacher. 2012;34(9):683–97. DOI: 10.3109/0142159X.2012.704437
    Back to article
  4. 4Ravesloot CJ, Van der Schaaf MF, Muijtjens AMM, Haaring C, Kruitwagen CLJJ, Beek FJA, et al. The don’t know option in progress testing. Advances in Health Sciences Education. 2015;20(5):1325–38. DOI: 10.1007/s10459-015-9604-2
    Back to article
  5. 5Rowley G, Traub R. Formula scoring, number-right scoring, and test-taking strategy. Journal of Educational Measurement. 1977;14(1). DOI: 10.1111/j.1745-3984.1977.tb00024.x
    Back to article
  6. 6Edwards W. The theory of decision making. Psychological Bulletin. 1954;51(4):380–417. DOI: 10.1037/h0053870
    Back to article
  7. 7Kahneman D, Tversky A. Prospect theory: An analysis of decision under risk. Econometrica. 1979;47(2):263–91. DOI: 10.2307/1914185
    Back to article
  8. 8Fraser C, Beattie M. The impact of risk aversion on formula scoring in multiple-choice tests. Applied Psychological Measurement. 2002;26(3):235–44. DOI: 10.1177/014662102760913645
    Back to article
  9. 9Banerjee M, Wiegand SA. The impact of formula scoring on ability estimates and validity in computer adaptive testing. Educational Measurement: Issues and Practice. 2010;29(4):17–29. DOI: 10.1111/j.1745-3992.2010.00174
    Back to article
  10. 10Lord FM, Lord FM. Formula Scoring and Validity. Educational and Psychological Measurement. 1963-12-01;23(4). DOI: 10.1177/001316446302300403
    Back to article
  11. 11Messick S. Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist. 1995;50(9):741–9. DOI: 10.1037/0003-066X.50.9.741
    Back to article
  12. 12Kampmeyer D, Matthes J, Herzig S. Lucky guess or knowledge: a cross-sectional study using the Bland and Altman analysis to compare confidence-based testing of pharmacological knowledge in 3rd and 5th year medical students. Advances in Health Sciences Education. 2015;20(2):431–40. DOI: 10.1007/s10459-014-9537-1
    Back to article
  13. 13Thompson J, Lewis C. Examining guessing behavior and test-wiseness in multiple-choice tests. Applied Measurement in Education. 2007;20(2):135–53. DOI: 10.1207/s15324818ame2002_3
    Back to article
  14. 14Koriat A. The self-consistency model of subjective confidence. Psychological Review. 2012;119(1):80–113. DOI: 10.1037/a0025648
    Back to article
  15. 15Muijtjens AM, Mameren HV, Hoogenboom RJ, Evers JL, van der Vleuten CP. The effect of a ‘don’t know’ option on test scores: number-right and formula scoring compared. Medical Education. 1999;33(4):267–75. DOI: 10.1046/j.1365-2923.1999.00292.x
    Back to article
  16. 16Rowley GL, Traub RE. Formula Scoring, Number-Right Scoring, and Test-Taking Strategy. Journal of Educational Measurement. 1977;14(1):15–22. DOI: 10.1111/j.1745-3984.1977.tb00024.x
    Back to article
  17. 17Kubinger K, Wolfsbauer C. On the risk of certain psychotechnological response options in multiple-choice tests: Does a particular personality handicap examinees? European Journal of Psychological Assessment. 2010;26(4). DOI: 10.1027/1015-5759/a000040
    Back to article
  18. 18Budescu D, Bar-Hillel M. To Guess or Not to Guess: A Decision-Theoretic View of Formula Scoring. Journal of Educational Measurement. 1993/12/01;30(4). DOI: 10.1111/j.1745-3984.1993.tb00427.x
    Back to article
  19. 19van Wijk EV, Donkers J, de Laat PCJ, Meiboom AA, Jacobs B, Ravesloot JH, et al. Computer Adaptive vs. Non-adaptive Medical Progress Testing: Feasibility, Test Performance, and Student Experiences. Perspectives on Medical Education. 2024;13(1). DOI: 10.5334/pme.1345
    Back to article
  20. 20Chang H-H. Psychometrics behind Computerized Adaptive Testing. Psychometrika. 2015;80(1):1–20. DOI: 10.1007/s11336-014-9401-5
    Back to article
  21. 21Downing SM. Item response theory: applications of modern test theory in medical education. Medical Education. 2003;37(8):739–45. DOI: 10.1046/j.1365-2923.2003.01587.x
    Back to article
  22. 22Framework for Undergraduate Medical Education 2021 [updated 2021-08-20T11:17:25 + 02:00; cited July 2023]. Available from: https://www.nfu.nl/en/themes/professional-future/medicine-programmes/framework-undergraduate-medical-education.
    Back to article
  23. 23Tio RA, Schutte B, Meiboom AA, Greidanus J, Dubois EA, Bremers AJA, et al. The progress test of medicine: the Dutch experience. Perspectives on Medical Education. 2016;5(1):51–5. DOI: 10.1007/S40037-015-0237-1
    Back to article
  24. 24Traub RE. Classical Test Theory in Historical Perspective. Educational Measurement: Issues and Practice. 2005;16(4):8–14. DOI: 10.1111/j.1745-3992.1997.tb00603.x
    Back to article
  25. 25Rice N, Pêgo JM, Collares CF, Kisielewska J, Gale T. The development and implementation of a computer adaptive progress test across European countries. Computers and Education: Artificial Intelligence. 2022;3:100083. DOI: 10.1016/j.caeai.2022.100083
    Back to article
  26. 26Warm TA. Weighted likelihood estimation of ability in item response theory. Psychometrika. 1989;54(3):427–50. DOI: 10.1007/BF02294627
    Back to article
  27. 27van Wijk EV, van Blankenstein FM, Donkers J, Janse RJ, Bustraan J, Adelmeijer LGM, et al. Does ‘summative’ count? The influence of the awarding of study credits on feedback use and test-taking motivation in medical progress testing. Advances in Health Sciences Education. 2024. 2024-03-19. DOI: 10.1007/s10459-024-10324-4
    Back to article
  28. 28Fraley C, Raftery AE. Model-Based Clustering, Discriminant Analysis, and Density Estimation. Journal of the American Statistical Association. 2002;97(458). DOI: 10.1198/016214502760047131
    Back to article
  29. 29R Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria; 2021 [cited 2024]. Available from: https://www.R-project.org/.
    Back to article
  30. 30Scrucca L, Fop M, Murphy TB, Raftery AE. mclust 5: clustering, classification and density estimation using Gaussian finite mixture models. The R Journal. 2016;8(1):289–317. DOI: 10.32614/RJ-2016-021
    Back to article
  31. 31Biernacki C, Celeux G, Govaert G. Assessing a mixture model for clustering with the integrated completed likelihood. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2000;22(7):719–25. DOI: 10.1109/34.865189
    Back to article
  32. 32McLachlan GJ, Rathnayake S. On the number of components in a Gaussian mixture model. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery. 2014;4(5):341–55. DOI: 10.1002/widm.1135
    Back to article
  33. 33Simpson E. The interpretation of Interaction in Contingency Tables. Journal of the Royal Statistical Society Series B (Methodological). 1951;12(2):3. DOI: 10.1111/j.2517-6161.1951.tb00088.x
    Back to article
  34. 34Artino A, Dong T, DeZee K, Gilliland W, Waechter D, Cruess D, et al. Development and initial validation of a survey to assess students’ self-efficacy and metacognitive strategies in medical school. Academic Medicine. 2012;87(2):205–10. DOI: 10.1097/ACM.0b013e31823bcbfe
    Back to article
  35. 35Cecilio-Fernandes D, Kerdijk W, Jaarsma ADC, Tio RA. Development of cognitive processing and judgments of knowledge in medical students: Analysis of progress test results. Medical teacher. 2016;38(11):1125–9. DOI: 10.3109/0142159X.2016.1170781
    Back to article
  36. 36Maki RH, Jonas D, Kallod M. The relationship between comprehesion and metacomprehension ability. Psychological Bulletin Review. 1994;1(1):126–9. DOI: 10.3758/BF03200769
    Back to article
  37. 37Kruger J, Dunning D. Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Pers Soc Psych. 1999;77(6):1121–34. DOI: 10.1037//0022-3514.77.6.1121
    Back to article
  38. 38Brydges R, Butler D. A reflective analysis of medical education research on self-regulation in learning and practice. 2015;49(1):55–63. DOI: 10.1111/medu.12517
    Back to article
  39. 39Cleave-Hogg D, Morgan PJ. Experiental learning in an anaesthesia simulation centre: analysis of students’ comments. Medical Teacher. 2009;24:23–6. DOI: 10.1080/00034980120103432
    Back to article
  40. 40Dornan T, Scherpbier A, King N, Boshuizen N. Clinical teachers and problem-based learning: a phenomenological study. Medical Education. 2005;39(2):163–70. DOI: 10.1111/j.1365-2929.2004.01914.x
    Back to article
  41. 41Lindblom-Ylänne S, Parpala A, Postareff L. What constitutes the surface approach to learning in the light of new empirical evidence? Studies in Higher Education. 2019;44(12):2183–95. DOI: 10.1080/03075079.2018.1482267
    Back to article
  42. 42Krathwohl DR. A Revision of Bloom’s Taxonomy: An Overview. Theory Into Practice. 2002;41(4). DOI: 10.1207/s15430421tip4104_2
    Back to article
  43. 43Gardner-Medwin AR. Confidence assessment in the teaching of basic science. Research in Learning Technology. 1995;3(1). DOI: 10.3402/rlt.v3i1.9597
    Back to article
  44. 44Cash B, Mitchner NA, Ravyn D. Confidence-Based Learning CME: Overcoming Barriers in Irritable Bowel Syndrome With Constipation. Journal of Continuing Education in the Health Professions. 2011;31(3). DOI: 10.1002/chp.20121
    Back to article
  45. 45Luetsch K, Burrows J. Certainty rating in pre-and post-tests of study modules in an online clinical pharmacy course - A pilot study to evaluate teaching and learning. BMC Medical Education. 2016;16(1). DOI: 10.1186/s12909-016-0783-1
    Back to article
  46. 46Gardner-Medwin AR. Analysis of exams using certainty-based marking. Proc Physiol Soc. 2006;3.
    Back to article
  47. 47Smrkolj Š, Bančov E, Smrkolj V. The reliability and medical students’ appreciation of certainty-based marking. Int J Environ Res Public Health. 2022;19(3):1706. DOI: 10.3390/ijerph19031706
    Back to article
  48. 48Wu C, Qu Y, Wang L. Confidence calibration, risk preference, and certainty-based marking: a prospect-theory-based psychometric analysis. Psychometrika. 2021;86(3):741–63. DOI: 10.1007/s11336-021-09759-0
    Back to article
DOI: https://doi.org/10.5334/pme.1673 | Journal eISSN: 2212-277X
Journal RSS Feed
Language: English
Submitted on: Dec 30, 2024
Accepted on: Oct 19, 2025
Published on: Dec 3, 2025
Published by: Ubiquity Press
In partnership with: Paradigm Publishing Services
Publication frequency: 1 issue per year

© 2025 Elise V. van Wijk, Jeroen Donkers, Peter C. J. de Laat, Ariadne A. Meiboom, Bram Jacobs, Jan Hindrik Ravesloot, René A. Tio, Frederike M. M. Oud, Jeroen P. Kooman, André J. A. Bremers, Alexandra M. J. Langers, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.

Previous articleVolume 14 (2025): Issue 1