Comparison of validity, repeatability and reproducibility of the Peer Assessment Rating (PAR) between digital and conventional study models

Pasapula, Sridhar; Sherriff, Martyn; Breckon, Jeremy; Bister, Dirk; Abela, Stefan

doi:10.21307/aoj-2020-126

Abstract

Introduction

The validity, reliability and inter-method agreement of Peer Assessment Scores (PAR) from acrylic models and their digital analogues were assessed.

Method

Ten models of different occlusions were digitised, using a 3 Shape R700 laser scanner (Copenhagen, Denmark). Each set of models was conventionally and digitally PAR-scored twice in random order by 10 examiners. The minimum time between repeat measurements was two weeks. The repeatability was assessed by applying Carstensen’s analysis. Inter-method agreement (IEMA) was assessed by Carstensen’s limit of agreement (LOA).

Results

Intra-examiner repeatability (IER) for the unweighted and weighted data was slightly better for the conventional rather than the digital models. There was a slightly higher negative bias of -1.62 for the weighted PAR data for the digital models. IEMA for the overall weighted data ranged from −8.70 – 5.45 (95% Confidence Interval, CI). Intra-class Correlation Coefficients (ICC) for the weighted data for conventional, individual and average scenarios were 0.955 (0.906 – 0.986 CI), 0.998 (0.995 – 0.999 CI). ICC for the weighted digital data, individual and average scenarios were 0.99 (0.97 – 1.00) and 1.00. The percentage reduction required to achieve an optimal occlusion increased by 0.4% for the digital scoring of the weighted data.

Conclusion

Digital PAR scores obtained from scanned plastic models were valid and reliable and, in this context, the digital semi-automated method can be used interchangeably with the conventional method of PAR scoring.

References

1. Richmond S, Shaw WC, O’Brien KD, Buchanan IB, Jones R, Stephens CD et al. The development of the PAR Index (Peer Assessment Rating): reliability and validity. Eur J Orthod 1992;14:125-39.10.1093/ejo/14.2.125
Search in Google Scholar Back to article
2. Richmond S, Shaw WC, Roberts CT, Andrews M. The PAR Index (Peer Assessment Rating): methods to determine outcome of orthodontic treatment in terms of improvement and standards. Eur J Orthod 1992;14:180-7.10.1093/ejo/14.3.180
Search in Google Scholar Back to article
3. Committee BOSCS. Guidelines for Primary Care Trusts and Local Health Boards to assess the treatment outcome of patients treated by specialist orthodontists or dentists using the Peer Assessment Rating (PAR) Index. 2009.
Search in Google Scholar Back to article
4. Firestone AR, Beck FM, Beglin FM, Vig KW. Evaluation of the peer assessment rating (PAR) index as an index of orthodontic treatment need. Am J Orthod Dentofacial Orthop 2002;122:463-9.10.1067/mod.2002.128465
Search in Google Scholar Back to article
5. DeGuzman L, Bahiraei D, Vig KW, Vig PS, Weyant RJ, O’Brien K. The validation of the Peer Assessment Rating index for malocclusion severity and treatment difficulty. Am J Orthod Den-tofacial Orthop 1995;107:172-6.10.1016/S0889-5406(95)70133-8
Search in Google Scholar Back to article
6. Houston WJ. The analysis of errors in orthodontic measurements. Am J Orthod 1983;83:382-90.10.1016/0002-9416(83)90322-6
Search in Google Scholar Back to article
7. Colton T. Controlled clinical trials. Am Rev Respir Dis 1974;110 Part 2:20-4.
Search in Google Scholar Back to article
8. Roberts CT, Richmond S. The design and analysis of reliability studies for the use of epidemiological and audit indices in orthodontics. Br J Orthod 1997;24:139-47.10.1093/ortho/24.2.1399218112
Search in Google Scholar Back to article
9. M N. IUPAC Compendium of Chemical Terminology 2nd Edition. IUPAC Compendium of Chemical Terminology Gold Book, 2. 1997 (2nd Edition).
Search in Google Scholar Back to article
10. Hajeer MY, Millett DT, Ayoub AF, Siebert JP. Applications of 3D imaging in orthodontics: part I. J Orthod 2004;31:62-70.10.1179/14653120422501134615071154
Search in Google Scholar Back to article
11. Shastry S, Park JH. Evaluation of the use of digital study models in postgraduate orthodontic programs in the United States and Canada. Angle Orthod 2014;84:62-7.10.2319/030813-197.1868304723742197
Search in Google Scholar Back to article
12. Rheude B, Sadowsky PL, Ferriera A, Jacobson A. An evaluation of the use of digital study models in orthodontic diagnosis and treatment planning. Angle Orthod 2005;75:300-4.
Search in Google Scholar Back to article
13. Whetten JL, Williamson PC, Heo G, Varnhagen C, Major PW. Variations in orthodontic treatment planning decisions of Class II patients between virtual 3-dimensional models and traditional plaster study models. Am J Orthod Dentofacial Orthop 2006;130:485-91.10.1016/j.ajodo.2005.02.02217045148
Search in Google Scholar Back to article
14. Mayers M, Firestone AR, Rashid R, Vig KW. Comparison of peer assessment rating (PAR) index scores of plaster and computer-based digital models. Am J Orthod Dentofacial Orthop 2005;128:431-4.10.1016/j.ajodo.2004.04.03516214623
Search in Google Scholar Back to article
15. Institution BS. British Standards Glossary of Dental Terms BS - 4492 London: BSI; 1983.
Search in Google Scholar Back to article
16. Carstensen B. Comparing and predicting between several methods of measurement. Biostatistics 2004;5:399-413.10.1093/biostatistics/kxg043
Search in Google Scholar Back to article
17. Brown R, Richmond S. An update on the analysis of agreement for orthodontic indices. Eur J Orthod 2005;27:286-91.10.1093/ejo/cjh078
Search in Google Scholar Back to article
18. King GJ, McGorray SP, Wheeler TT, Dolce C, Taylor M. Comparison of peer assessment rat-ings (PAR) from 1-phase and 2-phase treatment protocols for Class II malocclusions. Am J Or-thod Dentofacial Orthop 2003;123:489-96.10.1016/S0889-5406(03)00045-3
Search in Google Scholar Back to article
19. Cicchetti DV, Sparrow SA. Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior. Am J Ment Defic 1981;86:127-37.
Search in Google Scholar Back to article
20. Fleiss JL. Statistical Methods for Rates and Proportions, 2nd Edition. New York: Miley; 1981.
Search in Google Scholar Back to article
21. Bell A, Ayoub AF, Siebert P. Assessment of the accuracy of a threedimensional imaging sys-tem for archiving dental study models. J Orthod 2003;30:219-23.10.1093/ortho/30.3.219
Search in Google Scholar Back to article
22. Tomassetti JJ, Taloumis LJ, Denny JM, Fischer JR Jr. A comparison of 3 computerized Bolton tooth-size analyses with a commonly used method. Angle Orthod 2001;71:351-7.
Search in Google Scholar Back to article
23. Mullen SR, Martin CA, Ngan P, Gladwin M. Accuracy of space analysis with emodels and plaster models. Am J Orthod Dentofacial Orthop 2007;132:346-52.10.1016/j.ajodo.2005.08.044
Search in Google Scholar Back to article
24. Santoro M, Galkin S, Teredesai M, Nicolay OF, Cangialosi TJ. Comparison of measurements made on digital and plaster models. Am J Orthod Dentofacial Orthop 2003;124:101-5.10.1016/S0889-5406(03)00152-5
Search in Google Scholar Back to article
25. Stevens DR, Flores-Mir C, Nebbe B, Raboud DW, Heo G, Major PW. Validity, reliability, and reproducibility of plaster vs digital study models: comparison of peer assessment rating and Bolton analysis and their constituent measurements. Am J Orthod Dentofacial Orthop 2006;129:794-803.10.1016/j.ajodo.2004.08.02316769498
Search in Google Scholar Back to article
26. Malik OH, Abdi-Oskouei M, Mandall NA. An alternative to study model storage. Eur J Orthod 2009;31:156-9.10.1093/ejo/cjn07119028672
Search in Google Scholar Back to article
27. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986;1:307-10.
Search in Google Scholar Back to article
28. Andrews CK. Validity and reliability of peer assessment rating index scores of digital and plaster models. Thesis. The Ohio State University, 2008.
Search in Google Scholar Back to article