Psychometric Validation of an Instrument for Measuring Patient Experiences with Outpatient Healthcare

Sočan, Gregor; De Boer, Dolf; Murko, Eva; Kralj, Marcel; Ropret, Nina; Zaletel, Metka

doi:10.2478/sjph-2023-0021

Abstract

Aim

Recently, a patient-reported experience measure (PREM) was developed in Slovenia to assess patients’ experiences with outpatient specialist healthcare clinics. The aim of this study was to evaluate the psychometric properties (including factor structure, reliability, convergent validity, and response distribution) of the questionnaire.

Methods

The sample consisted of 8,406 adult participants treated in 171 specialist clinics from different medical fields. Participants voluntarily and anonymously responded to either the paper or online survey.

Results

Descriptive statistics show meaningful response patterns with a general tendency towards favourable assessments. The psychometric analyses of the scales evaluating doctor’s and nurses’ work, respectively, generally showed a good fit of the unidimensional factor model as well as the Rasch model, high factor loadings and very good to satisfactory reliability. The Rasch scaling showed that these scales were most informative for patients with relatively unfavourable experience ratings.

Conclusions

The results are similar to those found in previous evaluations of PREMs in other countries. Given its good psychometric properties, the Slovenian PREM can be recommended for healthcare evaluations in Slovenia and as a model for the development of similar PREMs in other countries.

References

Goodwin N. Towards people-centered integrated care: From passive recognition to active co-production. Int J Integr Care. 2016;16(2):15. doi: 10.5334/ijic.2492.
Open DOI Search in Google Scholar Back to article
Constand MK, MacDermid JC, Dal Bello-Haas V, Law M. Scoping review of patient-centered care approaches in healthcare. BMC Health Serv Res. 2014;14:271. doi: 10.1186/1472-6963-14-271.
Open DOI Search in Google Scholar Back to article
Carinci F, Van Gool K, Mainz J, Veillard J, Pichora EC, Januel JM, et al. Towards actionable international comparisons of health system performance: expert revision of the OECD framework and quality indicators. Int J Qual Health Care. 2015;27(2):137–146. doi: 10.1093/intqhc/mzv004.
Open DOI Search in Google Scholar Back to article
De Rosis S, Cerasuolo D, Nuti S. Using patient reported measures to drive change in healthcare: The experience of the digital, continuous and systematic PREMs observatory in Italy. BMC Health Serv Res. 2020;20:315. doi: 10.1186/s12913-020-05099-4.
Open DOI Search in Google Scholar Back to article
Bull C, Byrnes J, Hettiarachchi R, Downes M. A systematic review of the validity and reliability of patient reported experience measures. Health Serv Res. 2019;54(5):1023–1035. doi: 10.1111/1475-6773.13187.
Open DOI Search in Google Scholar Back to article
Organization for Economic Co-operation and Development. Recommendations to OECD ministers of health from the high-level reflection group on the future of health statistics [Internet]. 2017 [cited 2022 Mar 15]. Available from: https://www.oecd.org/health/Recommendations-from-high-level-reflection-group-on-the-future-of-health-statistics.pdf
Search in Google Scholar Back to article
Fernandes S, Fond G, Zendjidjian X, Michel P, Baumstarck K, Lancon C, et al. The Patient reported experience measure for improving quality of care in mental health (PREMIUM) project in France: Study protocol for the development and implementation strategy. Patient Prefer Adherence. 2019;13:165–177. doi: 10.2147/PPA.S172100.
Open DOI Search in Google Scholar Back to article
Fujisawa R, Klazinga N. Measuring patient experiences (PREMs): Progress made by the OECD and its member countries between 2006 and 2016. OECD Health Working Papers, No. 102. OECD Publishing: Paris; 2017. doi: 10.1787/893a07d2-en.
Open DOI Search in Google Scholar Back to article
Sjetne, I, Bjertnaes O, Olsen RV, Iversen HH. The Generic Short Patient Experiences Questionnaire (GS-PEQ): Identification of core items from a survey in Norway. BMC Health Serv Res. 2011(1):88. doi: 10.1186/1472-6963-11-88.
Open DOI Search in Google Scholar Back to article
OPEQ, Garratt AM, Bjaertnes OA, Krogstad U, Gulbrandsen P. The OutPatient Experiences Questionnaire (OPEQ): Data quality, reliability, and validity in patients attending 52 Norwegian hospitals. Qual Saf Health Care. 2005;14:433–437. doi: 10.1136/qshc.2005.014423.
Open DOI Search in Google Scholar Back to article
CAHPS, The American Consumer Assessment of Healthcare Providers and Systems (CAHPS) [Internet]. 2012 [cited 2023 April 3]. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3748811/pdf/nihms391989.pdf
Search in Google Scholar Back to article
Delnoij DM, Rademakers JJ, Groenewegen PP. The Dutch Consumer Quality Index: an example of stakeholder involvement in indicator development. BMC Health Serv Res. 2010;10:88. doi: 10.1186/1472-6963-10-88.
Open DOI Search in Google Scholar Back to article
OECD. Health at a glance [Internet]. 2021 [cited 2023 April 4]. Available from: https://www.oecd-ilibrary.org/social-issues-migration-health/health-at-a-glance-2021_ae3016b9-en
Search in Google Scholar Back to article
Ministrstvo za zdravje. Nacionalna anketa [Internet]. 2007 [cited 2023 April 3]. Available from: https://www.pb-begunje.si/gradiva/Rezultati2007135143657636.pdf
Search in Google Scholar Back to article
Murko E, Kralj M, Poldrugovac M, Ropret N, Šetinc M, Zaletel M. Izkušnje pacientov z zunajbolnišnično specialistično zdravstveno obravnavo: raziskava v Sloveniji med odraslimi pacienti. Javno zdravje. 2021;1:1–9. doi: 10.26318/JZ-2021-1.
Open DOI Search in Google Scholar Back to article
Pettersen K, Veenstra M, Guldvog B; Kolstad A. The Patient Experiences Questionnaire: Development, validity and reliability. Int J Quality Health Care. 2004;16(6):453–463. doi: 10.1093/intqhc/mzh074.
Open DOI Search in Google Scholar Back to article
Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Struct Equ Modeling. 1999;6(1):1–55. doi: 10.1080/10705519909540118.
Open DOI Search in Google Scholar Back to article
Bond TG, Fox CM. Applying the Rasch model: Fundamental measurement in the human sciences. 2nd ed. Mahwah: Lawrence Erlbaum Associates Publishers; 2007.
Search in Google Scholar Back to article
R Core Team. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2021.
Search in Google Scholar Back to article
Rosseel Y. lavaan: An R package for structural equation modeling. J Stat Softw. 2012;48(2):1–36. doi: 10.18637/jss.v048.i02.
Open DOI Search in Google Scholar Back to article
Kelley K. MBESS: The MBESS R Package. R package version 4.9.0 [Internet]. 2022 [cited 2023 April 3]. Available from: https://CRAN.R-project.org/package=MBESS
Search in Google Scholar Back to article
Mair P, Hatzinger, R, Maier MJ. eRm: Extended rasch modeling 1.0–2 [Internet]. 2021 [cited 2023 April 3]. Available from: https://cran.r-project.org/package=eRm.2021
Search in Google Scholar Back to article
Mair P, De Leeuw J. aspect: A general framework for multivariate analysis with optimal scaling. R package version 1.0–5. [Internet]. 2018 [cited 2023 April 3]. Available from: https://cran.r-project.org/package=aspect
Search in Google Scholar Back to article
Revelle W. psych: Procedures for personality and psychological research, version 2.1.9. [Internet]. 2021 [cited 2023 April 3]. Evanston: Northwestern University; 2021. Available from: https://CRAN.R-project.org/package=psych
Search in Google Scholar Back to article
Stekhoven DJ. missForest: Nonparametric missing value imputation using random forest. R package version 1.5. 2022.
Search in Google Scholar Back to article
McCoach DB, Gable RK, Madura JP. Instrument development in the affective domain: School and corporate applications. 3rd ed. New York: Springer; 2013.
Search in Google Scholar Back to article
Raykov T, Marcoulides GA. Introduction to psychometric theory. New York: Routledge, Taylor & Francis; 2011.
Search in Google Scholar Back to article
Bruster S. The Picker Patient Experience Questionnaire: Development and validation using data from in-patient surveys in five countries. Int J Qual Health Care. 2002;14(5):353–358. doi: 10.1093/intqhc/14.5.353.
Open DOI Search in Google Scholar Back to article
de Boer D, Delnoij D, Rademakers J. The discriminative power of patient experience surveys. BMC Health Serv Res. 2011;11:332. doi: 10.1186/1472-6963-11-332.
Open DOI Search in Google Scholar Back to article
Hays RD, Skootsky SA. Patient experience with in-person and telehealth visits before and during the COVID-19 pandemic at a large integrated health system in the United States. J Gen Intern Med. 2022;37(4):847–852. doi: 10.1007/s11606-021-07196-4.
Open DOI Search in Google Scholar Back to article
Damman OC, Hendriks M, Rademakers J, Spreeuwenberg P, Delnoij DM, Groenewegen PP. Consumers’ interpretation and use of comparative information on the quality of health care: the effect of presentation approaches. Health Expect. 2012;15(2):197–211. doi: 10.1111/j.1369-7625.2011.00671.
Open DOI Search in Google Scholar Back to article
OECD. Measuring what matters: The patient reported indicator surveys: Patient reported indicators for assessing health system performance [Internet]. 2019 [cited 2023 April 17]. Available from: https://www.oecd.org/health/health-systems/Measuring-what-matters-the-Patient-Reported-Indicator-Surveys.pdf
Search in Google Scholar Back to article
Van der Veer SN, Jager KJ, Visserman E, Beekman RJ, Boeschoten EW, de Keizer NF, et al. Development and validation of the Consumer Quality index instrument to measure the experience and priority of chronic dialysis patients. Nephrol Dial Transplant. 2012;27(8):3284–3291. doi: 10.1093/ndt/gfs023.
Open DOI Search in Google Scholar Back to article
NHS, Care Quality Commission. Adult inpatient survey: quality and methodology report [Internet]. 2021 [cited 2023 April 17]. Available from: https://www.cqc.org.uk/sites/default/files/2022-09/20220920_aip22_QualityMethodology.odt
Search in Google Scholar Back to article
Johnson TP, Wislar JS. Response rates and nonresponse errors in surveys. JAMA. 2012;307(17):1805–1806. doi: 10.1001/jama.2012.3532.
Open DOI Search in Google Scholar Back to article
Faraz A, Burt J, Roland M. Measuring patient experience: Concepts and methods. Patient. 2014;7:235–241. doi: 10.1007/s40271-014-0060-5.
Open DOI Search in Google Scholar Back to article
Garratt AM, Bjørngård JH, Dahle KA, et al. The Psychiatric Out-Patient Experiences Questionnaire (POPEQ): Development, reliability and validity. Nordic J Psychiatry (in press).
Search in Google Scholar Back to article
Millsap RE. Statistical approaches to measurement invariance. New York: Routledge; 2011.
Search in Google Scholar Back to article

Psychometric Validation of an Instrument for Measuring Patient Experiences with Outpatient Healthcare

Abstract

Aim

Methods

Results

Conclusions

Paradigm

My account