Introduction
Children and young people (CYP) who are medically complex have multiple and/or significant chronic health problems. This group represents approximately 10% of patients attending children’s hospitals, but accounts for 60% of expenditure [12]. CYP who are medically complex require support from different health care teams frequently based across more than one service and location. In the absence of planned coordinated care for these CYP, missed and/or duplicated care increases costs for the health system and out-of-pocket expenses for families [3]. The cost of uncoordinated care is estimated to be up to 35% higher than coordinated care [4].
A care coordination service for CYP with chronic and/or complex health conditions, developed at the Sydney Children’s Hospitals Network (SCHN) in Sydney, New South Wales (NSW), Australia, known as Kids Guided Personalised Service (KidsGPS) has already demonstrated successful outcomes in the metropolitan setting [3]. The service is focussed on CYP with increased care coordination needs due to several factors, including medical complexity, vulnerability, involvement of multiple health care teams, and reduced access to appropriate services. Evaluation of KidsGPS demonstrated a 40% reduction in emergency department (ED) presentations and 42% reduction in day-only admissions over 2 years for 534 CYP who had over 50 different diagnoses [3]. Through the adoption of KidsGPS, approximately $2.5 million/annum was saved for the tertiary hospital network, and >50,000 km of family travel was prevented [3]. Systematic review and meta-analyses data published by our research team have indicated that paediatric integrated care interventions significantly improve CYP related quality of life scores compared to usual care [5].
In NSW, Australia, 500,000 children, 30% of the state’s total, live outside major cities. To address the healthcare needs of this population, Rural Kids Guided Personalised Service (RuralKidsGPS) was co-designed with parents and professionals as an innovative child-centred model of care for CYP living with medical complexity in rural and remote regions. Within the RuralKidsGPS model, a new role was established in each LHD for a paediatric care coordinator responsible for facilitating care integration and establishing a circle of coordination formed by the leads from the health service, community and family [1]. The paediatric care coordinators work closely with local and tertiary care teams at SCHN, and families, to meet the CYP’s needs through the development of shared care plans and technology-enabled healthcare. Importantly, the RuralKidsGPS model of care aims to build capacity for the family and local healthcare teams.
This paper describes a research protocol aimed at evaluating the impact of RuralKidsGPS on health service utilisation, implementation outcomes, barriers and enablers to delivering RuralKidsGPS and the cost-effectiveness of this program for families and the healthcare system.
Methodology
Study design
The evaluation of RuralKidsGPS will consist of an integrated impact, implementation and economic evaluation embedded within a quasi-experimental study across four rural NSW local health districts (LHDs) in Southern, Northern and Western New South Wales and Murrumbidgee (Figure 1).
Figure 1
Local Health Districts involved in the RuralKidsGPS study.
The primary aim of this evaluation is to assess the impact of RuralKidsGPS on health service utilisation (ED presentations, hospital admissions, and outpatient (OP) appointments), among CYP enrolled in RuralKidsGPS intervention compared to usual care.
The secondary aims are to:
Describe trends in health service utilisation (ED presentations, hospital admissions, OP appointments, and general practice (GP) attendances) over time;
Assess the change in parent reported satisfaction with care, parental wellbeing, and parent and CYP reported quality of life at enrolment, at 6 and 12 months after recruitment to RuralKidsGPS;
Assess the implementation of RuralKidsGPS model from the perspective of families, CYP, healthcare providers, managers, and practitioners, and factors that help or hinder this implementation to inform further adaption of the model and ensure scalability;
Assess the cost-effectiveness of RuralKidsGPS model of care compared with standard care.
Ethics
This study will be guided by the RuralKidsGPS Steering and Evidence Translation Committee which includes in its membership representatives from all participating local health districts and consumers. Ethical oversite for this study is held by the SCHN Human Research Ethics Committee (2021/ETH01055).
Study cohort
The study population includes CYP with medical complexity aged 0 to 18 years, referred for care coordination, who meet the following criteria:
– A diagnosis expected to last at least 12 months
– And if ≥ 6months of age, frequent use of health services within the last six-month period, defined as:
Two or more ED presentations, OR
Hospital admission for seven days or more, OR
Two or more outpatient appointments.
A cohort of CYP meeting these criteria and referred for care coordination, their parents and guardians, and clinical and non-clinical staff involved in the planning and/or delivery of the RuralKidsGPS model of care, will all form a part of the study cohort.
Recruitment
Children meeting the inclusion criteria for care coordination as described above will be recruited within each of the four LHDs. For children enrolled into the local service, the paediatric care coordinator (PCC) will discuss with each family their potential participation in the evaluation project. After providing a brief overview of the project, the PCC will ask permission for the contact details (parent name, contact number, email, and child name) to be shared with the researcher embedded with the local team. The embedded researcher will then contact the parent/carer and provide them with the participant information sheet and consent form.
Clinical staff (e.g. healthcare providers), and non-clinical staff (e.g. service managers) involved in the care coordination of the children will be recruited from each of the four LHDs and SCHN. Members of the research team will make direct contact with clinical and non-clinical staff and will provide them with an information sheet and consent form which describes what is involved in the research and that participation is voluntary. Upon agreement and consent, the implementation evaluation team will organise a suitable time to conduct the interview or focus group. Links to study surveys will be sent via an invitation email including the participant information form. These forms will also will be embedded in the electronic surveys and participants will be asked to confirm that they have read these and that they consent prior to completing the survey.
Intervention
The RuralKidsGPS model uses the circle of coordination for CYP with medical complexity, which places the CYP and their needs at the centre of the system for integrating care (Figure 2). Families are guided by local PCCs who are responsible for facilitating care integration and establishing the circle of coordination which is formed by leads from the hospital, community, and family.
Figure 2
RuralKidsGPS Circle of Coordination (adapted from Cohen, et al. 2011).
The PCCs work with families to:
Understand their personalised needs.
Develop and implement an integrated family-centred shared care plan.
Build capacity for family self-management and deliver practical holistic support.
The PCCs also work with treating teams to identify opportunities to integrate and share care among central units and local teams, deliver virtually enabled care, streamline appointments, avoid unnecessary hospitalisations, and prevent avoidable ED attendances.
Impact evaluation
Health Service Use
A cohort-level evaluation of health utilisation (primary outcome) will be conducted. A cohort of children receiving the RuralKidsGPS will be recruited from the participating LHDs. Routinely collected administrative health data for these children will be requested and linked by an independent group, the NSW Centre for Health Record Linkage (CHeReL) (Figure 3). Using this deidentified linked data, we will compare the change and trend in health utilisation of children (ED presentations, hospital admissions, and outpatient (OP) appointments) who received RuralKidsGPS, to that of a matched comparison group.
Figure 3
Sources of data for linkage.
The comparison group will be identified using the linked administrative health data (ED presentations, hospital admissions, and OP appointments) and frequency matched with 20 children (to each case) by clinical diagnosis, year of birth, sex, local health district who and meet the eligibility criteria for health service use within the past 6 months (outlined in the Study Cohort section above) but who did not access RuralKidsGPS. A population-level evaluation will also be conducted which will include all CYP in the 4 health districts meeting the inclusion criteria.
Impact Evaluation: Patient and family reported outcomes data collection
In our intervention cohort, secondary child and parent reported outcomes will assess family reported experiences and parental satisfaction with care integration (PICS) [6], out of pocket expenses (adapted Mumford Scale) [7], parent and child quality of life (EQ5D5L) [8], (PedsQL4.0) [9], (CHU9D) [10] and parental wellbeing (WEMWBS) [11]. These measures and basic demographic data (e.g. date of birth, postcode for estimation of Indexes of Relative Advantage and Disadvantage [12], gender, Aboriginal and Torres Strait Islander Status) will be collected via phone and secure electronic survey by a researcher embedded in each of the LHDs, at three time points: enrolment, 6 months and 12 months later to enable longitudinal comparisons, (Table 1).
Table 1
Family reported outcome measures.
| OUTCOME MEASURES | PURPOSE |
|---|---|
| Paediatric Integrated Care Survey (PICS) [6] | Measures experiences of integration of child’s care |
| Adapted Mumford scale [7] | Measuring non-medical out of pocket expenses for families |
| EuroQol 5 Dimension 5 Level (EQ-5D-5L) [8] | Health status measure |
| Paediatric Quality of Life Measure (PedsQL4.0) [9] | Measuring health paediatric health outcomes |
| Child Health Utility 9D (CHU 9D) [10] | Instrument used to adjust quality of life years from PedsQL4.0 score for economic evaluation |
| Warwick-Edinburgh Mental Well-being Scale (WEMWBS) [11] | Understanding and measuring mental health and wellbeing |
Impact Evaluation Analyses
Based on local data from the participating LHDs, we aim to recruit 400 children across the four LHDs. We will match each case by 20 controls, resulting in a total of 8,000 controls. This will provide 90% power to detect a 20% difference in the number of health service use occasions, including outpatient visits and hospital admissions between intervention and comparison groups. Having 400 cases will also provide the power of 90% to detect a within-group 20% decrease in the number of health service use occasions. This sample size will also allow us to detect a clinically important difference (4.36 points) for child health related quality of life in children post- vs pre-intervention change in the PedsQL4.0 Total Scale Score [9].
We will calculate rate of health service use (e.g. ED, hospital admissions and OP visits) in person-years for the intervention and comparison groups during the baseline (3 years prior to enrolment) and the intervention period (12 months after enrolment), respectively. To assess the effectiveness of this intervention, we will conduct multivariable Poisson (negative binomial if over-dispersion is present) regression analysis with intervention status (intervention versus comparison) as the main independent variable and the rate of health service use (ED, hospital admissions and OP visits, separately and combined) during the intervention period as the outcome variable.
To examine how health service use changes over time between intervention and comparison groups, we will analyse rates of health service use during baseline and intervention periods using mixed effect Poisson regression models, including the interaction term between intervention status and time (after RuralKidsGPS implementation versus before RuralKidsGPS). In all regression analyses, we will control for participant socio-demographics (e.g., age at enrolment, sex, Aboriginal and Torres Strait Islander status, socio-economic status) and clinical factors (e.g., type and severity of health conditions), and baseline rate of health service use. Considering potential clustering effect by site, we will use the generalised estimating equations (GEE) approach in the regression analysis.
Additionally, we will plot monthly rate of health service use for both intervention and comparison groups over baseline and intervention periods. We will undertake an interrupted time series (ITS) analysis to gain a better understanding of how the effect of intervention changes over time (e.g. to identify when the intervention starts to take effect). ITS is one of the most effective quasi-experimental evaluation methods and has been used in many health intervention evaluations [13]. An ITS analysis allows us to control for time-varying confounders such as seasonality and co-occurring competing interventions, thereby addressing important threats to internal validity and causal inference. Figure 4 illustrates using ITS to measure change before and after intervention.
Figure 4
Using interrupted time series analysis to measure change before and after intervention. Adapted from Kontopantelis et al 2015 [13].
Regarding patient and family reported outcomes, we will describe them at each survey time point using mean and standard deviation, or median and inter-quartile range. We will conduct regression analysis of these outcomes at each survey (pre-intervention, 6-month and 12-month post-intervention), respectively, with intervention status as the main independent variable. We will control for participants’ socio-demographic, clinical and baseline outcome variables, and use the GEE approach to account for potential clustering effect by site. We will conduct intention-to-treat analysis.
Implementation Evaluation
We will undertake a mixed methods implementation evaluation, underpinned by the Consolidated Framework for Implementation Research (CFIR) [14]. The CFIR is a comprehensive meta-theoretical framework that provides a taxonomy of standardised operationally defined constructs to guide systematic assessment in the identification of factors (barriers and enablers) influencing implementation of complex programs within complex contexts [14]. Contextually relevant strategies for successful implementation as well as barriers and facilitators for adoption, delivery and sustainability will be identified to inform future rollouts of the RuralKidsGPS model of care. The mixed methods research will also assess implementation outcomes including acceptability, appropriateness, and adaptability of the RuralKidsGPS model as viewed by children and families, healthcare providers, and service managers, informing answers to specific questions that reflect implementation outcomes proposed by Proctor et al., (2011) (Table 2) [15].
Table 2
Defining implementation outcomes for RuralKidsGPS, based on Proctor’s framework.
| OUTCOME | QUESTION |
|---|---|
| Acceptability | Do clinician and non-clinicians, parents and children view RuralKidsGPS as agreeable? |
| Adoption | To what extent do clinicians and parents use RuralKidsGPS? |
| Appropriateness | Do all stakeholders perceive RuralKidsGPS as relevant & useful? |
| Fidelity | Is RuralKidsGPS applied as intended? Are all component parts of the intervention delivered as planned? |
| Feasibility | Are the component parts of RuralKidsGPS practical to deliver within the service? |
| Coverage | How many service users of those eligible are reached? |
| Cost* | How much does it cost to successfully implement RuralKidsGPS? |
| Sustainability | What are the factors that will allow RuralKidsGPS to be scaled-up further? |
[i] *This outcome will be informed by the economic evaluation.
Implementation evaluation qualitative data collection
Semi-structured interviews will be undertaken by trained researchers with CYP aged over seven years, their parents, or caregivers, after they have been accessing RuralKidsGPS for at least 6 months. Children aged over seven years are considered capable of understanding the purpose of the research and of making informed decisions as to whether to participate in a brief interview or not [16].
Healthcare providers and managers involved in delivering the model of care will also be interviewed between 6 and 12 months after implementation of RuralKidsGPS. Purposive sampling will be used to recruit a diverse sample of service users (children, young people, and families) and providers across the four sites. Key stakeholder workshops held within each LHD, will present findings to enable reflection on the appropriateness and acceptability of the model, identify future adjustments to the model and implementation strategies with consideration of local contexts. The outcomes of the workshops will inform the development of an implementation guide to support future scaling up of RuralKidsGPS.
Implementation evaluation quantitative data collection
Data will be collected for each child before and after enrolment in the RuralKidsGPS care coordination model. In addition, key LHD staff involved in the delivery or management of RuralKidsGPS will be sent a link to a secure online survey, which includes the Acceptability, Appropriateness and Feasibility measure [15], Normalization Process Theory (NoMAD) tool [17] and Routinisation and Sustainability of work practices in long term care [18], at three time points: enrolment, 6 months and 12 months later as described in Table 3.
Table 3
Quantitative data collection for children and families, clinician and non-clinicians at each of the four sites.
| GROUP | INSTRUMENT | BASELINE (ENROLMENT) | 6 MONTHS | 12 MONTHS |
|---|---|---|---|---|
| Families | Paediatric Integrated Care Survey [6] | x | x | x |
| Clinician and non-clinicians | Intervention Acceptability, Appropriateness and Feasibility measure [15] | x | x | x |
| Clinicians and non-clinicians | NoMAD (Normalization Process Theory tool) [17] | x | x | |
| Clinician and non-clinicians | Routinisation and sustainability of work practices in long term care [18] | x |
Implementation evaluation analyses
All instruments will be scored according to the manuals provided by the authors who initially developed these tools. Measures will be compared over time using t-tests for simple comparisons, or analysis of variance (ANOVA) for comparisons across three time points based on available data (see Table 3). The Routinization and Sustainability of work practices in long term care tool [18] will be used only after RuralKidsGPS has been operational for at least 12 months, providing a one-time measure. This may be repeated in the future if it is decided that the model of care will continue to be supported beyond the current study period.
The three-step iterative process for coding the interview and focus group data will be used to ensure intercoder reliability [19]. The initial stage will involve researchers using the CFIR to independently code an interview transcript. This coding along with inductive insights identified in transcripts will be compared for reliability among the research team and any inconsistencies will be discussed and agreed upon. This process has been shown to lead to a high degree of intercoder reliability among the research team [20]. Once intercoder reliability is established, the remainder of the transcripts will be coded. We will be guided by the Consolidated criteria for Reporting Qualitative research (COREQ) checklist for qualitative research [21]. Qualitative and quantitative data will be synthesised through holistic triangulation [22] to arrive at a deep understanding of the data to answer the questions posed in Table 4.
Table 4
Sources, measures, and tools for evaluation.
| RESEARCH QUESTION | DATA SOURCE |
|---|---|
| 1. What is the impact of RuralKidsGPS on health service use? | Linked data from multiple sources Figure 3 |
| 2. What is the impact of RuralKidsGPS on family reported experiences, and parental satisfaction with care and care coordination? | Paediatric Integrated Care Survey [6] |
| 3. What is the family impact related to the child’s healthcare needs (travel and accommodation costs, care for siblings, missed work, missed school days)? | Adapted Mumford Survey [7]. |
| 4. What is the impact of RuralKidsGPS on parent and child related Quality of Life? | Parents: EQ-5D-5L [8] Children aged <7 years: Parent reported Paediatric Quality of Life Inventory [9] CYP aged 7–17 years: CHU9D [10] |
| 5. What is the impact of RuralKidsGPS on Parental mental wellbeing? | Warwick-Edinburgh Mental Well-being scale [11] |
| 6. What is the cost of implementing the model of care? What is the cost effectiveness? | Average costs per encounter in health service use (ED presentation, hospital admission, outpatients encounter); family cost estimates according to adapted method of Mumford et al., 2018 [7]; Quality adjusted life-years (QALYs) |
| 7. What are the aspects of the model of care and its implementation that support feasibility, acceptability and contextual appropriateness? | Qualitative semi-structured interviews guided by the Consolidated Framework for Implementation Research [25]; and the Proctor Implementation Outcomes Framework [15]. |
| 8. What are the barriers and enablers for implementing the model at scale? | Qualitative semi-structured interviews guided by the Consolidated Framework for Implementation Research [25]; and the Proctor Implementation Outcomes Framework [15]. |
| 9. What are the factors that will allow RuralKidsGPS to be scaled-up further? | Survey for healthcare professionals, researchers and health care managers that includes validated tools: Short intervention acceptability, appropriateness, and feasibility measure [15], Normalization Process Theory Nomad Tool [17], Routinisation and sustainability of work practices in long term care [18]. |
Using NVivo qualitative data analysis software [23] we will undertake thematic analysis taking a hybrid inductive/deductive approach [24] to identify factors that help or hinder implementation, and these factors will be categorised under the CFIR constructs. Thematic analysis will specifically focus on barriers and enablers of effective behaviour change in terms of engagement with services and operation of the new model of care, whilst taking local site contextual factors into account. Such factors may include details of the setting, resource availability, staff attitudes, capabilities and capacity, and level of support from management.
Economic evaluation
An economic analysis will determine incremental cost-effectiveness for the new model of care. Health utilisation data will combine the costs of health services pre and post enrolment for each enrolled child. Average costs per activity (ED presentation, hospital admission, OP visit, GP attendance) will be applied to estimate differences in total costs. Resource use will be multiplied with unit costs to calculate the total expenditure per service area and per client before implementation of the new model of care, as well as over the study period and for each of the follow-up intervals. Relevant healthcare utilisation data will be obtained through the data linkage of the routinely collected national administrative datasets. Detailed data will be collected about care coordination, including the type and modality of care provided, duration and personnel involved in care coordination tasks and interactions with enrolled participants from primary sources. The costs to implement the RuralKidsGPS model of care will also be estimated. The study will undertake detailed probabilistic sensitivity analysis and scenario analysis to account for uncertainty within model parameters and model structure.
To estimate broader personal costs incurred by families, we will conduct a parent survey to estimate the costs associated with attending health services for their child, including accommodation, parking, meals, distances travelled (per kilometre average costs for an average vehicle); lost education opportunities (number of school days missed by the child) and missed work time based on estimates published by Mumford et. al. [7]. The cost analysis will be estimated based on the net savings of down-stream resource utilisation and funding impacts of the new model, from a health service perspective (participating health districts, as well as the state’s health system).
Outcome data will capture changes to parent and child reported quality of life assessed using the validated instruments EQ5D5L15 (parent) [8] and Paediatric Quality of Life Inventory (PedsQL4.0), and Child Health Utility 9D (CHU9D) [10] (internal consistency 𝛼 = 0.89 children and 0.92 parent report). These utilities will be included in the estimation of QALYs (quality-adjusted life-years) by calculating the area under the curve, assuming a linear change between the survey dates. QALYs calculated for the year before the intervention starts are based on the health state utility measured at baseline, which was assumed to be stable during the year before.
Cost-effectiveness will be expressed as incremental change in costs (measured by ED presentations, hospital admissions, or OP visits averted), and cost-utility by changes in QALYs gained. Cost-effectiveness will be measured between pre and post implementation of the intervention and calculated according to the formulas below:
Imputation methods will be applied to missing values and sensitivity analyses will be conducted for key cost parameters in the data sets to ensure the robustness of the estimates. An additional bootstrap analysis with random 1,000-fold resampling of the original population will be conducted to ensure the validity of confidence intervals.
Discussion
CYP with medical complexity living in rural and remote areas deserve equitable access to high quality health care, as experienced by their urban counterparts. In Australia, like many other countries, the healthcare system struggles to meet the needs of CYP with medical complexity in general, and particularly those living remotely. This is because our health system is designed for episodic care, provided by health care professionals highly specialised to manage single organ systems, who work in “silos” defined by their medical specialties, institutions, geographical areas, and informal professional networks [26]. Parents/caregivers of CYP with medical complexity and their healthcare providers experience frustration and confusion when accessing and providing healthcare, because of the disconnectedness and complexity of the Australian healthcare system [27].
Paediatric care coordination as a model for CYP with medical complexity, has received more attention in recent years [2829]. This has been the result of published state and national integrated care strategic frameworks and specific recommendations from paediatric reviews [303132]. While CYP with medical complexity constitute a wide array of different diagnoses, there are similarities amongst this heterogenous group, that lends itself to a diagnoses agnostic model of care, such as RuralKidsGPS [3]. These similar characteristics include; ongoing complex health care needs, frequent usage of multiple health services, need for team-based interdisciplinary care, and for some, use of specialised medical appliances, equipment and disability services [1].
Although the KidsGPS was proven effective in the largest metropolitan children’s hospital network in Australia, its value in rural areas cannot be assumed [3]. The adaptability of the KidsGPS model to rural healthcare settings has not been evaluated, nor has this model undergone rigorous and detailed health economic evaluation to determine whether the costs of the service represent value-for-money through gains in reduced hospital utilization and shifts to lower cost settings (e.g. in primary care) where appropriate. The impacts on quality of life, for children and families, and the views of health professionals and managers implementing these models of care are poorly understood in Australia. These are important considerations for the adaptation and sustainability of such models to varied rural contexts. The mixed methods, integrated impact, implementation, and economic evaluation study outlined in our protocol will provide robust evidence of effectiveness of the model using data from four rural health settings with a geographical coverage of 420,095 km2. This evaluation will be the first in Australia to assess the impact, implementation, and cost effectiveness of a novel model of paediatric care coordination that could improve the health and wellbeing of children and families living in rural communities.
Successful outcomes of this research could have far-reaching healthcare policy implications for rural CYP with medical complexity. A demonstrable change in health service use and acceptability of the model by families and health care personnel, could provide strong incentive for institutionalising this model of care, providing the opportunity for children to receive quality health care in a more equitable manner, closer to home.
Conclusion
This study protocol is designed to conduct a robust evaluation of a rural paediatric care coordination intervention. The results of this study will provide an insight into the impact of this intervention on the child and family and the healthcare system within the Australian context. Moreover, the findings will provide evidence to inform practice and policy in national and international contexts.
Acknowledgements
We would also like to acknowledge the contribution of key partners including Matthew O’Meara, John Preddy, Susan Woolfenden, Danielle Smith, Allan Kerrigan, Tammy O’Connor, Jeffery Fletcher, Joanne Ging, Mick Scarcella, Diana Collings, Srinidhi Paranji, Micheal Kundukulam; the paediatric care coordinators and researchers implementing this program in the Murrumbidgee and Northern, Southern and Western NSW LHDs: Sarah Macauley, Charlene Van Der Put, Lisa Gabb, Philippa Street, Sonia Wainwright, Amie Gough, Sarah Hill, Olivia Rothery, Jack Murphy, Anneliese De Groot, Pramesh Ghimire and Megan Holmes; the KidsGPS team at The Sydney Children’s Hospitals Network and the families involved in co-designing this program.
Funding Information
This research has been supported by NSW Ministry of Health’s Translational Research Grants Scheme (H20/93572) and Paediatric Priority Funding initiative. RL received support through the Financial Markets Foundation for Children, NN received support through the Financial Markets Foundation for Children and NHMRC Investigator Grant (APP1197940).
Competing Interests
The authors have no competing interests to declare.