Introduction
Many hospitalized older adults with multiple health conditions and social circumstances are considered complex cases [1] and face barriers to being discharged from acute care settings [2]. In Canada, individuals who are admitted to hospital with chronic and/or complex conditions are increasingly being designated as needing an “alternate level of care”, or ALC. This occurs when the patient is occupying an acute care resource at an inappropriate and unnecessary level for their needs, such as a hospital bed while awaiting discharge to a less-resource intensive care setting [3]. Common reasons for ALC designation are that the patient is waiting for home care, long-term care (LTC), or palliative care [4]. This impedance to patient flow further prevents hospital admissions for other patients and has led to ALC being considered as “choke points” in the healthcare system [5] which makes healthcare more costly and less accessible [6].
With approximately 15–20% of hospital beds being occupied by ALC patients [7, 8] and the estimation that one emergency department (ED) bed being occupied by an ALC patient prevents four ED admissions per hour [5], the system costs are exorbitant. The personal health cost of lengthy hospitalization is also high, with older adults being at particular risk of deconditioning, mental health changes, and nosocomial infection [6, 9]. Coordinated hospital-to-home programs that integrate care between hospital and community organizations have been identified as an effective approach to improving outcomes [6] and avoiding inappropriate hospital readmission in older adults [10].
This study aimed to evaluate the implementation of a newly developed, hospital-to-home program for older adults called North York Community Access to Resources Enabling Support (NYCARES). Coordinating health, long-term, and community care sectors for this population with complex needs, NYCARES is an integrated care program per Nolte and Pitchforth’s [11] definition and was launched during the early stages of the COVID-19 pandemic in North York, a metropolitan suburb of Toronto, Ontario. The inception of NYCARES coincided with a time of significant health system transformation including the establishment of Ontario Health Teams (OHTs) by the Province of Ontario. OHTs are intersectoral partnerships of health and social care organizations that are accountable fiscally and clinically for delivering integrated care to their attributed population [12]. This new OHT model requires organizations that traditionally function independently to collaborate and pool resources in ways that improve shared patient outcomes.
While there are currently over 50 OHTs [12], the North York Toronto Health Partners (NYTHP) OHT was among the first to form in December 2019. Each OHT identified their priority populations and could create initiatives to address specific needs; the NYTHP OHT launched with NYCARES as its flagship program. A process evaluation was conducted to understand the feasibility of the program, and to optimize ongoing program refinement and implementation [13]. The purpose of a process evaluation is not to investigate the effectiveness of the program, but to determine its core elements and produce a refined program theory to support ongoing implementation and decision-making [13, 14].
Methods
Study Setting – the NYCARES Program
In November 2020, key stakeholders and service providers convened to conceptualize the NYCARES program during a session called Design Day 1.0 (Figure 1). The resulting program aimed to provide comprehensive, in-home care to ALC patients at higher intensity than typically available in the community (but at lower intensity than in hospital) through hospital-community partnership; NYCARES was envisioned to enable hospitalized older adults to live safely at home with the necessary health and social care services “wrapped around them” [15]. Referrals could also be made for community-dwelling older adults at high risk of hospitalization and being designated ALC.
Figure 1
Timeline of NYCARES iterations and key features.
ALC: alternative level of care; FY: fiscal year; TBD: to be determined
*Nine patients from NYCARES 1.0 continued receiving services during fiscal year 2021/2022; 31 new patients were enrolled into NYCARES 2.0; total caseload for fiscal year 2022/2023 was unreached at the time of this study which took place in fiscal year 2021/2022.
Envisioned as a three-month transitional program (e.g., until patients are accepted into a LTC home), patients would be cared for by an interprofessional team of a primary care provider (i.e., physician, nurse practitioner), occupational and physical therapists, personal support workers (PSWs), and with digital health applications (e.g., remote patient monitoring). Care coordinators would arrange services and equipment from acute, primary, and community care providers to enable patients to be cared for in their homes. A “care navigator” role was created to ease burden on patients and caregivers/family members and provide a single point of contact for NYCARES program needs.
This study spanned two fiscal years of the NYCARES program (Figure 1): December 2020–March 2021 (NYCARES 1.0) and April 2021–March 2022 (NYCARES 2.0). The process evaluation began in October 2021 after the initial pilot was launched. The first author joined the NYCARES team as an embedded researcher and played an active role in this next stage of program feasibility testing [13] by providing feedback from the formative process evaluation to help the team understand and address implementation challenges. This study was approved by the research ethics board of North York General Hospital.
Study Design and Objectives
An embedded process evaluation approach was undertaken to understand what, how, and in which circumstances [14] the intervention/program was working. To enable this, the NYCARES program was conceptualized as theories-in-action and broken down into a set of hypotheses that expressed the “how” and “why” of the program [16], linking program components to outcomes and using a logic model to represent the program theory [17]. Logic models can “make sense of complexity, adding value by achieving a better understanding of the interactions between the intervention, its implementation, and its multiple outcomes among a given population and context” [18].
This study sought to describe and assess NYCARES by its program theory, contexts, implementation, and mechanisms of impact per the Medical Research Council’s framework for process evaluation of complex interventions [13]. We intended to develop logic models to illustrate the evolution of program components and processes. Specific objectives were to:
Elucidate the evolving program theory of change using logic models (theoretical design with program description and causal assumptions)—Phase 1 Theory Development;
Understand the program context and implementation conditions (key considerations, resources, fidelity, adaptations)—Phase 2 Implementation Assessment; and,
Identify mechanisms of impact (patterns of interaction among the program, mediators, and consequences)—Phase 3 Mechanism Identification.
Data Collection
Quantitative and qualitative data collection occurred in Phases 1 and 2. Phase 1 data was collected via extensive researcher field notes taken during NYCARES team meetings and analysis of program documentation (e.g., organizational reports, online material, program proposals and descriptions, program manuals and written materials). Phase 2 data sources included implementation records (e.g., enrollment data, demographic data, program metrics), and one-to-one interviews with NYCARES patients, caregivers, and members of the design and implementation teams (i.e., providers).
Semi-structured interview guides were created for each group. The NYCARES providers were interviewed on the developmental origins of the program, factors influencing its design, content, and process, and their motivations, attitudes, perceptions, and beliefs. Patients and caregivers were interviewed on their program experiences and perceptions. Interviews were conducted by interviewers with no prior relationship with participants and no formal role in the implementation of NYCARES. A master’s student experienced in qualitative research (second author) interviewed providers and a PhD-trained health services researcher (first author) interviewed caregivers and patients. Following participant informed consent, interviews were audio-recorded, transcribed using a third-party service, and de-identified and reviewed for accuracy by the research team.
Data analysis
Quantitative data from implementation records (e.g., enrollment data, demographic data, program metrics) were summarized using descriptive statistics and histograms. Since presenting the totality of implementation data is beyond the purview of this paper, we have focused on textually summarizing the most relevant trends of the program metrics indicated in the process-oriented logic model (i.e., outputs and outcomes) (Figure 2).
Figure 2
NYCARES 2.0 process-oriented logic model.
ALC: alternative level of care; HCCSS: home and community care support services; IPT: interprofessional team; NYTHP: North York Toronto Health Partners; PCHC: patient and caregiver health council; PCP: primary care provider; PSW: personal support worker; RPM: remote patient monitoring.
The analysis of interview data specifically has been published elsewhere as a qualitative case study [23]; therefore, the analytic method is described here in brief. Interview transcripts from both datasets (patient and caregivers; providers) were analyzed using reflexive thematic analysis [19] and managed using NVivo data analysis software. Two authors independently coded (identified units of meaning) three transcripts, discussed the coding schemes, and created a coding dictionary. The first author analyzed the remaining transcripts by reading each transcript multiple times, coding, creating categories of similar codes, and developing sub-themes by combining related categories. Sub-themes from both datasets were compared and synthesized into overall themes and a thematic map that was refined by the research team.
Textual materials were analyzed using qualitative document analysis. The first author reviewed program documentation and researcher field notes multiple times, noting recurring program concepts, service planning obstacles, motivations voiced by team members, key decisions, and communication patterns among groups. The notes were organized into broader topics relevant to logic model development and related aspects were identified as causal assumptions. The output of the document analysis of materials from the first fiscal year of operations (e.g., program proposal, business case) was a system-based logic model of NYCARES 1.0, depicted as resources/inputs, activities/strategies, outputs, outcomes, and impact. Independent of author involvement, the NYCARES providers created a system-based logic model for NYCARES 2.0 (at the start of the second fiscal year), prior to the start of this study. During Phase 1 (theory development), these two logic models were compared to understand the initial program theory and how its conceptualization had changed over time.
Partway through the second fiscal year, the output of analyzing the formative implementation data, interview data, and field notes was a process-oriented logic model which “graphically displays the processes and causal pathways that lead from the intervention to its outcomes. Unlike the system-based logic model, it recognizes a temporal sequence of events and aims to explain how an intervention exerts its effect” [18, p. 41]. This process-oriented logic model represents the NYCARES program theory “in action,” whereas the system-based logic models represent the expected NYCARES program theory. Moreover, implementation was assessed at the end of the second fiscal year against the process-oriented logic model to identify and understand factors such as intervention fidelity and adaptations. In effect, we tested this logic model.
Following Phase 2 (implementation assessment), the entire data set (i.e., all data sources and implementation records to date) was iteratively cross-referenced and synthesized to identify mechanisms of impact and contextual factors. Mechanisms were causal factors that linked contexts or activities to hypothesized outcomes. We used abductive reasoning to identify mechanisms, an approach that oscillates between inductive and deductive thinking [20]. First, we identified elements from the interviews that implied cause-and-effect, then referred to other data sources for corroborating or alternate explanations. These inferences were subsequently compared to the process-oriented logic model to determine whether the mechanism aligned with a hypothesized pathway or formed an unexpected pathway. In cases of unexpected pathways, our analysis sought the underlying reasons for why the anticipated pathway did not occur. These analytic choices are consistent with our pragmatic approach, including that knowledge is socially constructed and experientially modulated, that research creates knowledge for action, and that an admixture of quantitative and qualitative data can be appropriate to answer the research question [20].
A final program evaluation report was prepared for the NYCARES leadership team to support ongoing implementation and decision-making, outlining key program components, resources, fidelity, challenges, and recommendations.
Results
Objective 1. Elucidating program theory
Three logic models were created (NYCARES 1.0 logic model, NYCARES 2.0 process-oriented logic model) or referenced (NYCARES 2.0 logic model), to demonstrate the NYCARES’ evolving program theory.
NYCARES 1.0 system-based logic model
A system-based logic model depicted the conceptual foundations of the program, core components, and intended impact and target groups (Table 1). Given that the data were documents that preceded the launch of services, there was heavy emphasis on resources and activities. The measures selected were largely process measures and outcome measures were a blend of unprioritized shorter- and longer-term outcomes. Program enrollment initially targeted two groups of patients: one via a hospital model and one via a community referral model.
Table 1
Logic model of NYCARES 1.0.
| RESOURCES/INPUTS | ACTIVITIES/STRATEGIES | OUTPUTS | OUTCOMES | IMPACT |
|---|---|---|---|---|
|
|
|
|
|
[i] ALC: alternative level of care; ED: emergency department; IPT: interprofessional team; LTC: long-term care; NYTHP: North York Toronto Health Partners; PCHC: patient and caregiver health council.
NYCARES 2.0 system-based logic model
In contrast, the NYCARES 2.0 system-based logic model (Table 2) described outputs with more specificity and directionality (e.g., decreased # hospitalizations resulting in ALC designation); however, the six outcomes were still quite broad and did not specify how they would be measured. Although the outcomes were developed from survey responses of over 40 NYCARES 1.0 stakeholders, there were no clear linkages from outputs to outcomes. Further, the outcomes are listed without distinguishing their timeframe, such as whether they are short- or long-term, and whether some are expected to facilitate later outcomes. Discrepancies between NYCARES 1.0 and 2.0 system-based logic models also indicated evolving conceptualization of program elements, such as whether the metric “number of unnecessary ED visits avoided” is an output or outcome.
Table 2
Logic model of NYCARES 2.0.
| RESOURCES/INPUTS | ACTIVITIES/STRATEGIES | OUTPUTS | OUTCOMES | IMPACT |
|---|---|---|---|---|
|
|
|
|
|
[i] ALC: alternative level of care; ED: emergency department; NYTHP: North York Toronto Health Partners; PCP: primary care provider (physician, nurse practitioner).
The NYCARES 2.0 logic model also showed greater refinement of the community model’s eligibility criteria (addition of 3 referral streams), which was more specific than the first logic model’s activity of “identify appropriate patients for referral.” Virtual care and digital health were emphasized in the NYCARES 2.0 logic model because there was more time and resources for the procurement, logistics, and training required for set up than was available during NYCARES 1.0. While they were considered as resources in the first logic model, they were emphasized as activities in the second logic model, indicating the intention to increase their prominence. Other changes included eliminating “meeting transportation needs” and adding a nurse practitioner to provide more care hours for patients at a lower cost than through a physician. Reflecting the desire for program expansion, target models were augmented with the addition of a third model for rehabilitating hospital patients at home instead of hospital.
Data which informed the reasoning for some of the NYCARES 2.0 logic model evolutions was extracted from the researcher field notes. For example, the activity of “Provide NYCARES education and marketing via online and newspaper methods to referral sources” was added due to a diminishing volume of program referrals at a time when the enrollment goal had doubled to 40 patients. The team thought that more referral sources needed to be made aware of the program and the addition of the third target model (i.e., patients needing rehabilitation). Correspondingly, the team felt that eligibility criteria and referral processes needed updating to accommodate the three streams of the community care model.
NYCARES 2.0 process-oriented logic model
The process-oriented logic model (Figure 2) was generated using the formative findings from this study; it represented an updated program theory with explicit linkages of components into pathways to illustrate how the change was meant to happen. All outcomes were linked to the overall program goal and metric, which was the number of ALC days saved (calculated as the number of days from start of in-home services to program discharge). Metrics for two outputs, “IPT & PSW services” and “HCCSS services,” were added as critical implementation data. (See Figure 2 for acronyms; HCCSS is the central agency for home and community care in the province.) Some new metrics were collected as baseline data (e.g., virtual care onboarding) while other metrics were compared to existing benchmarks (e.g., patient care costs). Other components from earlier iterations were omitted because they were either not collected (e.g., # ED visits avoided) or were not measurable within the evaluation timeframe (i.e., long-term outcomes).
There were other notable changes to how the process-oriented logic model refined program components and their linkages. Three resources (NYTHP Support, Partners and Sectors, PCHC) were key inputs that affected all parts of the program. The resource, “Patient and Caregiver Health Council (PCHC),” replaced the NYCARES 2.0 system-based logic model’s activity, “Co-design with patients and caregivers.” The PCHC membership is NYTHP’s formalized group of patients and caregivers who have lived experience of hospital- and community-based care. A few PCHC members participated in co-designing the NYCARES program and advised on its ongoing implementation. These PCHC members had extensive prior or current experience with caring for family members who were either in/awaiting transfer to a LTC home from hospital or who were often moving between hospital and community settings.
“Strategic direction and leadership guidance” was the only resource that linked to just one activity. Four activities led to five outputs. In particular, the activity of “provide wraparound basket of services within budget” led to two output categories, which were health and social care categories. A pathway for virtual care and digital health was built to reflect the emphasis on these components that were intended to increase care capacity within the program. Although there were minor changes to the language of the program goal, the focus remained on helping seniors live safely at home.
Objective 2. Understanding implementation – Logic model in action
Program context
While the NYCARES program was operational, key process improvements were needed to better define the target population, improve referral and intake processes, and develop consistent measurement and reporting mechanisms. The program could have addressed pain points by better implementing two related activities in the process-oriented logic model: “diversify enrollment” and “improve referrals.” “Diversity enrollment” was strongly affected by contextual factors, specifically by the resource/input of “Strategic direction and leadership guidance.” This resource linked only to “diversify enrollment,” but it should have been linked as well to “improve referrals” because the NYCARES team was expecting the leadership table to provide operational clarity on their conceptualization of the eligible groups. However, the leadership table was rarely able to provide strategic direction in a timely way in the early days of the program. The NYCARES clinical team continued existing processes while awaiting responses to requests for clarity on eligibility criteria. Consequently, enrollment efforts to clarify target models and update forms felt hamstrung, resulting in provider frustration. The “Strategic direction and leadership guidance” resource/input did not function optimally as an implementation facilitator.
Key considerations and resources
It was thought that the patient caseload for NYCARES 2.0 could be increased by adapting the program design to include a third target model (rehabilitation), tailoring community recruitment to three distinct streams, and aligning the referral and intake forms and processes to these streams. The resulting increase led to the need to maximize existing health human resources, particularly the responsibilities of the care navigator, and to hire a nurse practitioner at more hours instead of staffing a physician who was only available during restricted hours. However, findings from the documentation analysis revealed that program capacity was not truly expanded with the hiring of a nurse practitioner because at the same time a hospital-based care coordinator position dedicated to handling NYCARES referrals was eliminated.
Increased enrollment also did not materialize through the rehabilitation target model as it was never actually operationalized (Figure 3, depicted as greyed out text). Although program metrics showed that the total patient caseload increased from 17 in NYCARES 1.0 to 40 in NYCARES 2.0, it did not result for the hypothesized reasons. Rather, the enrollment rate remained similar, but NYCARES 2.0 occurred over one full fiscal year whereas NYCARES 1.0 spanned half a fiscal year, and some of the 40 patients receiving services during NYCARES 2.0 were enrolled in the first fiscal year. More importantly, the care navigator emerged as a major program facilitator whose contributions included streamlining the referral and intake processes; their contributions are discussed below.
Figure 3
NYCARES 2.0 implementation summary map.
ALC: alternative level of care; HCCSS: home and community care support services; IPT: interprofessional team; PSW: personal support worker; QoL: quality of life; RPM: remote patient monitoring. Thicker outlines and arrows indicate mechanisms of impact (care navigator, basket of services); circular component indicates the novel care navigator mechanism; transverse lines (≠) indicate actions that did not occur; greyed out boxes and text indicate under-implemented components; boxes with broken lines indicate outcomes that should be added.
Program fidelity and adaptations
Virtual care (i.e., assessments and visits) and digital health (i.e., remote patient monitoring and use of tablets to communicate with providers) were activities expected to approximate the hospital’s around-the-clock monitoring and care but at lower costs. However, program metrics (i.e., virtual care onboarding outputs) indicated that these services were under-used (Figure 3, depicted as greyed out boxes) compared to the expectations of the NYCARES team who had anticipated a higher demand from patients and caregivers, believing the two groups would feel more at ease by being able to connect with providers through these methods. Rather, some caregivers declined digital health services because they viewed tablet use as inappropriate due to the patient’s cognitive difficulties or challenges with technology and/or language. Caregivers also seemed to prefer contacting the care navigator directly. Around-the-clock care was only truly experienced by the few patients who qualified for 24-hour PSW service, which was costlier than virtual care and digital health.
Conversely, two components from the interview data analysis appeared to be program outcomes but were not reflected in any logic model: better patient quality of life and reduced caregiver burden. All interview groups expressed that patients would have a better quality of life at home than in hospital due to a familiar environment and proximity to family. Providers and caregivers alike mentioned a general reduction in caregiver burden, but there was a spectrum of caregiver experiences. Some caregivers expressed greater distress at the start of NYCARES when they felt like they were responsible for all the decisions about care at home. Others felt extremely stressed when the program was nearing the three-month endpoint and were uncertain of the post-NYCARES plan. Between those service bookends, caregivers tended to feel relieved for the reprieve the program provided and very grateful for the comprehensive services coordinated through one team. These two outcomes have been combined into “improved experiences” and depicted as a short-term outcome (Figure 3, depicted as red boxes with broken lines).
Objective 3. Identifying mechanisms of impact
Mechanisms of impact help explain how NYCARES ultimately achieved its goal of providing care for this population at home instead of in hospital, as measured by the number of ALC days saved, which was a key program metric. One mechanism of impact (Figure 3, depicted with bolded outlines) was, as expected, delivery of truly integrated care (i.e., provision of a comprehensive basket of coordinated services). Unfortunately, there was an operational inefficiency that negatively affected the referral pathway in the process-oriented logic model. This operational inefficiency was evidenced by the enrollment data, observed in the documentation analysis, and corroborated by interview data. Rather than directly submitting a referral form for immediate processing, potential referrers found that the program criteria were unclear so often inquired with the care navigator about patient eligibility before completing a referral form. This additional step slowed the enrollment process and increased the workload of the care navigator.
As noted by the clinical team early on, the referral form criteria needed updating to match NYCARES 2.0’s broadened target models and referral streams. However, the leadership table was slow to provide population clarity to the NYCARES implementation team, who were therefore hesitant to operationalize eligibility criteria for the clinical team (led by the care navigator). The subsequent confusion about program eligibility criteria impeded any education and marketing efforts to increase referrals. These blockages are depicted as transverse lines in Figure 3.
Alternate mechanism
An unanticipated mechanism of impact appeared in the “improve referrals” pathway via the care navigator in response to slow decision making. Unable to obtain timely guidance from the other groups, the care navigator led the clinical team to update the referral form with eligibility criteria that matched the target models. Further, the care navigator assumed additional responsibilities given the elimination of the hospital care coordinator’s position, such as soliciting referrals from hospital-based social workers. The care navigator also attempted to improve patient flow through the program by streamlining intake processes; once a referral form was received, they expedited the onboarding meeting and initiated in-home services as soon as possible. Unfortunately, program enrollment and patient flow ground to a halt whenever the care navigator was absent since no one else on the team was able to conduct intakes or discharges. All of this illustrates how the care navigator was a significant mechanism of impact in this program.
Discussion
As a new integrated care program, NYCARES was developed to alleviate pressures on hospital bed shortages during the COVID-19 pandemic. During implementation, the program evolved from its initial design, as depicted through the three logic models presented in this paper. Challenges to program scalability included uncertainty about the target population and the heavy care navigator workload.
Based on a review of the literature, studies that report developing a logic model typically do not use them to illustrate changes in program theory over the course of a program. Further, many do not test the logic model as we have done here done by assessing implementation factors (i.e., putting the process-oriented logic model into action) and evaluating the fidelity of the initial design and its real-world application (visually summarized as Figure 3). Testing the logic model is also a practical opportunity for further refinement by the team. For example, upon finding that “improved experiences” was not explicitly stated in any of the logic models, all NYCARES groups can now discuss whether the program should include a pathway to intentionally improve patient quality of life and reduce caregiver burden, including linking it to activities, outputs, and impact.
Some deviations from original program design arose due to operational and conceptual needs that were not addressed in timely ways by those with decision-making authority. The likelihood of program success is reduced when there are misalignments between the leadership table who decides program direction and team members who operationalize those directions, especially when alignment partially depends on the degree of system integration [21]. NYCARES was the flagship program for the NYTHP OHT model where health and social care organizations share clinical and fiscal responsibility for patients in the program.
This new configuration of organizational collaboration and some unforeseen complexities seemed to contribute to the misalignments. New organizational relationships and governance are critical implementation influences, especially when organizations that were previously in competition (e.g., for funding dollars) are now collaborating using shared resources. Spending time developing shared values and leveraging pre-existing connections between the groups could mitigate these challenges [22]. Doing so would build on the goodwill and desire to do the right thing for patients and families among NYCARES organizations, which drove the initiation of the program in the first place [23].
As evidenced in a major system change of stroke services in the United Kingdom (UK) in 2010 [24], better leadership-stakeholder alignment can be achieved through top-down (system) and/or bottom-up (distributed) approaches. Greater clarity on stakeholder relationships and accountability for top-down leadership would likely have improved decision-making for NYCARES teams in the early stages of OHT formation. However, this did not happen for NYTHP in its nascency where cross-sectoral leaders in the OHT may have been figuring out how to collaborate in the new context. For example, the program co-leads were from the hospital and the community care sectors and may not have wanted to domineer the other, thus defaulting to a bottom-up approach by the NYCARES team. Even so, there lacked a strong sense of authority with which to make decisions or the confidence that this was the best leadership approach.
Effective leadership is essential during any health system change and particularly for pilot projects [22], regardless of whether a top-down or bottom-up approach is taken. Given that this new program and leadership structure was quickly implemented during the COVID-19 pandemic, more time would have been needed to smooth out wrinkles in leadership. For NYCARES, frequent small-scale adaptations were made in response to operational inefficiencies and uncertainties, indicating an overall program resiliency through a bottom-up approach and can-do ethos of providers who were personally aligned and committed to the program vision.
Mechanisms of impact were difficult to identify in the NYCARES program, which is coherent with how other health service modernization efforts have found mechanisms of change difficult to pinpoint and understand, although the NYCARES basket of services is resonant with literature identifying the integration of services across providers as a mechanism of impact [25]. One alternate mechanism could be the role of clearly defined leadership decision-making because challenges in this area appeared to hamper patient enrollment processes. However, the care navigator role was a better fit for the current data as a mechanism because it directly facilitated the outputs whereas it was less obvious how decision-making by formative leadership does so.
Identification of the care navigator role as a mechanism of impact is an example of how developing a deep understanding of the program theory can reveal what the necessary core components are to drive intervention fidelity [26]. Program roles that are key to the intervention should also be indicated individually as a resource/input in logic modelling. We found that the care navigator shouldered a taxing workload due to factors such as performing multiple essential functions that were not duplicated in the program and being the “single point of contact” for patients and caregivers. The care navigator experience mirrors that of care navigators in other programs who have extensive responsibilities for communication and accessibility [27].
However, the NYCARES care navigator was also the clinical team lead, which meant that they were responsible for handling multi-directional issues among stakeholder groups. This additional responsibility may likely have been a source of stress beyond the multifaceted responsibilities performed by care navigators in other programs, which typically centre on administrative functions, service coordination, and patient supports [28]. In some care models, the care navigator is not a healthcare professional [28] in contrast to the NYCARES care navigator, who was a registered social worker simultaneously providing care within their scope of practice. The NYCARES care navigator functioned in a boundary-spanning role [25] to bridge hospital- and community-based care, which is also a sub-mechanism identified by evaluators of a large-scale healthcare innovation in the UK.
Given this multi-function conceptualization of the role, the NYCARES care navigator needed to be better supported. For example, communication channels must be strengthened and formalized among the clinical, implementation, and leadership groups. Timelier decision making and strategic guidance from the authorized group would reduce the pressures on the care navigator. Without sufficient support for the care navigator, their ability to be the “single point of contact” for NYCARES families will likely degrade.
Despite current barriers, there was still evidence of relational cohesiveness within the clinical and implementation teams as providers were committed to the OHT vision of integrated care and were willing to problem-solve encountered challenges. This suggests that group cohesion and value alignment were mediating factors. Future research is needed to understand how the mediating factors affect program outcomes, both positively and negatively. More work is also needed to develop guidance for teams operating within changing health system contexts, specifically addressing implementation decision making for new integrated care programs. We submit our study as evidence for the value of using process-oriented logic modelling with its subsequent testing and recommend for program data and measurement to be aligned with program theory.
Key learnings
Process evaluation can identify contexts and implementation conditions for integrated care teams to be aware of and address early in a program’s lifecycle. Other programs can benefit from the advantages of developing logic models for integrated care programs:
To clarify program objectives and essential components with a system-based logic model;
To illustrate the evolution of the program from design through implementation;
To test real-world implementation against a process-oriented logic model; and,
To refine the theory of change that will underpin future evaluation on program effectiveness.
Implementation of an integrated care program was affected by the context of local healthcare system transformation. Our study noted the importance for programs in similar contexts:
To clearly establish decision-making authorities and processes among leadership, implementation, and clinical teams;
To conceptualize program target groups and their operationalization as they evolve; and,
To identify mechanisms of impact and address alternate pathways supporting program aims and processes.
Limitations
This was a complex study with several moving parts and sources of data. Given the interpretive approach to analyzing the multiple data types, other interpretations may validly produce other mechanisms. However, the mechanisms identified in our analysis are internally coherent with the overall evaluation findings and explain their impact on other components in the logic model, specifically outputs and outcomes. The contexts and implementation conditions we have described may not apply to other programs, but we have discussed processes that may support intersectoral staff to collaborate more effectively in the delivery of integrated care [22].
Although this process evaluation does not report numerical-based conclusions on program outcomes, we have chosen to value the use of qualitative data to inform conclusions without overly emphasizing quantitative measures of effectiveness research [29]. This approach allowed us to uncover the “improved experiences” outcome by integrating the interview data analysis, which was not reflected in the earlier system-based logic models. However, the logic models as presented will not enable other OHTs to replicate NYCARES because they were not intended to comprehensively specify all program elements, but rather, to illustrate the key aspects underpinning the program theory.
Conclusion
This process evaluation of a newly developed integrated care program described the development and evolution of its program theory, contexts, implementation, and mechanisms of impact. We have shown how an operational integrated care program can face contextual and operational threats to its sustainability even as it evolves. Study findings highlighted how a process evaluation can identify unexpected mechanisms of impact and additional outcomes for consideration. The refined program theory produced from a process evaluation can then be the basis of a more accurate evaluation of program effectiveness.
Acknowledgements
We wish to thank the NYCARES Team for their support and enabling of program data collection. We are grateful to the caregivers, patient, and providers who shared their valuable perspectives and experiences.
Reviewers
Dr AnnMarie Crozier, Medical Director, SLHD Hospital in the Home and Senior Lecturer, University of Sydney, Australia.
Jia Lu Lilian Lin, Institute of Health Policy, Management and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada.
Competing Interests
At the time of writing, Adora Chui was the NYTHP Evaluation Scientist whose salary was paid by the NYTHP Ontario Health Team. However, none of the NYCARES Team or NYTHP Core or Alliance Members were involved in the writing, review, or approval of this manuscript.
Author Contributions
KND conceived of this research project and analyzed the data. KS interviewed participants. AC refined the research project conceptualization, interviewed participants, collected and analyzed the data, and wrote the manuscript. All authors approved the final manuscript.