Introduction
Organisational level, occupational health interventions (hereon referred to as organisational interventions) are “planned, behavioural, and theory-based actions that aim to improve employees’ health and wellbeing by changing the way work is designed, organised, and managed” (Nielsen, 2013, p. 1030). These interventions are the recommended approach for improving employees’ health and wellbeing (EU-OSHA, 2016; ILO, 2001). However, some reviews have concluded that evidence showing the effectiveness of organisational interventions is inconsistent (Fox et al., 2022; Montano et al., 2014). The lack of consistency in the evidence of organisational interventions may be due to the heterogeneity in their designs (e.g., various approaches to develop action plans), implementation strategies (e.g., using different drivers of change including multi-level managers and employees), contexts (e.g., changes in the organisation during the intervention), and outcomes (e.g., using different outcome measures) (Nielsen & Miraglia, 2017; Roodbari et al., 2021). Further, some reviews have concluded that few organisational intervention studies have examined why and how organisational interventions have succeeded or failed (Fox et al., 2022; Murta et al., 2007). As such, to inform future organisational interventions, we need evaluation models that show why and how specific organisational interventions in certain contexts produce specific outcomes. Using such evaluation models improves the understanding of what works for whom under which circumstances, and consequently reduces the inconsistency of empirical evidence of organisational interventions (Nielsen & Miraglia, 2017; Roodbari et al., 2021).
A variety of evaluation models for organisational interventions have evolved. The evaluation models proposed by Nielsen and Randall (2013), Nielsen and Abildgaard (2013), and Nielsen, De Angelis, et al. (2022) include evaluation categories of (1) intervention context, (2) intervention design and implementation, (3) participants’ mental models (of the intervention and their work situation), and (4) outcome. On the other hand, the evaluation model proposed by Fridrich et al. (2015) includes evaluation categories of (1) intervention context, (2) implementation process, (3) change process (i.e., the individual and collective dynamics triggered by the implementation process), and (4) outcome. As can be seen, the evaluation models used different terminologies to evaluate organisational interventions. More specifically, while Nielsen and Abildgaard (2013) named two process evaluation categories as (1) intervention design and implementation and (2) participants’ mental models, Fridrich et al. (2015) named the same evaluation categories as (1) implementation process and (2) change process, respectively. To overcome the possible confusion caused by diverse terminologies of these previous evaluation models, we suggest using realist evaluation concepts of mechanisms, contexts, and outcomes (Pawson & Tilley, 1997). Our model conceptualises mechanisms, contexts, and outcomes in a way that encompass the evaluation categories of the previous models. Using realist evaluation concepts of mechanisms, contexts, and outcomes is helpful as it can facilitate the transferability of knowledge in future organisational interventions (Nielsen & Miraglia, 2017).
In addition, the previous models provided examples for each evaluation category and recommended examining the relationships between such evaluation categories in the evaluation of organisational interventions. For instance, Fridrich et al. (2015) emphasised considering the reciprocal relationship between intervention process and context. Similarly, von Thiele Schwarz et al. (2016) stressed the consideration of the relationship between intervention process and context and suggested contextualising the intervention process. However, there is still a gap in knowledge about how evaluation categories interact with each other (Nielsen & Miraglia, 2017). This current article highlights crucial intervention mechanisms in different phases of organisational interventions and develops examples of specific Context-Mechanism-Outcome (CMO) configurations based on these mechanisms. These specific CMO configurations show how the interactions between specific mechanisms and specific contexts can produce certain outcomes (Pawson & Tilley, 1997). Exploring the interactions between CMO elements and developing CMO configurations is crucial as it improves the understanding of what works for whom in which circumstances.
In essence, this article contributes to the evaluation of organisational intervention as it (1) conceptualises evaluation categories as contexts, mechanisms, and outcomes as required by realist evaluation, (2) explains how to follow realist evaluation and develop CMO configurations, and (3) provides examples of specific CMO configurations based on critical mechanisms of organisational interventions.
Realist Evaluation
This article uses realist evaluation as its theoretical approach because realist evaluation has been recommended as a suitable theoretical approach to evaluate complex organisational interventions (Nielsen & Miraglia, 2017). Nielsen and Miraglia (2017) argued that realist evaluation offers an opportunity to develop an integrated context, process, and outcome evaluation framework that may advance our theoretical understanding of which elements of organisational interventions may produce positive outcomes and in which conditions. Realist evaluation seeks to answer the question of “what works for whom in which circumstances?” through studying what the mechanisms of an intervention are (what makes an intervention work?), the contexts in which these mechanisms are triggered (what are the conditions in which the mechanisms are operative/effective?), and the outcomes these mechanisms produce (what are the observed patterns of outcomes?) in CMO configurations where contexts + mechanisms = outcomes. CMO configurations not only allow reflecting on context, process, and outcome, but also help to overcome inconsistencies of evaluation categories of the previous phased models.
To develop an evaluation model based on realist evaluation, this article builds on the five-phase model proposed by Nielsen and Abildgaard (2013). Among the above-mentioned evaluation models, the five-phase model is the only model that was developed based on realist evaluation (as was explicitly mentioned by the authors of the model). To incorporate realist evaluation in their model, Nielsen and Abildgaard (2013) conceptualised mechanisms as organisational members’ mental models and behaviours and viewed each phase as the outcome of the previous phase. This realist evaluation approach is critical as it helps us to understand how CMO elements in one phase influence what happens in the next. For instance, in the intervention study by Tafvelin et al. (2019), two mechanisms of employees’ participation and line managers’ support had reciprocal interactions with each other over the intervention phases to ultimately produce job satisfaction.
The Five-Phase Model and Its Limitations
We improve the five-phase model into a new model by addressing its limitations. The first limitation is that the five-phase model does not include some crucial intervention components that should be evaluated in each intervention phase. In particular, the five-phase model does not include the recruitment process of organisational units (in terms of how organisational units were recruited and how such processes affected the intervention), the implementation processes (in terms of what was planned, what actually took place, and why there were differences (if any) between them?), and maintenance of the intervention (in terms of whether and how the intervention was institutionalised and how participants designed, organised, and managed their jobs differently that lasted in the organisation). In this article, we integrate the contents of the RE-AIM framework (with dimensions of Reach, Effectiveness, Adoption, Implementation, and Maintenance) (Glasgow et al., 1999) into the five-phase model. First, in their review of process measures in interventions, Nielsen et al. (2022) identified two most prominent models in public health that can guide process evaluation: RE-AIM and Linnan and Steckler (2002) Process Evaluation for Public Health Interventions. We focused on RE-AIM as it includes the missed intervention components in the five-phase model (e.g., maintenance of the intervention). Second, the focus of the five-phase model is primarily on people’s perceptions of the intervention process and context, whereas RE-AIM focuses on objective measures of the uptake of the intervention. When combined with the five-phase model, this can promote the understanding of the intervention components that influence the intervention’s outcomes as well as the understanding of how perceptions drive uptake (Nielsen et al., 2022). Our integrated model, therefore, has more intervention components than the five-phase model which allows us to evaluate how intervention components affect intervention outcomes thus improves the understanding of “what works for whom in which circumstances” (Nielsen et al., 2010b; Nielsen & Miraglia, 2017). Since RE-AIM was proposed to evaluate community-based, health-promoting interventions, this article discusses RE-AIM dimensions in the context of organisational interventions and then integrates the dimensions into the five-phase model (Moullin et al., 2020).
The second limitation is that the five-phase model, although inspired by the CMO configuration idea of realist evaluation, does not follow the full realist evaluation cycle, that is, it does not explain when and how to conduct the four steps of the realist evaluation cycle. The realist evaluation cycle includes developing initial CMO configurations, collecting empirical data, analysing and synthesising empirical data, and testing initial CMO configurations (Pawson & Tilley, 1997, 2004). Following the four steps of the realist evaluation cycle helps accumulate valid, consistent empirical evidence (Pawson & Tilley, 2004) that can inform future organisational interventions. This article, therefore, discusses crucial intervention components from a realist evaluation perspective (i.e., CMO perspective) and by following the four steps of the realist evaluation cycle, provides guidance on when, why, and how to develop and test CMO configurations for the intervention components.
In addition to these two specific limitations of the five-phase model, the concepts of contexts, mechanisms, and outcomes in realist evaluation are vaguely defined and inconsistently used (Lacouture et al., 2015; Lemire et al., 2020; Nielsen et al., 2022). As such, there is uncertainty about how CMO elements can be conceptualised and operationalised to capture the complexities of organisational interventions (Roodbari et al., 2021).
Developing the IREMOI
This article addresses the above limitations of the five-phase model. First, we explain the concepts of mechanism, context, and outcome. Then, we integrate the contents of the RE-AIM framework into the five-phase model to include crucial intervention components. Next, we explain how to follow a realist evaluation cycle, provide guidance on when, why, and how to develop and test CMO configurations for intervention components, and develop examples of CMO configurations for intervention components. In doing so, we develop an Integrated Realist Evaluation Model for Organisational Interventions (IREMOI).
Following this model helps to consider crucial intervention components upfront, develop initial CMO configurations, design and implement the intervention based on the initial CMO configurations, collect empirical data, analyse and synthesise empirical data, and test the initial CMO configurations. This recycling process of CMO configurations accumulates knowledge about “what works for whom in which circumstances.” Given these, this article demonstrates how working with CMO configurations systematically in intervention may improve the understanding of “what works for whom in which circumstances” and thereby the likelihood of intervention success.
Mechanism, Context, and Outcome
Mechanisms are defined as interpretations, considerations, decisions, and ultimately behaviours of intervention participants that produce outcomes (Pawson, 2013). In this article, as suggested by Dalkin et al. (2015), we consider (1) the resources that interventions provide to participants (e.g., participation in action planning, communication) and (2) instant, consequent changes in participants’ individual and collective reasoning and reactions (e.g., social learning) as two constructs of a mechanism. Also, as suggested by Nielsen and Miraglia (2017), we categorise mechanisms into process and content mechanisms. Process mechanisms are the processes of designing and implementing the interventions, and content mechanisms are the nature of changes focused on in the interventions including the content of action plans.
Contexts are defined as the conditions in which interventions are introduced that are relevant to the operation of mechanisms (Pawson & Tilley, 2004). In this article, as suggested by Nielsen and Randall (2013), we categorise contexts into omnibus and discrete contexts. Omnibus contexts refer to the general intervention setting (e.g., pre-intervention working conditions), and discrete contexts refer to the concurrent changes taking place during the intervention (e.g., organisational restructuring).
Outcomes are defined as the intended and unintended consequences of interventions, resulting from the activation of different mechanisms in different contexts (Pawson & Tilley, 2004). In this article, as suggested by Fridrich et al. (2015), we categorise outcomes into proximal, intermediate, and distal outcomes. Proximal outcomes are changes in psychosocial risk management (e.g., participants awareness of and capacity to manage psychosocial working conditions), intermediate outcomes are changes in psychosocial working conditions (e.g., job autonomy), and distal outcomes are changes in employees’ health and wellbeing (e.g., job satisfaction) and organisational outcomes (e.g., financial performance).
Integrating the Contents of RE-AIM into the Five-Phase Model
This section discusses the RE-AIM dimensions of Adoption, Reach, Implementation, Effectiveness, and Maintenance in the context of organisational interventions and describes how and why these dimensions are integrated into the five-phase model. The RE-AIM dimensions of Reach, Effectiveness, and Maintenance operate at the individual level, and Adoption, Implementation, and Maintenance operate at the organisational level (Glasgow et al., 2019).
Adoption evaluation, at the organisational level, assesses the recruitment process of organisational units and, at the individual level, assesses the recruitment process of intervention providers (managers) (Gaglio et al., 2013). Reach evaluation, at the individual level, evaluates the recruitment process of intervention participants (employees) (Gaglio et al., 2013). In organisational interventions, however, the recruitment process is conducted at the organisational level meaning the organisational units, either worksites, organisational departments, or working teams are identified and recruited (Gupta et al., 2018; Ulrica von Thiele Schwarz et al., 2017). The recruitment process of organisational units can be evaluated as a mechanism as the recruitment process influences the motives of organisational members for engaging or disengaging in the intervention activities (Nielsen et al., 2022). As such, the recruitment process of organisational units (not covered in the five-phase model) can be evaluated as a mechanism in the preparation phase of the intervention.
Implementation evaluation, at the organisational level, assesses intervention fidelity, adaptations made to the intervention, and consistency of intervention delivery across different organisational units and employees (Gaglio et al., 2013). In organisational intervention evaluation, the implementation process is evaluated in terms of intervention fidelity (i.e., the extent to which the intervention delivered is consistent with its protocol), dose delivered (i.e., the extent to which the number or amount of intervention activities was delivered to intervention participants), and dose received (i.e., the extent to which intervention participants received and participated in the intervention activities) (Nielsen & Randall, 2013). Evaluating fidelity and dose is important as it helps to identify what was planned, what actually took place, and why there were differences (if any) between what was planned and what actually took place. As such, achieving a higher level of fidelity and dose (partially covered in the five-phase model) can be evaluated as mechanisms in the implementation phase of the intervention. Achieving a higher level of fidelity can be evaluated as a mechanism as it can influence participants’ mental models of the intervention. For instance, Augustsson et al. (2015) reported that, when the intervention was introduced and supported in the same way, achieving a higher level of fidelity influenced participants’ belief that the intervention would have positive effects on their health.
Effectiveness evaluation, at the individual level, assesses both intended and unintended intervention outcomes (Gaglio et al., 2013). Similarly, the five-phase model evaluates both intended and unintended outcomes in the evaluation phase of the intervention.
Finally, Maintenance evaluation, at the organisational level, assesses the extent to which the intervention is integrated into the organisation’s day-to-day operation and maintained over time (Gaglio et al., 2013). In organisational intervention evaluation, the maintenance of the intervention is determined by, first, the extent to which the intervention aims and objectives were aligned with organisational aims and values (i.e., strategic alignment) and, second, the extent to which the intervention activities were integrated into organisational policies and practices (i.e., operational alignment) (U. von Thiele Schwarz & Hasson, 2013). Evaluating maintenance is important as it helps to determine if changes in the organisation were maintained over time, resulting in long-term improvements in the psychosocial working conditions and employees’ health and wellbeing (K. Nielsen & Noblet, 2018). Therefore, (1) the process of aligning the intervention with organisational aims and values (not covered in the five-phase model) should be evaluated as a mechanism in the preparation phase and (2) the process of integrating the intervention with organisational policies and practices (not covered in the five-phase model) should be evaluated as a mechanism in the action planning phase. Maintenance evaluation, at the individual level, assesses the long-term effects of the intervention six months or more after the last intervention contact (Gaglio et al., 2013).
Table 1 shows where the contents of each RE-AIM dimension can be integrated into the five-phase model to form the IREMOI. In Table 1, the first column shows evaluation actions in each phase of the five-phase model; then, the second column shows the evaluation actions in each dimension of RE-AIM; and last, the third column shows the results of combining RE-AIM and five-phase models to form the IREMOI.
Table 1
The contents of the Five-Phase Model, the RE-AIM Framework, and the IREMOI.
| EVALUATION ACTIONS IN THEFIVE-PHASE MODEL | EVALUATION ACTIONS IN THE RE-AIM FRAMEWORK | EVALUATION ACTIONS IN THE IREMOI |
|---|---|---|
| Evaluation Evaluating:
| Effectiveness
| Evaluation Planning Conducting process and outcome evaluations through CMO configurations by:
|
| Preparation Evaluating:
| Adoption and Reach Evaluating the recruitment process of organisational units Maintenance
| Preparation Developing CMO configurations about the mechanisms of:
|
| Screening Evaluating:
| - | Screening Developing CMO configurations about the mechanisms of:
|
| Action planning Evaluating:
| Maintenance
| Action planning Developing CMO configurations about the mechanisms of:
|
| Implementation Evaluating:
| Implementation Evaluating:
| Implementation Developing CMO configurations about the mechanisms of:
|
The IREMOI
In the IREMOI, an organisational intervention is viewed as a collection of CMO configurations which theorise the ongoing interactions between mechanisms and contexts (in the form of intervention resources + contextual factors → participants’ individual and collective reasoning and reactions) produce proximal, intermediate, and distal outcomes, gradually (Figure 1).
Figure 1
The IREMOI.
In the following, we elucidate how the IREMOI can be used. Since the “evaluation phase” runs throughout the entire intervention, it is important to determine when, why, and how to evaluate intervention components upfront. Hence, we call the first phase “evaluation planning.” Below, we explain the evaluation planning by describing the four steps of the realist evaluation cycle and highlighting when each step should be taken.
Intervention Phase 1: Evaluation
Step 1: Developing initial CMO configurations. This step takes place before initiating the intervention. Data are collected from the organisational interventions literature (cf. Roodbari et al., 2021), national policies, organisation databases, researchers, occupational health practitioners, policymakers, and organisation’s managers and employees (Pawson & Tilley, 2004). The collected data are, then, analysed based on themes of contexts, mechanisms, and outcomes and are synthesised by following “retroduction” that requires identifying mechanisms, contexts associated with such mechanisms, and possible outcomes based on their causal links to develop initial CMO configurations (Greenhalgh et al., 2017). Developing initial CMO configurations upfront helps to ensure all crucial intervention components are considered upfront, which helps to design the intervention and ensure necessary empirical data are collected during the implementation and at the follow-up.
Step 2: Collecting empirical data. This step takes place from the baseline to the last follow-up. Different methods, including before-and-after intervention measures (i.e., questionnaires), interviews, focus groups, observations, and process tracking can be used to collect empirical data (Pawson & Tilley, 2004; Teddlie & Tashakkori, 2009).
Step 3: Analysing and synthesising empirical data. This step takes place after the last follow-up when all empirical data are collected. The main purpose of data analysis and synthesis is to identify produced intervention outcomes, identify patterns of outcomes, and develop empirically grounded CMO configurations based on these patterns of outcomes (Pawson & Tilley, 2004). Marchal et al. (2012) suggested that qualitative data should be analysed by thematic content analysis using the themes of contexts, mechanisms, and observed outcomes, and quantitative data should be analysed to assess the effectiveness of the intervention and to validate or invalidate the empirical CMO configurations (cf. von Thiele Schwarz et al., 2017).
Step 4: Testing initial CMO configurations. This step takes place after empirical CMO configurations are developed. In this step, the initial CMO configurations are tested against the empirical CMO configurations to confirm, refute, or modify the initial CMO configurations. These empirically tested CMO configurations can be tested again in the next cycle in the same organisation until the observed patterns of outcomes are fully explained or can be used as initial CMO configurations for other interventions in other organisations. The repetition of this realist evaluation cycle results in more valid CMO configurations which are better tested and increasingly refined (Pawson & Tilley, 2004).
In the following, we show how steps two and three of the realist evaluation cycle (i.e., collecting, analysing, and synthesising empirical data) are taken. The IREMOI suggests developing CMO configurations for the mechanisms highlighted in the following phases. Table 2 shows how to develop CMO configurations (i.e., which questions to be asked to develop CMO configurations) and provides examples of CMO configurations.
Table 2
How to develop CMO configurations for the intervention components and examples of CMO configurations.
| QUESTIONS TO BE ASKED TO DEVELOP CMO CONFIGURATIONS | EXAMPLES OF CMO CONFIGURATIONS |
|---|---|
| PREPARATION | |
Recruitment process of organisational units
| If organisational units have reasonably good working conditions, organisational actors have a moderate to a good level of health and wellbeing, and their change valence and collective efficacy are high (contextual factors); then a recruitment process of organisational units in which all intervention actors are informed about the goals and processes of the intervention and are invited to voluntarily participate in the intervention can trigger employees’ and managers’ positive appraisal of the intervention (mechanisms), improve their readiness for changing problematic working conditions (proximal outcomes), and improve their perceived organisational support (intermediate outcomes). |
Multi-level management onboarding process
| If managers have necessary individual resources (e.g., knowledge, skills, a good level of health and wellbeing) and there are organisational resources (e.g., budget, time) (contextual factors); then multi-level management onboarding and their support of the intervention can trigger employees’ commitment to the intervention (mechanisms) improve employees’ collaboration in changing problematic working conditions (proximal outcomes), and improve their perceived managerial support (intermediate outcomes). |
Aligning intervention aims and objectives with organisation vision and values
| If there are necessary resources in the organisation for conducting the intervention and organisational and employees’ readiness for change are high (contextual factors); then aligning aims and objectives of the intervention with the vision and values of the organisation can trigger diffusion of expected positive intervention outcomes among employees and managers (mechanisms), improve their reflection on working conditions (proximal outcomes), and minimise their role conflict (intermediate outcomes). |
Establishment of steering groups, the assignment of a project champion, and their support of the intervention
| If individuals who are interested in becoming a member of steering groups or the project champion have the necessary autonomy and resources, including skills, influence, and credibility (contextual factors); then transparent and participatory establishment of steering groups and assignment of the champion can trigger employees’ and managers’ shared meaning of ownership of the intervention (mechanisms), improve their collaboration in identifying and managing problematic working conditions (proximal outcomes), and improve their perceived autonomy (intermediate outcomes). |
Communication strategy
| If there is a climate of openness, trust, and respect in the organisation and there are enough resources in terms of time, energy, and infrastructure (contextual factors); then a communication strategy which directs effective communication across the organisation about the intervention aims and processes can trigger employees’ and managers’ interpersonal influence (mechanisms), improve their awareness of their psychosocial working conditions (proximal outcomes), and improve organisational culture (intermediate outcomes). |
| SCREENING | |
Tailoring risk assessment methods
| If there are necessary organisational resources for conducting a risk assessment and organisational and employees’ readiness for change are high (contextual factors);then using a tailored risk assessment method that measures local psychosocial working conditions can trigger employees’ and managers’ acceptance of intervention activities (mechanisms), improve their sense-making of their psychosocial working conditions (proximal outcomes), and improve targeted psychosocial working conditions (intermediate outcomes). |
Reporting the results of the risk assessment
| If there are necessary organisational resources (e.g., infrastructure for meetings), the existing working conditions allow managers and employees to attend meetings, and there is a culture of trust and openness in the organisation (contextual factors); then reporting the results of the risk assessment to both employees and managers through regular meetings where the results can be discussed in the meetings can trigger employees’ and managers’ social learning (a mechanism), broaden their horizon regarding psychosocial working conditions (proximal outcomes), and improve their perceived social support (intermediate outcomes). |
| ACTION PLANNING | |
Process of participatory action planning
| If existing job design and employees’ and managers’ health and wellbeing are reasonably good and there are necessary resources in the organisation (e.g., time, infrastructure) (contextual factors); then participatory action planning can trigger employees’ and mangers’ shared meaning of ownership of the intervention and their interpersonal influence (mechanisms), empower them to manage their psychosocial issues (proximal outcomes), and improve employees’ autonomy (intermediate outcomes). |
Contents of action plans
| If employees and managers have necessary individual resources (e.g., readiness for change, knowledge, skills), there are organisational resources (e.g., time, infrastructure), and employees and managers have a shared understanding of psychosocial working conditions (contextual factors); then jointly determining the contents of action plans by targeting adverse psychosocial working conditions to change can trigger their shared meaning of ownership of the intervention and social learning (mechanisms), improve their sense-making of their psychosocial working conditions (proximal outcomes), and improve targeted psychosocial working conditions (intermediate outcomes). |
Integrating intervention activities into organisational policies and practices
| If jobs are well-designed, and change valence and collective efficacy are at high levels (contextual factors); then integrating intervention activities into organisational policies and practices triggers diffusion of expected positive intervention outcomes among employees and managers (mechanisms), improve their reflection on their psychosocial working conditions (proximal outcomes), and minimise their role conflict (intermediate outcomes). |
| IMPLEMENTATION | |
Process of implementing action plans
| If employees and managers have necessary individual resources (e.g., knowledge, skills) and there are organisational resources (e.g., time, budget, infrastructure) (contextual factors); then a participatory process of implementing action plans can trigger employees’ and managers’ shared meaning of ownership of the intervention (mechanisms), improve their collective efficacy to change problematic working conditions (proximal outcomes), and improve employees’ autonomy (intermediate outcomes). |
Achieving a higher level of intervention fidelity
| If the theory behind the intervention has a strong theoretical basis and addresses the existing psychosocial working problems and managers and employees have positive appraisals of the intervention and participate in the intervention activities (contextual factors); then work done to achieve a higher level of intervention fidelity to initial CMO configurations can trigger employees’ and managers’ excitement to keep the intervention and feeling of moving forward (mechanisms), improve their awareness of their psychosocial working conditions (proximal outcomes), and improve their psychosocial working conditions (intermediate outcomes). |
Dose delivered and dose received
| If there are necessary organisational resources (e.g., infrastructure, time, budget) and employees’ and managers’ individual resources (e.g., motivation, skills), and a supportive culture facilitate the implementation process (contextual factors); then high levels of dose delivered and dose received can trigger employees’ and managers’ shared meaning of ownership of the intervention, interpersonal influence, and social learning (mechanisms), empower them to manage their psychosocial working conditions (proximal outcomes), and promote their psychosocial working conditions (intermediate outcomes). |
Intervention Phase 2: Preparation
Recruiting organisational units. RE-AIM requires evaluating the recruitment process of organisational units. This evaluation is essential because the recruitment process of organisational units can influence their readiness for change which in turn affects the implementation of intervention activities. For instance, if the intervention is forced upon organisational units in recruiting organisational units, they may not be ready for change and, consequently, may not complete intervention activities (Framke & Sørensen, 2015).
Organisational units’ readiness for change. The five-phase model requires evaluating organisational units’ readiness for change. This evaluation is important because organisational units’ readiness for change can influence the implementation and outcomes of the intervention. For instance, if employees do not see the intervention’s benefits, they resist the intervention, and consequently, the intervention may not produce positive outcomes (Albertsen et al., 2014). The literature shows that various omnibus contextual factors influence organisational units’ readiness for change, including pre-intervention levels of employees’ health and wellbeing (von Thiele Schwarz et al., 2017), pre-intervention working conditions (Nielsen & Randall, 2012), previous experience with change processes and resultant positive appraisal of change processes (Framke et al., 2019), the change valence (i.e., the extent to which organisational actors perceive the change as needed, important, or worthwhile) (Weiner, 2009), a shared understanding of the needed changes among managers and their employees (Hasson et al., 2013), a shared positive vision for the future among managers and their employees (Nielsen et al., 2010b), and the collective efficacy (i.e., the extent to which organisational actors feel capable of solving the problems as a group and of making changes to psychosocial working conditions) (Abildgaard et al., 2020).
Multi-level management onboarding process. RE-AIM requires the evaluation of the managers’ onboarding process. This evaluation is important because the onboarding process of managers can influence their support of the intervention. For example, Busch et al. (2017) reported that to get management onboard, they were assured that the intervention would be low cost (by utilising the services provided by non-profit agencies); subsequently, managers offered the intervention to their employees and supported them the intervention activities.
Multi-level management support. The five-phase model requires evaluating managers’ support of the intervention. This evaluation is crucial because it helps determine how multi-level management support through different mechanisms promotes intervention outcomes. For instance, senior managers may support the intervention by introducing the mechanisms of committing to the intervention at the start of the intervention (Schelvis et al., 2016), allocating resources, and facilitating development and implementation of the intervention (Busch et al., 2017). Similarly, middle managers may support the intervention through the mechanisms of commitment to the intervention at the start of the intervention (Schelvis et al., 2016), participation in the development and implementation of the intervention (Abildgaard et al., 2018), and the performance of transformational leadership (Lundmark et al., 2017).
Aligning the intervention aims and objectives with organisational vision and values. RE-AIM requires evaluating maintenance of the intervention. A key to achieving maintenance is aligning aims and objectives of the intervention with vision and values of the organisation. This evaluation is important because it helps to understand how the intervention was institutionalised and maintained in the organisation. The literature shows that aligning the intervention aims and objectives with organisational vision and values works in two ways. First, through affecting the perceptions of senior managers about the alignment of the intervention aims and objectives with organisational goals (Schelvis et al., 2016). Second, through affecting the perceptions of middle managers and employees about aligning the intervention aims and objectives with their shared values (Nielsen et al., 2017a).
Establishing steering groups and assigning a project champion and their support of the intervention. The five-phase model requires evaluating the establishment of steering groups, the assignment of a project champion, and their support of the intervention. Evaluating the establishment of steering groups and their roles is essential because it helps to understand how steering groups were formed, the selection criteria for including members, their representativeness of the entire organisation, their decision latitude, and how they influenced intervention activities. Previous research has found that if the steering groups have the necessary autonomy and resources and consist of members with influence and credibility, they can enable employees to contribute their ideas and provide honest feedback that affect intervention outcomes (Jenny et al., 2015). Also, evaluating the assignment of a project champion and its roles is important because it helps to determine how a project champion was assigned, the required competencies for the role, the champion’s decision latitude, and how the champion managed the intervention. The literature shows that champion involvement is a key strategy for awareness-raising and culture change, provided that the champion possesses the personal characteristics, seniority, and skills required by the role (Brakenridge et al., 2018).
Communication strategy. The five-phase model requires evaluating the communication strategy regarding the intervention. This evaluation is essential as it helps to understand what kind of information has been distributed, to whom, and how it has been received and perceived. The literature shows that a communication strategy containing rationale behind the intervention, process and progress of the intervention, and expected outcomes that uses two-way communication, including the main and feedback channels, raises employees’ awareness of the intervention (DeJoy et al., 2010), increases the chance of cognitive appraisal of employees (Nielsen et al., 2014), triggers co-learning processes (Nielsen & Randall, 2012), and increases the quality of action plans which in turn improves working conditions and employees’ health and wellbeing (DeJoy et al., 2010).
Intervention Phase 3: Screening
Tailoring risk assessment methods. The five-phase model requires evaluating risk assessment methods. This evaluation is vital as it helps to determine how risk assessment methods were tailored to fit the organisational context (e.g., existing psychosocial working conditions) and how intervention activities were prioritised and planned based on the results of the risk assessment methods. The literature shows that using specific risk assessment methods may function as a mechanism in developing action plans. For instance, Nielsen et al. (2014) found a tailored questionnaire enabled participants’ understanding of their working conditions and made it easier to develop initiatives specific to their working conditions.
Reporting the results of the risk assessment. The five-phase model requires evaluating how the risk assessment results were reported to employees and managers. This evaluation is crucial as it helps to understand how the reporting process enabled employees and managers to make sense of their working conditions and determine further intervention activities. The literature shows that reporting the risk assessment results to employees and managers facilitates developing concrete action plans, leads to more intervention activities, and influences the success of the intervention (Bourbonnais et al., 2006; Nielsen et al., 2014).
Intervention Phase 4: Action Planning
Process of participatory action planning. The five-phase model requires evaluating the process of action planning. This evaluation is critical as it helps to identify how employees and their (line) managers collectively translated risk assessment results into action plans and how and why activities were prioritised. The literature shows that the processes of action planning affect intervention outcomes. For instance, von Thiele Schwarz et al. (2017) found that using the Kaizen board to develop and implement action plans increased employees’ awareness of and capacity to manage psychosocial issues and their wellbeing. Sørensen and Holman (2014) reported that developing action plans in workshops and refining these plans by employees’ initiative leaders (who were appointed to refine the plans and to coordinate the subsequent implementation process) improved relational job characteristics and burnout.
Contents of action plans. The five-phase model requires evaluating the contents of action plans. Evaluating the contents of action plans is essential as it helps to understand how the contents of action plans produced intervention outcomes. The literature shows that changing specific working conditions produce specific intervention outcomes. For instance, Holman and Axtell (2016) found that managing administrative tasks improved employees’ job control (outcome) and clarifying the performance criteria improved feedback. Sørensen and Holman (2014) found that targeting task uncertainty, task ambiguity, job complexity, and task interdependencies to change improved relational job characteristics and burnout.
Integrating the intervention activities into organisational policies and practices. RE-AIM requires evaluating maintenance of the intervention. A key to achieving maintenance is integrating the intervention activities into organisational policies and practices. The evaluation of integrating intervention activities into organisational policies and practices is important as it helps to determine how the intervention activities were embedded into day-to-day practices of the organisation and how participants designed, organised, and managed their jobs differently that lasted in the organisation. The literature shows that to evaluate the effects of integrating the intervention activities into organisational policies and practices on the intervention outcomes, three issues should be explored. First, it should be explored how the intervention activities were integrated into the existing management system, primarily quality improving and production systems such as Lean production (von Thiele Schwarz et al., 2017). Second, it should be explored how the intervention activities were integrated into the work routine of the organisation. For instance, participatory decision making, in addition to focusing on employees’ health and wellbeing, can be employed in other organisational processes such as HR practices (Nielsen et al., 2017b). Third, it should be explored how the integration of intervention activities into organisational policies and practices was perceived by managers and employees (Nielsen & Randall, 2012).
Intervention Phase 5: Implementation
Process of implementing action plans. The five-phase model requires evaluating the process of implementing action plans. This evaluation is crucial because it helps to understand whether and how employees and their (line) managers collectively prioritised action plans and implemented the action plans. The literature shows that the process of implementing action plans affects intervention outcomes. For instance, DeJoy et al. (2010) reported that implementing action plans by an “Action Team” (consisting of 8–12 employees from different departments) improved organisational commitment, job satisfaction, and employees’ health and wellbeing. Holman and Axtell (2016) found that forming implementation teams consisting of employees with a team leader to implement the intervention activities and holding regular meetings among researchers, employee representatives, team leaders, and managers to discuss the progress of implementation improved employees’ job control and wellbeing.
Achieving a higher level of intervention fidelity. RE-AIM requires evaluating intervention fidelity which is the extent to which the intervention was implemented according to its original protocol. Evaluating fidelity is important as it helps to identify what was planned, what actually took place, and why there were differences (if any) between them. Further, unless the evaluation of fidelity is made, it cannot be determined whether the failure of the intervention was due to poor implementation (programme failure) or inadequacies inherent in the intervention programme (theory failure). The literature shows that contextual factors can influence the relationship between intervention fidelity and intervention outcomes. For instance, Schelvis et al. (2016) reported that a high level of intervention fidelity was related to a low level of overall satisfaction due to a lack of employees’ involvement in the choice of intervention activities and a lack of mutual trust. Oude Hengel et al. (2012) concluded that failure of the intervention could be attributed to both medium level of intervention fidelity and a theory failure, because the theory behind the intervention did not address the problem (e.g., changes in communication were at the individual level by relying on workers, but it should be at the organisational level by relying also on supervisors and middle management).
The traditional concept of fidelity (i.e., fidelity to the original protocol), however, is less useful in realist evaluation and could be re-articulated to show fidelity to the initial CMO configurations (Wong et al., 2017). In realist evaluation, fidelity is measured based on the initial CMO configurations, not the intervention original protocol (Wong et al., 2017). This means, the empirical CMO configurations representing “what worked for whom in which circumstances?” should be compared with the initial CMO configurations representing “what might work for whom in which circumstances?” this comparison should be used to confirm, refute, or modify the initial CMO configurations in order to understand “what works for whom in which circumstances?” (Pawson & Tilley, 2004). As such, the IREMOI suggests that the work done to achieve a higher level of fidelity to initial CMO configurations as a mechanism as it can influence the mental models of participants regarding the intervention.
Dose delivered and dose received. RE-AIM requires evaluating dose delivered (i.e., how many intervention activities were delivered by intervention providers) and dose received (i.e., the extent to which intervention participants received and participated in intervention activities). Evaluating dose is important as it helps to identify the relationship between components delivered to participants and participants’ use of such components and their collective effects on the intervention outcomes. For instance, Sørensen (2016) reported that higher implementation intensity had stronger effects on employees’ health and organisational effectiveness, and Sørensen and Holman (2014) outlined that higher levels of dose delivered and dose received in an intervention group resulted in greater improvements in relational job characteristics compared to other groups. The literature also shows that contextual factors can influence the relationship between dose and intervention outcomes. For instance, Gupta et al. (2018) reported that 100% dose delivered with 69% dose received did not improve the intended outcomes since additional burden on the workers who already faced high demands and efforts at work caused the negative perception of the intervention.
Realist evaluation, however, criticises the terms of dose delivered and dose received in two ways. First, realist evaluation views intervention participants as active agents, rather than passive recipients of the intervention components delivered by intervention providers. Realist evaluation suggests that intervention providers and participants should engage in a “teacher-learner relationship” or “assisted sense-making relationship” and interact with each other and develop and test CMO configurations (Pawson & Tilley, 2004). Second, the terms of dose delivered and dose received imply quantitative measures (Gupta et al., 2018). Realist evaluation, however, advocates the use of qualitative measures to develop an in-depth understanding of how intervention participants interact with providers, how they perceive the intervention, and how they change their behaviours in response to the intervention. Hence, the IREMOI suggests combining quantitative and qualitative measures as it helps to provide a better measurement of dose and ultimately helps with testing CMO configurations.
Table 2 shows how to develop CMO configurations for the above intervention components and provides examples of CMO configurations.
Discussion
This article presented a model for what organisational intervention components to evaluate, when, why, and how based on realist evaluation. It integrated the RE-AIM dimensions (Glasgow et al., 1999) into the five-phase model (Nielsen & Abildgaard, 2013) to include crucial intervention components when evaluating organisational interventions. Further, it provided guidance on how to apply realist evaluation by describing when, why, and how to develop a CMO configuration for each intervention component and provided examples of CMO configurations for intervention components. As such, this article demonstrates how working with CMO configurations systematically in intervention may improve the understanding of “what works for whom in which circumstances” and thereby the likelihood of intervention success.
Regarding “what works for whom in which circumstances?” this article developed examples of CMO configurations for a range of mechanisms in preparation, screening, action planning, and implementation phases. Each CMO configuration shows how the mechanisms may be triggered, what contextual factors may influence the operation of each mechanism, and what outcomes each mechanism may produce (Nielsen & Miraglia, 2017). Future intervention studies can test and refine these CMO configurations to understand “what works for whom in which circumstances?” in organisational interventions.
Our proposed IREMOI has seven strengths. First, this model is more extensive than the five-phase model because some crucial intervention components were added which are essential for evaluating organisational interventions. Considering a larger set of intervention components in the evaluation sheds light on what intervention components to include, when, why, and how, this provides a more valid answer to the question of “what works for whom in which circumstances” regarding organisational interventions. Second, this model is a theory-driven model based on CMO configurations. If researchers and occupational practitioners use our model, it will enable them to develop initial CMO configurations and test whether empirical CMO configurations confirm, refute, or modify the initial CMO configurations (Nielsen & Miraglia, 2017; Pawson & Tilley, 1997). Third, this model has the flexibility that it allows users of the IREMOI to identify the most relevant and promising CMO configurations which are fit the specific intervention aims, the specific contexts, and the desired outcomes. Fourth, since CMO configurations explain “what works for whom in which circumstances,” following IREMOI that is a CMO-based model will most likely lead to higher internal and external validity of the organisational intervention findings (Nielsen & Miraglia, 2017). Fifth, in this model, evaluating empirical CMO configurations throughout the implementation process not only improves the understanding of how and why changes in the intervention components, participants, their roles, and their participation during the intervention affected intervention outcomes but also avoids retrospective sense-making of the intervention (Nielsen & Randall, 2013). Sixth, since CMO configurations are not equally important during all of the intervention phases, developing CMO configurations as shown in our model and aligning CMO configurations over the intervention period improves the accumulation of resources to achieve the intervention outcomes (Tafvelin et al., 2019). Last but not least, the suggested CMO configurations contained two mechanisms constructs (1) intervention resources and (2) participants’ individual and collective reasoning and reactions. These two constructs shed light on how interventions produce outcomes. To explore intervention resources, future studies can consult intervention processes identified in the reviews by Roodbari et al. (2021) and Nielsen, De Angelis, et al. (2022). Also, to explore participants’ individual and collective reasoning and reactions, future studies can consult the review by Nielsen et al. (2022), who provided examples of participants mental models (i.e., participants’ appraisals of the intervention process), and Karanika-Murray and Biron (2013), who suggested the ways an intervention can exert its impact on participants’ individual and collective reasoning and reactions. These suggested ways include: (1) diffusion, contagion, or spillover of the effects of an intervention (e.g., intended diffusion of expected positive effects among intervention groups can influence their participation), (2) shared meaning (e.g., participatory approaches can generate a sense of shared meaning of ownership of the change among individuals), (3) social identity (e.g., forming intervention teams can foster individuals’ identification with their teams and influence change), (4) social comparison (e.g., showing others with better wellbeing as role models can boost participation in the intervention), (5) interpersonal influence (e.g., communicating the intended positive outcomes with others can affect their participation), and (6) social learning (e.g., participatory approaches triggers learning on how to monitor adverse psychosocial working conditions).
We recommend future organisational intervention studies to study mechanisms through reflecting on both intervention resources and consequent participants’ individual and collective reasoning and reactions.
Limitations and Challenges of the IREMOI
The IREMOI has two limitations. First, the CMO configurations are extended hypotheses that take into account the contextual factors and participants’ individual and collective reasoning and reactions; however, the contextual factors and participants’ individual and collective reasoning and reactions could be different in different organisations which could produce diverse outcomes. Regarding contexts, both the omnibus contexts (e.g., financial situations of organisations) and discrete contexts (e.g., unexpected changes in organisational structure) could vary in different organisations. Also, regarding participants’ individual and collective reasoning and reactions, participants’ sense-making (i.e., how individuals and groups perceive the intervention and their working conditions) and agency (how they react to intervention activities) could be different in different organisations. As such, this model should only be seen as a guideline for evaluating organisational interventions; the suggested CMO configurations should be refined with the cooperation of local intervention stakeholders to fit the organisational contexts and individuals within organisations. Second, this model may be criticised for not addressing in-depth the questions about (1) which intervention component triggers which individual and collective reasoning and reactions of participants (i.e., two constructs of a mechanism); (2) how and which specific contextual factors affect mechanisms and how; and (3) what the resultant outcomes of such interactions would be. We argue that to answer these specific questions, each intervention research project should develop the most relevant and promising initial CMO configurations (based on its specific intervention goals, specific contexts, and desired outcomes) and empirically test these CMO configurations (Pawson & Tilley, 2004).
We acknowledge that there are three challenges related to applying the IREMOI. First, the application of this model is time-consuming and needs skilled researchers or occupational health practitioners. The processes of developing initial CMO configurations, designing and implementing the intervention, and testing the initial CMO configurations require skills in collecting, analysing, and synthesising mixed data over a long period of time. Second, to evaluate interventions, the researchers or occupational health practitioners should be aware of the complexity of psychological health and wellbeing and be able to causally relate contexts, mechanisms, and outcomes in CMO configurations. Third, collecting rigorous data is resource-consuming. To mitigate these challenges, we recommend focusing on the most relevant and promising CMO configurations in each intervention study.
Implications for Research and Practice
Organisational interventions are complex, and this complexity can advantageously be addressed by evaluation frameworks (Nielsen, 2013). In response to this call for evaluation frameworks, we propose the IREMOI to evaluate complex organisational interventions. Since the call for evaluation frameworks has arisen from research, practice, and policy levels, we briefly discuss the contribution of our model to each level. From the research perspective, our model is based on realist evaluation which is a recommended approach to evaluate complex organisational interventions (Nielsen & Miraglia, 2017). Therefore, our model provides a theoretical framework based on realist evaluation for researchers to evaluate organisational interventions. From the practice point of view, our model improves the understanding of change processes in organisations. Therefore, our model can be used by occupational health practitioners and organisational managers to improve employees’ health and wellbeing within organisations. Finally, from the policy perspective, our model has the potential to provide a basis for national policies whose aims are to manage psychological risks and ensure employees’ health and wellbeing. Thus, our model can, in the long term, be used by policymakers. Given these, the success of our model like other evaluation frameworks depends on the collaboration and support of researchers, occupational health practitioners, organisational managers, and policymakers.
Conclusion
In conclusion, this paper presents an Integrated Realist Evaluation Model for Organisational Interventions (IREMOI) that can be used by researchers, occupational health practitioners, and organisational managers to design, implement, and evaluate organisational interventions. The IREMOI is based on two evaluation frameworks of the five-phase model (proposed to evaluate organisational interventions) and RE-AIM (proposed to evaluate community-based, health-promoting interventions) and applies the CMO configuration of realist evaluation. We expect that applying this model will improve the understanding of “what works for whom in which circumstances?” in specific interventions, and that such understanding may increase the likelihood of future interventions successes in general.
Funding Inforamtion
Funding support for Hamid Roodbari was provided by Sheffield University Management School.
Competing Interests
The authors have no competing interests to declare.