What Works for whom in which Circumstances? An Integrated Realist Evaluation Model for Organisational Interventions

EVALUATION ACTIONS IN THEFIVE-PHASE MODEL	EVALUATION ACTIONS IN THE RE-AIM FRAMEWORK	EVALUATION ACTIONS IN THE IREMOI
Evaluation Evaluating: Intervention process Intervention outcomes	Effectiveness Evaluating outcomes at the completion of the intervention Maintenance Evaluating outcomes six months or more after the most recent intervention contact	Evaluation Planning Conducting process and outcome evaluations through CMO configurations by: Developing initial CMO configurations Collecting empirical data on the implementation process and intervention outcomes Analysing and synthesising empirical data and developing empirical CMO configurations Testing initial CMO configurations against empirical CMO configurations
Preparation Evaluating: Organisational readiness for change Employees’ readiness for change Multi-level management support Steering groups and project champion support Communication strategy	Adoption and Reach Evaluating the recruitment process of organisational units Maintenance Evaluating the process of aligning the intervention with organisational vision and values	Preparation Developing CMO configurations about the mechanisms of: Recruiting organisational units Organisational units’ readiness for change Multi-level management onboarding process Multi-level management support Aligning the intervention aims and objectives with organisational vision and values Establishing steering groups and assigning a project champion and their support of the intervention Communication strategy
Screening Evaluating: Existing systems Process of feeding back the results to employees	-	Screening Developing CMO configurations about the mechanisms of: Tailoring risk assessment methods Reporting the results of the risk assessment
Action planning Evaluating: Process of action planning Contents of action plans	Maintenance Evaluating the process of integrating the intervention into organisational policies and practices	Action planning Developing CMO configurations about the mechanisms of: Process of action planning Contents of action plans Integrating the intervention activities into organisational policies and practices
Implementation Evaluating: Process of implementing action plans Intervention activities against planned intervention activities	Implementation Evaluating: Intervention fidelity Dose delivered and dose received	Implementation Developing CMO configurations about the mechanisms of: Process of implementing action plans Achieving a higher level of intervention fidelity Dose delivered and dose received

Table 2

How to develop CMO configurations for the intervention components and examples of CMO configurations.

QUESTIONS TO BE ASKED TO DEVELOP CMO CONFIGURATIONS	EXAMPLES OF CMO CONFIGURATIONS
PREPARATION
Recruitment process of organisational units Mechanisms: how were the organisational units were recruited in terms of how organisational units were identified, how were they provided with information about the goals and processes of the intervention, how were they invited to participate in the intervention, and why did they accept or decline to participate in the intervention? Contextual factors: which contextual factors facilitated and impaired the recruitment process and how? Outcomes: how did the recruitment process affect participation in the intervention? Organisational units’ readiness for change Mechanisms: how did the organisational units perceive problems in the current situation, see the need for intervention, and believe the intervention would have the desired effects? Contextual factors: which contextual factors influenced organisational units’ readiness for change and how? Outcomes: how did the organisational units’ readiness to change affect participation in the intervention?	If organisational units have reasonably good working conditions, organisational actors have a moderate to a good level of health and wellbeing, and their change valence and collective efficacy are high (contextual factors); *then* a recruitment process of organisational units in which all intervention actors are informed about the goals and processes of the intervention and are invited to voluntarily participate in the intervention can trigger employees’ and managers’ positive appraisal of the intervention (mechanisms), improve their readiness for changing problematic working conditions (proximal outcomes), and improve their perceived organisational support (intermediate outcomes).
Multi-level management onboarding process Mechanisms: how did multi-level management get onboard in terms of how they were provided with information about the goals and processes of the intervention, how were they invited to participate in the intervention, and why did they accept or decline to participate in the intervention? Contextual factors: which contextual factors facilitated or hindered multi-level management onboarding process and how? Outcomes: how did managers’ onboarding process improve employees’ awareness of and engagement in the intervention? Multi-level management support of the intervention Mechanisms: how did managers at all levels support the intervention? Contextual factors: which contextual factors facilitated or hindered multi-level management support of the intervention and how? Outcomes: how did multi-level management support improve employees’ awareness of and engagement in the intervention?	If managers have necessary individual resources (e.g., knowledge, skills, a good level of health and wellbeing) and there are organisational resources (e.g., budget, time) (contextual factors); *then* multi-level management onboarding and their support of the intervention can trigger employees’ commitment to the intervention (mechanisms) improve employees’ collaboration in changing problematic working conditions (proximal outcomes), and improve their perceived managerial support (intermediate outcomes).
Aligning intervention aims and objectives with organisation vision and values Mechanisms: how were aims and objectives of the intervention aligned (or so-called philosophical fit) with vision and values of the organisation? Contextual factors: which contextual factors facilitated or impaired the aligning process and how? Outcomes: how were the process of aligning intervention aims and objectives with organisation vision and values perceived by managers (particularly senior managers) and employees?	If there are necessary resources in the organisation for conducting the intervention and organisational and employees’ readiness for change are high (contextual factors); *then* aligning aims and objectives of the intervention with the vision and values of the organisation can trigger diffusion of expected positive intervention outcomes among employees and managers (mechanisms), improve their reflection on working conditions (proximal outcomes), and minimise their role conflict (intermediate outcomes).
Establishment of steering groups, the assignment of a project champion, and their support of the intervention Mechanisms: how were the steering groups established, how was the project champion assigned, and how did they affect the process of the intervention? Contextual factors: which contextual factors facilitated or impaired establishment of steering groups, assignment of a project champion, and their support of the intervention, and how? Outcomes: how did establishment of steering groups, assignment of a project champion, and their support of the intervention influence intervention outcomes?	If individuals who are interested in becoming a member of steering groups or the project champion have the necessary autonomy and resources, including skills, influence, and credibility (contextual factors); *then* transparent and participatory establishment of steering groups and assignment of the champion can trigger employees’ and managers’ shared meaning of ownership of the intervention (mechanisms), improve their collaboration in identifying and managing problematic working conditions (proximal outcomes), and improve their perceived autonomy (intermediate outcomes).
Communication strategy Mechanisms: what did the communication strategy contain, and how did the communication strategy affect the participatory process of the intervention? Contextual factors: which contextual factors facilitated or impaired effective communication in the organisation about the intervention and how? Outcomes: how did employees and managers perceive the communication?	If there is a climate of openness, trust, and respect in the organisation and there are enough resources in terms of time, energy, and infrastructure (contextual factors); *then* a communication strategy which directs effective communication across the organisation about the intervention aims and processes can trigger employees’ and managers’ interpersonal influence (mechanisms), improve their awareness of their psychosocial working conditions (proximal outcomes), and improve organisational culture (intermediate outcomes).
SCREENING
Tailoring risk assessment methods Mechanisms: which risk assessment method with which measures was used to identify psychosocial working conditions and how? Contextual factors: which contextual factors facilitated or impaired the process of identifying psychosocial working conditions and how? Outcomes: how did the process of identifying psychosocial working conditions with its measures affect employees’ and managers’ awareness of and capacity to manage adverse psychosocial working conditions and subsequent process mechanisms (e.g., developing action plans) and content mechanisms (e.g., the content of action plans)?	If there are necessary organisational resources for conducting a risk assessment and organisational and employees’ readiness for change are high (contextual factors);*then* using a tailored risk assessment method that measures local psychosocial working conditions can trigger employees’ and managers’ acceptance of intervention activities (mechanisms), improve their sense-making of their psychosocial working conditions (proximal outcomes), and improve targeted psychosocial working conditions (intermediate outcomes).
Reporting the results of the risk assessment Mechanisms: how were the risk assessment results reported to employees and managers? Contextual factors: which contextual factors facilitated or hindered the reporting process and how? Outcomes: how did the reporting process influence employees’ and managers’ appraisal of the intervention and their sense-making of their psychosocial working conditions?	If there are necessary organisational resources (e.g., infrastructure for meetings), the existing working conditions allow managers and employees to attend meetings, and there is a culture of trust and openness in the organisation (contextual factors); *then* reporting the results of the risk assessment to both employees and managers through regular meetings where the results can be discussed in the meetings can trigger employees’ and managers’ social learning (a mechanism), broaden their horizon regarding psychosocial working conditions (proximal outcomes), and improve their perceived social support (intermediate outcomes).
ACTION PLANNING
Process of participatory action planning Mechanisms: how were action plans developed; in particular how did employees and managers in a participatory process jointly develop action plans? Contextual factors: which contextual factors facilitated or impaired the participatory process of action planning and how? Outcomes: how did the process of participatory action planning affect subsequent process mechanisms (e.g., implementation of action plans), content mechanisms (e.g., the content of action plans), and employees’ and managers’ awareness of and engagement in the intervention and their perceptions of their working conditions?	If existing job design and employees’ and managers’ health and wellbeing are reasonably good and there are necessary resources in the organisation (e.g., time, infrastructure) (contextual factors); *then* participatory action planning can trigger employees’ and mangers’ shared meaning of ownership of the intervention and their interpersonal influence (mechanisms), empower them to manage their psychosocial issues (proximal outcomes), and improve employees’ autonomy (intermediate outcomes).
Contents of action plans Mechanisms: what were the contents of action plans, in particular, what were the relevance and importance of the working conditions that were targeted to change? Contextual factors: which contextual factors influenced the contents of action plans and how? Outcomes: how did the contents of action plans affect subsequent process mechanisms (e.g., implementation of action plans) and employees’ and managers’ engagement in the intervention and their perceptions of working conditions?	If employees and managers have necessary individual resources (e.g., readiness for change, knowledge, skills), there are organisational resources (e.g., time, infrastructure), and employees and managers have a shared understanding of psychosocial working conditions (contextual factors); *then* jointly determining the contents of action plans by targeting adverse psychosocial working conditions to change can trigger their shared meaning of ownership of the intervention and social learning (mechanisms), improve their sense-making of their psychosocial working conditions (proximal outcomes), and improve targeted psychosocial working conditions (intermediate outcomes).
Integrating intervention activities into organisational policies and practices Mechanisms: how were intervention activities integrated into organisational policies and practices? Contextual factors: which contextual factors facilitated or impaired this integration process and how? Outcomes: how was the integration process perceived by managers and employees that influenced their engagement in the implementation of the intervention?	If jobs are well-designed, and change valence and collective efficacy are at high levels (contextual factors); *then* integrating intervention activities into organisational policies and practices triggers diffusion of expected positive intervention outcomes among employees and managers (mechanisms), improve their reflection on their psychosocial working conditions (proximal outcomes), and minimise their role conflict (intermediate outcomes).
IMPLEMENTATION
Process of implementing action plans Mechanisms: how were action plans implemented, in particular, how did employees and managers in a participatory process jointly implement action plans? Contextual factors: which and contextual factors facilitated or impaired the participatory process of implementing action plans and how? Outcomes: how did managers and employees perceive the implementation process that affected their behaviours and resultant intervention outcomes?	If employees and managers have necessary individual resources (e.g., knowledge, skills) and there are organisational resources (e.g., time, budget, infrastructure) (contextual factors); *then* a participatory process of implementing action plans can trigger employees’ and managers’ shared meaning of ownership of the intervention (mechanisms), improve their collective efficacy to change problematic working conditions (proximal outcomes), and improve employees’ autonomy (intermediate outcomes).
Achieving a higher level of intervention fidelity Mechanisms: what was done to achieve a higher level of fidelity to initial CMO configurations? Contextual factors: what contextual factors facilitated or impaired achievement of a higher level of fidelity and how? Outcomes: what were the outcomes of a higher level of fidelity?	If the theory behind the intervention has a strong theoretical basis and addresses the existing psychosocial working problems and managers and employees have positive appraisals of the intervention and participate in the intervention activities (contextual factors); *then* work done to achieve a higher level of intervention fidelity to initial CMO configurations can trigger employees’ and managers’ excitement to keep the intervention and feeling of moving forward (mechanisms), improve their awareness of their psychosocial working conditions (proximal outcomes), and improve their psychosocial working conditions (intermediate outcomes).
Dose delivered and dose received Mechanisms (regarding dose delivery): did intervention providers, including managers, steering groups, and external consultants engage in developing and implementing action plans and how (e.g., by holding regular meetings with employees)? Mechanisms (regarding dose received): did employees participate in developing and implementing action plans and how (e.g., by attending regular meetings)? Contextual factors: which contextual factors facilitated or impaired delivering and receiving dose and how? Outcomes: how did employees perceive their interactions with intervention providers regarding the intervention that affected their behaviours and, in turn, intervention outcomes?	If there are necessary organisational resources (e.g., infrastructure, time, budget) and employees’ and managers’ individual resources (e.g., motivation, skills), and a supportive culture facilitate the implementation process (contextual factors); *then* high levels of dose delivered and dose received can trigger employees’ and managers’ shared meaning of ownership of the intervention, interpersonal influence, and social learning (mechanisms), empower them to manage their psychosocial working conditions (proximal outcomes), and promote their psychosocial working conditions (intermediate outcomes).

References

Authors

Metrics

Articles in this issue

DOI: https://doi.org/10.16993/sjwop.171 | Journal eISSN: 2002-2867

Journal RSS Feed

Language: English

Page range: 4 - 4

Submitted on: Dec 13, 2021

Accepted on: May 16, 2023

Published on: Jun 13, 2023

Published by: Stockholm University Press

In partnership with: Paradigm Publishing Services

Keywords:

realist evaluation,

RE-AIM framework,

organisational interventions,

Context-Mechanism-Outcome configurations,

occupational health,

working conditions

© 2023 Hamid Roodbari, Karina Nielsen, Carolyn Axtell, published by Stockholm University Press
This work is licensed under the Creative Commons Attribution 4.0 License.

Volume 8 (2023): Issue 1

What Works for whom in which Circumstances? An Integrated Realist Evaluation Model for Organisational Interventions

Figures & Tables

Table 1

Figure 1

Table 2

Paradigm

My account