Organizational Change Through Computer-Supported Human-In-The-Loop Simulation: Supporting Constructive Change

Organizational Change Case and the Overall Goals Of Human-In-The-Loop Simulation Workshops

We conducted HITL simulations (“workshops”) between 2017 and 2024: Workshop 1 was held in 2017; Workshop 2 in 2018; Workshop 3 in 2019; Workshop 4 in 2022 and Workshop 5 in 2024.

These workshops were conducted in the context of a steady increase in the complexity of the security environment and the challenges of developing organizational structures and technologies sufficiently agile to deal with an ever-more complex operational environment (Johansson et al., 2003; Johansson, Berggren & Trnka, 2015). Introducing stealth aircraft such as the F-35 fighter formed part of this development (Binnendijk et al., 2020). The simulation-based workshops principally sought to contribute to the development of the procedures for decision-making and coordination in the use of F-35 planes to counter novel threats.

Throughout, a “cadre” of development officers and experienced officers provided input before, during, and after the workshops. First, one of the workshops primary uses has been to develop both procedures for coordinating the F-35 with other military entities in different types of missions, and for delegating authority to the F-35 and other entities (Workshops 1, 3, 4, and 5). Second, the workshops have been used to further develop delegation mechanisms between levels in the chain of command in air-defense scenarios (Workshop 2).

Based on these broad goals, we developed more specific goals for the workshops. Together with military personnel, we developed concepts – ideas about, for example, coordinating the implementation and use of F-35 aircraft with other entities. This part of the more detailed development prior to each workshop took on an open-ended constructive development loop. We went from conceptual ideas of how to use the F-35 to building what we term “tactical vignettes” – defined geographical context (area of operation), defined tasks (e.g. the defense of a territory), and defined friendly and enemy entities, and so on. We also defined both the principal types of information to be exchanged and the chain of command. This definition did not restrict when the participants could pass information, meaning that precise work routines were not fixed by the simulation software. We ran simulations and then drew lessons, permitting us to further refine concepts (Alberts & Hayes, 2002).

Designing an organization template for coordinating responses in a “multi-team” setting – that is, teams that collaborate to solve an overarching goal (Mathieu et al. 2018) – required that we develop a flexible organizational structure on the fly. In particular, it was necessary for us to be able to adjust the type and number of entities involved from workshop to workshop. We then designed the simulation technically to accommodate best practices, to support a good user-experience, and to contribute to the change initiatives; specifically, the aim was to contribute to procedural changes regarding C2-arrangements testable in real-life exercises. The simulation workshops were thus used specifically to support concrete organizational change initiatives.

In the section following, we discuss the intersection of organizational change and key aspects of HITL simulations in general before expanding on the specific methods used to facilitate and gather data from the workshops and reporting on results from the concrete workshops and change efforts.

Research on Simulation and Organizational Change

For Lyons et al. (2011, p. 210), change may be examined through “experimenting with the real organization or with a mathematical model of the organization”. With respect to simulations, the use of mathematical models may in general be important for theory development (Harrison et al., 2007, p. 1240); so far, this is the type of simulation most frequently discussed in the relevant research literature. Mathematical simulations have indeed been used by researchers to understand organizational change (Dooley, 2021), in particular using agent-based and computational models. The strength of agent-based simulations in research on organizational change lies, perhaps, in what they offer to the understanding of surprising patterns of change, and that they can be used by organizations to solve specific operations research problems (Dooley, 2021), potentially supporting prescribed change (Albert & Ganco, 2021; Waring et al., 2020).

Such simulations may indicate how a process of change may unfold, providing results for and permitting the participation of the organizational members (Lyons et al, 2011; Meier & Ingerslev, 2023). Both Dooley (2021) and Lyons et al. (2011, p. 219) suggest ways in which simulations can help organizational leaders “understand the incremental steps toward change success”. For example, mathematical simulation, specifically discrete event simulation, has the benefit of minimizing costly disruptions to operations at the same time as capturing the dynamic, discrete and stochastic elements of organizations. However, Lyons et al. (2011) note that simulation should be only one part of a larger effort to aid organizational change (Kotter, 1996).

Human-in-the-Loop Simulation and Organizational Change

In addition to computational and mathematical simulation, HITL simulation is associated with modeling and simulation (M&S) in the virtual domain, which involves real people operating simulated systems (Under Secretary of Defense for Acquisition Technology, 1998). HITL simulation is often referred to as– a simulation including human operators with the human as interacting with the simulation (Folds, 2015) such as those used by hospitals or aircrews (Christianson, 2019; Lacerenza et al., 2018).

HITL simulation is used to conduct wargames. Although wargames could broadly be seen as simulations themselves (Alme & Hvidsten, 2022), not all wargames are conducted through computer-simulations or computer-supported HITL simulations. Evensen et al. (2022, p. 324) indicate that their research institution increasingly draws on computer-supported wargames. HITL simulations make the organizational participants – F-35 pilots, naval and army operators, officers in C2 nodes, for example – more active in the overall simulation, with potential benefits for organizational change.

Prior research has discussed some aspects of HITL simulations for prescribed change; Salas and colleagues (2009), for example, put HITL simulation (including those not supported by computers) in the context of training competencies, while Billings (2012) considered ways to convey and organize feedback for the benefit of efficiency in training. Salas and colleagues (2009, p. 562) also briefly describe how training for prospective new tasks such as handling high-risk crisis situations can be supported (see also (Waller et al., 2014; Bakken et al., 2017), while others also discuss HITL simulation used to aid other types of prescribed change (Bell et al., 2008; Bennett et al., 2013; Meier & Ingerslev, 2023). Neither the research on training simulations nor organizational change, however, discusses the constructive use of simulation at any great length: the emphasis tends to lie on immediate training needs rather than explorations of novel ways of using technology, or the various integrations and delegations of authority that concern military entities. Additionally, training is most frequently done in highly specialized training simulations for single military platforms unconnected to other entities. It is often difficult to evolve beyond the predictable and well-known ways of operating. This lack of evolution may prove a liability in restricting the options of warfighters.

We consider how the depth or fidelity of HITL simulations may influence organizational change. “Fidelity” here refers to the extent to which a simulation replicates real-world systems and to its degree of resolution, which is the level of detail and precision used to represent real-world aspects (Brehmer & Dörner, 1993; Folds, 2015; Banks, 2001). Realism is closely connected to immersion (McLean & McLean, 2024), which concerns being “immersed in a task or setting as if it were the real world” (Lateef, 2010, 348). Immersion has also been defined by Martin (2023, p. 13) as “the objective level of fidelity of the sensory stimuli produced by a [virtual reality] system”.

“Breadth”, or the scope of the HITL simulation system, on the other hand, is a question of the number of different organizational entities involved in the simulation. These entities may or may not reflect numbers or types (Banks, 2001; McIntyre et al., 2013). “Breadth” may also refer to the number of possible ways a task can be solved, such as which entities could potentially perform various military tasks. With greater breadth, a simulation may support novel ways of solving tasks that are not necessarily realistic at a particular moment but remain relevant from an organizational change perspective in the future. Simulations can thus be used to examine many alternative ways of organizing (e.g., different ways of integrating the F-35 squadrons with other entities) and provide forecasts for change initiatives (indicating what type of integration in different F-35 missions is related to the best performance, for example; see Dooley, 2021; Lyons et al., 2011; Pereira, & Guillerm, 2019; Fiammante, 2010; Evensen et al., 2022). Typically, simulations can be used to examine a greater number of alternatives (for example, different approaches to integrating the F-35) in a shorter period of time than real-life exercises allow. Simulations may therefore serve as a force multiplier for military organizations undergoing extensive organizational changes.

Both fidelity and breadth can be conceptualized as continuous constructs. For simplicity, however, we discuss the implications of high versus low fidelity and high versus low breadth. Four configurations can be identified: (1) high fidelity and high breadth, (2) high fidelity and low breadth, (3) low fidelity and high breadth, and (4) low fidelity and low breadth (typically observed in the early development of HITL simulations). In the sections that follow, we focus on configurations 1, 2, and 3 in greater detail.

Taking the example of change in life-cycle as a typical prescribed change, HITL simulation may serve to provide the organization with ways to train prescribed competencies already decided upon by external regulations. A typical example could be the certification of organizational members, or teams, ensuring that they are competent and permitted to carry out certain predefined tasks. HITL simulation to support change through training could be facilitated by realistic immersive simulations in military setting: specifically, Martin (2023, p. 23), quoting Makransky and Peterson (2021, p. 949), suggests that “when interaction fidelity is maximized through the effective alignment of more immersive technologies and appropriate tasks, whose “physical activities are meaningful for the learning outcome,” increased learning gains can be achieved”. Indeed, some empirical research does indicate that immersive simulations contribute to training transfer (Martin, 2023), although the empirical results are mixed (Stevens et al., 2015).

Thinking of change initiatives from a life-cycle perspective, HITL simulation has the potential benefit of increasing change readiness (Lyons et al., 2011) and change efficacy (Wanberg & Banas, 2000). This could be done through exposure and rehearsal involving novel technology, constraints, processes, personnel, or organizational forms.

Within a prescribed evolutionary model, a set of predefined organizational forms could be simulated using HITL simulation, allowing the best-performing form to be selected; pre-specified and realistic organizational forms that include centralized decision making authority or a decentralization of decision making authority, for example, could be simulated. While HITL simulation could be used for these phases of prescribed change processes (where the best performing organizational forms are selected, for example), this may need to be complemented by discussions and dialogue among organizational members to establish the precise type of change required (in a life-cycle model, for example), and any alternatives and selection criteria in a potential evolutionary model (Aitken et al., 2021).

Turning to the constructive type of organizational change, where change is more open-ended than prescribed, precise models, and hence realistic simulations, could be more difficult and less practical to construct than they are for the accurate modeling of instances of prescribed change. In this context, it may be more important, and feasible, to incorporate novel elements into the simulation, for example by increasing the breadth of the HITL simulation.

In constructive change, organizational forms may be created in situ during HITL simulation sessions, allowing a high degree of flexibility and few restrictions on how tasks are solved or which organizational entities and forms are simulated. For example, the allocation of decision-making authority may be determined during the simulation rather than being prespecified in detail beforehand. Participants may also have the opportunity to explore the steps involved in decision making. In a dialectical model, tensions among the dimensions of organizational forms can be explored, such as trade-offs between centralized and decentralized decision-making authority. Several novel approaches to assigning decision-making authority may be tested within such a change model. Synthesis of organizational forms can occur, for example, during reflections following HITL simulation runs, potentially resulting in novel or hybrid organizational structures (Beck et al., 2008). In teleological types of organizational change, HITL simulation may be used as part of an evaluation of implementation efforts, taking into account specific organizational contexts when assessing the usefulness of, for example, a new organizational form.

In constructive change, the future characteristics of the organization may be less well known than in the prescribed change. Hence, there should be less requirement, and need, for high-fidelity or immersive simulations. The emphasis, rather, could be on support for multiple scenario-driven approaches that relax the requirement for high-fidelity. Increasing the number of entities should be a requirement – particularly in the dialectical mode, where one is exploring novel constellations of collaboration. The change and modification of entities in the simulation is also key, and the computer support should allow for rapid modifications.

This being said, detailed simulations may in some circumstances also be used in constructive change. Using high-fidelity simulations may be relevant if the purpose is not necessarily any change in what the detailed simulation is imitating (fighter aircraft, for example), but, rather, how such entities connect with other parts of the organization. The type of HITL simulation could thus vary based on the type of organizational change process in focus. In hybrid types of organizational change where both prescribed and constructive change play a role (see, for example, Edwards et al., 2020), one could expect that both a clearly defined and bounded HITL simulation and a more loosely defined and changeable HITL simulation could be used in combination. This suggests that the four different configurations of HITL simulation could be used strategically to support different organizational change needs.

Methods

HITL simulators typically represent military systems included in the given vignette (Babcock, 2004). We defined the vignettes and subtasks of the simulated entities (Rasmussen et al., 1990). The set of elements was constructed based on the chosen vignettes and the military systems and roles relevant to the experiment. The simulation network we built began with a foundation of C2 systems (i.e., military decision-support systems), facilitated operator inputs to the control systems through user interfaces, and included more detailed simulators (Walker et al., 2017). Over time, the simulation environments have evolved and expanded to include an increasing number of military system simulators and C2 systems. When the behavior of a system or capability has lower complexity, computer-generated forces (Bennett, Schreiber, & Andrews, 2002) may represent it, for example with behaviors guided by artificial intelligence (Strand et al., 2023). In our case, we used a mix of computer-generated forces and forces operated by humans.

Typically, friendly non-military entities and enemy entities were computer-generated, as were some of our own forces. However, some military equipment (the F-35, for instance) was operated by humans who could steer and control it through a user interface resembling the real thing. We used computer and simulation network standards such as the Distributed Interactive Simulation (DIS) and High Level Architecture (HLA) protocols, as well as Tactical Data Link (TDL), which made it easier to add or remove simulators from the environment as needed. Each simulator component was customized for each experiment and was thus under continuous development. Standalone simulations of more detailed aspects of mission accomplishment were also included in the workshop, though these were not technically connected to the HITL simulation network.

Preliminary full-scale tests of the simulation environment, for the most part with military experts, as a part of the verification and validation of the simulation environment and/or detailed discussion of simulation functionality, was done before each workshop (Pace, 2003; Pace, 2013; Meyer & Booker, 2001; Kerkering, 2002).

Together this provided the basis for conducting the workshops. We provide an overview of the workshops conducted, with respect to the purpose of the simulation workshops, the specific type of computer-support configuration (breadth and fidelity), and the perceptions of change, in Table 1 below, with a more detailed account of the survey items below. All workshops concerned mission types relevant to the F-35 or the F-35 system. While the F-35 simulation was not part of the 2018 workshop, the overarching scenario concerned the defense of F-35 bases and how this affected the use of the F-35, thus making it relevant for generating knowledge about the aircraft’s introduction.

Importantly, we both collected data on perceptions of change and measured the breadth of the simulations conducted. We constructed two items intended to reflect either the prescribed or the constructive aspects of change, along with measures of the influence of the change on participants’ real organizations and perceptions of fidelity, all scored on a seven-point Likert scale (see Appendix 1 for details on how these constructs were measured and on the data collection procedure). We quantified the breadth of each simulation by counting the maximum number of different types of units involved in a task during each simulation workshop (e.g., fighter aircraft units, frigates, decision nodes, and so forth).

Some key precautions were taken to ensure research ethics and to ensure security (Sjøgren et al., 2024). During data collection, the respondents were able to withdraw or to decline to answer survey items. The data collected were anonymized throughout the processes of analysis and presentation and stored in government data servers. With respect to security, we informed the cadre of officers that the specific survey data could be used for unclassified publication; the specific procedures and tasks, like the units involved in the workshops, being classified, are not mentioned in this article. General description of the missions and the units involved are, however, unclassified.

Findings Concerning the Relationship Between the HITL Simulations and Organizational Change

We now present data shedding light on prescribed and constructive aspects of organizational change related to each of the five workshops, based on data gathered from those who participated in the simulation. We show the average responses for all items for each workshop in Figure 1 below and the correlations and descriptive statistics in Table 2.

Figure 1

A rough characterization of the perception of fidelity of the simulation setup for each workshop (see Table 1) indicate that they ranged from relatively medium fidelity (2017, 2018, 2022 and 2024) to a bit lower (2019). Taken together, the fidelity of the simulation seemed to be perceived as very stable. The breadth varied to a larger degree: It ranged from comparatively low (2018) to medium (2019) to very high (2017, 2022, and 2024).

The correlations and descriptive statistics presented in Table 2 indicate that constructive change was significantly and positively associated with the recency of the simulation workshop (r = .73, p < .01) and with the breadth of the simulation (r = .58, p < .01). Year and breadth were also significantly and positively associated (r = .93, p < .01). Perceptions of prescribed change were rated higher (M = 5.7) than constructive change (M = 5.11). Over the years, participants’ perceptions of simulation fidelity have remained largely stable and were not significantly related to either perceptions of prescribed or constructive change. We also note relatively high scores for the relevance of the workshops across all sessions (M = 6.19), which may indicate that the HITL simulations have distal consequences. A caveat is that the number of respondents and simulation workshops is small, so the quantitative data should be interpreted only as indicative. Thus we complement our analysis with a more qualitative evaluation of the workshops.

Support for prescribed change

The first workshop (2017) supported prescribed change to a higher degree than constructive change, although the simulation included a relatively large breadth of entities. While the reactions on average among participants in the 2017 workshop were low for the constructive dimension, one of the participants in the 2017 workshop indicated the potential for the simulation to support constructive change in a free-text response: “The technical solution worked well (from my perspective), [it supports] the desire to be able to be challenged on ‘new’ scenarios every time, to be able to say something about command and control.”

In the second workshop of 2018, both prescribed and constructive change were rated higher than in 2017, despite a low breadth (i.e., few entities involved). The specific focus of this workshop was the delegation of decision rights rather than coordination, so the number of entities may have been sufficient to stimulate discussions of C2 related to delegation. Notably, this workshop received the highest score for clarity of purpose. A characteristic feature of the 2018 workshop was the relatively high number of detailed tasks illustrating various challenges for the operator; while this indicated high-fidelity simulation support, participants’ perceptions of fidelity were similar to those in the first workshop. The high score for constructive change may reflect the smaller number of participants, which could have made it easier for individuals to voice their opinions compared with workshops that included more participants.

The first and second workshop therefore nuance our theoretical stances. High-fidelity simulations with low breadth may support constructive change, but this may be due to the number of participants and the room for discussion after each simulation run due to the structuring of the discussions.

Support for constructive change

Over the years, the simulations appear to have increasingly supported constructive change, particularly by instigating discussions about new ways of organizing, rather than enhancing the prescribed aspects of change. This tendency toward greater constructive change seems to be related to a steady increase in breadth (i.e., the number of different units involved) but may also reflect the way post-simulation discussions were structured. In the subsequent workshops in 2019 and 2022, which had somewhat lower fidelity than the 2018 workshop, stimulation of discussions of command and control (constructive change) received similar (2019) or higher (2022 and 2024) scores than clarity of purpose (prescribed change).

In the fourth simulation workshop, conducted in 2022, where the constructive aspect was scored higher than the prescribed, the debrief sessions after each simulation run included broad discussions about the organization and procedures to be employed, as well as some technical experiences noted as it ran. In 2022, two different tasks were tried out in the HITL simulation, one focusing on using the Air Force and Navy entities, the other on Air Force and Army entities (land entities). Several potential amendments to standard operating procedures were identified after each HITL simulation run. Creative ways of improving procedures were suggested. Since the two tasks were repeated, the participants were able to try out these new suggestions. Even more experiences were gathered, and solutions to the procedural shortcomings suggested. It is likely that the inclusion of more entities contributed to this reflection. The 2022 workshop saw a broader and deeper involvement in the preparation of the workshop from a broader set of development officers, which may also have positively influenced the relatively higher score on constructive change.

The 2022 workshop provided input to the procedures being used for the air and maritime task. During the simulation, many shortfalls – including differences in terminology between the services, unclear descriptions of key roles, and a lack of awareness of the timing of the input to the task resolution made by various entities – were disclosed. These identified shortfalls were discussed extensively in the debrief sessions of the simulation workshop, with any resolution noted by those responsible for writing the procedures, and changes subsequently made to the joint air and maritime procedure. Revised procedures were tested in real-life exercises later in 2022. Participants noted that it would have been difficult to conduct the real-life exercise without the extensive simulation workshop, which served to clarify many difficulties arising in the course of the collaboration. Key personnel participated in both the simulation and the exercise. The development of procedures and subsequent testing could indicate that the type of simulation, emphasizing breadth rather than high fidelity, facilitated constructive and creative development in the organization.

In 2024 we ran the same simulation as in 2022 for the air maritime tasks and with the same number of entities, but with fewer manned positions. As in the 2022 workshop, two runs of the same task were conducted. Unlike the 2022 simulation, however, we provided an additional communication challenge between the units in the second run, requiring multiple roles to be capable of assuming leadership functions. One participant who had participated in the 2022 simulation suggested that the 2022 workshop itself had influenced joint procedures (as noted above). This participant also suggested that the operational level should exploit such arenas more to develop joint processes; this might include maritime procedures, with vignettes even exercised live, for example. This participant (among others) suggested in the course of the discussion that this kind of workshops should be mandated by military leadership in order to ensure participation be prioritized and workshops regularly conducted. Two participants who had also participated in the 2022 workshop indicated that experiences and discussions from the 2022 workshop had been very useful (rated 6 out of 7), and that the simulation had lower fidelity compared to the one conducted in 2022.