Making Hybrid War Teachable and Learnable: Combining Artificial Intelligence and Virtual Reality in Military Education

Learning Context

Hybrid war represents a challenge to liberal democracies and to military education. We describe an exploratory approach to this challenging knowledge domain to see how AI and VR may make it teachable and learnable within higher military education. We employ a VR environment, a 3D map, with an AI companion: an avatar with a built-in generative large language model (LLM). The AI companion is built on InWorld’s Character Engine to emphasize personality, context, and relationships. The companion acts both as “guide on the side” and the interface between the learner and the information resources in the VR map. Students can thus draw on two resources, independently or in combination. In addition, we employed a problem-based learning method inspired by red teaming (roughly, the simulation of hostile action) combined with debriefing to offer learners ways to navigate the prototype test.

We convey the preliminary findings of the exploration guided by two questions. First, how can the combination of AI companions and immersive VR scenarios influence learning in higher military education? And, second, how can the prototype application aid military students’ understanding of hybrid war? We present our approach within the problem-gap-hook heuristic, to identify and discuss these educational challenges (Lingard, 2015).

The Gap: Ontological Shifts and Lack of Relevant Knowledge

The changes in how the West perceive threats from Russia and China represent an ontological shift. Hybrid threats are not war, a phenomenon based on a set of well-established laws, policies, and doctrines. But Europe cannot said be to be enjoying peace. Russia and China target our grey zones through open, covert, and clandestine means to undermine our societies. Ontological shifts imply that learning objectives must be revised. In the social sciences, current times are characterized by ambiguity and complexity (Thomassen, 2009, 2018). In educational literature, there is an increasing focus on liminality and uncertainty in learning processes, and how new and troublesome knowledge creates difficult thresholds for learners to pass (Meyer & Land, 2003). Existing knowledge, no longer as relevant as it was, requires expansion founded on an understanding of new, still emerging, circumstances. Here, liminality – from the Latin term limen (threshold) – describes a pattern of transitions, from separation with “old“ knowledge, to a transitional phase, to the incorporation of new knowledge. In the transitional phase, antecedent phenomena have been left behind but the new has yet to take hold (Turner, 1970). This is to say that pre-defined and measurable learning outcomes need to be expanded through the development of a broader set of skills, enabling learners to be both prepared for situations where existing knowledge may be applied and ready for the unknown – a condition relevant to the hybrid wars (Hokstad & Gundrosen, 2019).

The Hook: Scaling Learning Thresholds

One way of scaling learning thresholds is to expose students to liminal situations where navigating liminality is essential. Indeed, we had already gained some experience with facilitating such learning environments before this study, albeit in another context – the protection of civilians. We learned then that VR was a powerful learning tool, providing new ways of learning for master’s degree students. A simple VR map with visual information about threats seemed to provide increased motivation for learning. However, we also saw limitations in the learning design, most importantly in the lack of agency. Students were too often passive observers of VR content rather than active learning agents (Garcia et al., 2024). Therefore, in mid-2023, the team of researchers, military practitioners, and developers gathered to explore if we could accommodate threshold-scaling in an educational setting related to hybrid war.

We designed and developed a prototype with VR and AI to facilitate a liminal transition among students in the elective course “Full-spectrum War and Societal Consequences“ (later renamed “Hybrid War and Societal Consequences“) at the master’s degree in military studies at the Norwegian Defence University College. The exploration of the application constituted one of many learning activities in May 2024. The elective course is a six-week activity with ten study points, forming part of a master’s degree. The exploration, lasting a single day, was run in week three of six. The application combined VR technologies and an AI companion with an LLM, supported by red teaming and debriefing as problem-based learning methodologies (Balint et al., 2018; Savin-Baden, 2014). We explored the prototype application with seven master’s degree students.

While threshold concepts form the overarching theoretical and educational framework for our approach, the learning design for the test rests on two major ideas: red teaming as problem-based learning (inspired by the tenets of advanced adult learning), and combining an AI companion and a VR environment (see Table 1). The VR environment consists of a fictional map with several information layers, set out in the section below.

The students were tasked with developing a strategy to inflict damage to the “West” without the use of force (most hybrid threats are not kinetic). They were encouraged to take the perspective of the “other”, to better understand how hybrid threats may appear. Red teaming is a well-established methodology in military training, education, and planning (Murray, 2011). It is widely applied in fields such as business simulations, cybersecurity, and risk management (Hoffman, 2017; Zenko, 2015). Its objective is to offer alternative perspectives, challenging conventional thinking with insights that are counterintuitive or provocative. By adopting the mindset of an adversary, red teaming exposes internal weaknesses, vulnerabilities, and external threats. It addresses organizational and cognitive biases like groupthink, mirror imaging, and tunnel vision, which can hinder effective decision making (Janis, 1972). This shift in perspective helps participants understand adversarial motives, goals, and available tools. The method involves role playing the “other,” encouraging participants to question assumptions and develop new viewpoints. This approach enhances situational awareness and promotes better decision-making by broadening understanding.

The advent of AI has presented challenges and affordances to the educational system (Michel-Villarreal et al., 2023). A current trend is to investigate how to design learning paths employing AI-human collaboration (Kshetri, 2023). An AI companion was introduced to this end. The role of the AI companion is to function as a resource to be summoned, if necessary. It is a “guide on the side”; a mentor. The AI companion adds dialogue, authenticity, and the possibility to test one’s own understanding of a situation in real time, selecting and limiting the available information. The AI companion has access to the same information as the students, with the additional capacity to provide information sourced from the internet based on the students’ questions. It functions as an interface between the student and the knowledge domains, the map, and the companion itself.

The VR/AI Application

The application is developed in collaboration with the companies NorthKingdom and Anorak. The application runs on Meta Quest 3 goggles and consists of two core functions. One is a 3D map depicting a fictitious geographical region recalling European countries and a fictitious version of Russia. The others is an AI companion that provides additional knowledge based on a script uploaded to a free-to-use application for customized AI characters developed by the company InWorld (an AI platform allowing users to create AI-powered avatars which can interact in real-time through natural language). The script was text-based, consisting of about 7000 words, describing information about fictitious countries, actors, and events, inspired by the real world and built on insights from peer-reviewed publications on hybrid war. The platform integrates more than 20 LLMs throughout their platform, including Open AI, in addition to text-to-speech and speech-to-text. InWorld offers tools to develop avatars with complex personalities, behaviors, and backstories, making them capable of realistic dialogues and engagement.

The user control of the AI companion is enabled through voice, with the user able to speak to the companion, by touch, with the user able to access interface elements on the map and screens with virtual hands, and a menu, accessible through the motion of the wrist. The LLM draws mostly on information from the fictitious scenario but is capable of accessing the internet, albeit in a limited fashion.

In the application, the user is immersed in a dark environment, where the only visible objects are the AI companion, seated behind a horizontal 3D-map (see Figure 1), and panes with text and image displays. The panes provide instructions and information about the scenario (see Figure 2). The students see their own hands displayed as virtual hand; a clickable UI menu is found on the back of one hand, enabling elements on the map to be activated. The UI menu includes five filters, each inspired by hybrid, real-world, threat events in various societal domains detailed in the book The Weaponization of Everything (Galeotti, 2022). We have included information on criminality, critical infrastructure, financial institutions, and legal aspects in the map – all domains that have been exploited in hybrid wars. Each item on the map displays a pane with text outlining the key features when clicked by the virtual hand. By pressing an info button on the 3D map, the user will be briefed by the companion and can ask “him” about relevant aspects of the country selected. The conversation between the user and the AI is captured on a visible dialogue display in real time, which can be placed anywhere in the virtual environment (VE).

Figure 1

Figure 2

Research Design, Methods, and Ethics

We apply a mixed-methods research design. First, a quasi-experimental setting allows for early testing of the prototype application. The event was structured to generate relevant data related to the research questions while allowing for the unexpected. The researchers participated as observers during the experiment and periodically facilitated the students’ discussion through a collaborative debriefing. One of the researchers, a former officer, held this role throughout the day. Although a senior figure, he approached the student discussion with curiosity and asked open questions to allow for reflections rather than yes or no answers. As such, we played an active part in generating data of interest. Second, the student survey captures the students’ immediate self-reported impressions from their participation in the testing. While seven students are far too few to generate quantitative statistics of relevance for larger populations, the survey helps us understand qualitative variations in this student group. Third, we ran a group interview a few weeks after the experiment to allow for the experience to be internalized. We followed a semi-structured approach, where we asked questions along four predetermined themes, remaining open to off-topic matters and insights not captured by the interview guide.

Master’s degree students at the military college in Norway are similar in age, background, education, and professional experience – and, being uniformed, even in the way they dress. As such, although we have a small group of respondents, they are representative of the population of master’s degree students. They are, however, also individuals, possessed of individual preferences for how they best learn. In addition, hybrid war is a new topic in military education, providing few best practices to lean on. As such, we do not expect to be able to develop a “one-size-fits-all“ solution. Our aim is to explore new learning tools that may prove to be effective alongside more traditional ways of learning.

Our main ethical concern was military student interaction with an LLM that we did not know in detail. We knew InWorld’s solution was based on OpenAI, and we urged our students to be cautious when discussing topics with the AI companion, so as not to disclose personal or sensitive information. Although our narrative is fictitious, it is possible to ask the companion about any topic at all. We solved this through informed consent forms and on-site guidance.

The power of presence and agency in virtual reality

This application uses VR technology accessed through a head-mounted displays (HMD). Three key concepts of learning through VR as set out by Makransky & Petersen (2021. p. 943) are immersion, presence, and agency. Here, “immersion” refers to the degree to which a system can block the real world, making it an objective measure. When VR is experienced through an HMD, the degree of immersion is considered high. “Presence” is a subjective experience often described as the feeling of “being there”, while “agency” indicates “a feeling of generating and controlling actions”.

The Cognitive Affective Model of Immersive Learning (CAMIL) builds on existing research to explain how immersive virtual reality (IVR) can enhance learning outcomes (Makransky & Petersen, 2021). The model highlights presence and agency as the general psychological affordances (specifically, facilitated immersion, control factors, and representational fidelity) that IVR offers. A high degree of immersion provides the students with a feeling of being present in the VE. Control factors and representational fidelity strengthen this feeling by allowing the students a degree of control of the environment. However, CAMIL indicates that while IVR creates an engaging environment, its effectiveness in promoting learning depends not only on these affordances but also on the instructional method in operation.

CAMIL explains how presence and agency influence six cognitive and affective factors leading to better learning. This includes situational interest, intrinsic motivation, cognitive load, and self-regulation. “Situational interest” refers to students’ focus and emotional response to stimuli, with both presence and agency boosting it, enhancing learning through greater engagement. “Intrinsic motivation” denotes the satisfaction students derive from learning that surpasses the desire for external rewards. Both presence and agency increase motivation, fostering persistence and curiosity (Makransky & Petersen, 2021; Ryan & Deci, 2000). Cognitive load (CL) arises when learning tasks exceed the capacity of working memory. Intrinsic CL is influenced by task complexity and expertise, and extraneous CL is shaped by task design. The precepts of CAMIL explain that presence and agency can increase extraneous CL due to VR’s complexity and the freedom to explore content. Self-regulation, the ability to stay focused despite distractions, can be hindered by engaging but demanding IVR lessons. However, high agency can support self-regulation and enhance learning, depending on the lesson design (Boyd et al., 2005; Makransky & Petersen, 2021).

AI literacy for learning

While “literacy” fundamentally refers to basic reading and writing skills grounded in the tradition of printed books (McMillan, 1996), the term is today applied to competences extending beyond reading and writing to a broader range of technological skills (Bawden, 2001; Brown, 1998). This shift has led to specialized literacies such as information literacy (Eisenberg, 2008; Onyancha, 2020) and digital literacy (Lycett, 2020). The development of AI technologies in education have broadened the scope of studying literature, introducing AI literacy as a skill set for interacting with large language models (Grassini, 2024; Ng et al., 2021b). AI technologies require skills of a particular type: while digital technologies cover a wide array of computing tools, AI specifically focuses on replicating human intelligence, a fact bringing unique ethical and moral challenges less emphasized in previous technologies (Siau & Wang, 2020).

Broadly speaking, AI literacy involves the skills and competencies needed to engage with AI in an effective and ethical manner (Long & Magerko, 2020). Several factors influencing AI literacy have been identified (Wang et al., 2023). Computational thinking – the ability to solve problems by thinking like a computer (Wing, 2006) – and efforts to close the digital divide are critical to the development of AI literacy (Celik, 2023). Education researchers have explored AI literacy within formal schooling, examining how AI-related topics are being integrated into modern curricula (Druga et al., 2019; Kandlhofer et al., 2016; Xiao & Bie, 2019). These studies highlight areas like machine learning, data structures, intelligent agents, and algorithms as key aspects of AI literacy (Kandlhofer et al., 2016). Building on research from digital literacy, scholars have developed more comprehensive models for AI literacy.

AI literacy has been defined as a skill set that includes understanding, utilizing, evaluating, and even creating AI technologies (Ng et al., 2021a). Achieving AI literacy requires a deeper understanding of the technologies underlying AI applications and involves grasping key concepts like computational thinking and data structures (Burgsteiner et al., 2016; Kandlhofer et al., 2016). Moreover, promoting practical AI applications encourages “AI thinking” – an approach that combines logical reasoning, problem-solving, and data management skills (How & Hung, 2019; Noh, 2017; Vazhayil et al., 2019). Beyond simply using AI tools, critically evaluating and creatively engaging with AI is crucial. AI literacy fosters inquiry, problem-solving, and community engagement, helping to build a scientifically informed society (Han et al., 2023; Long & Magerko, 2020). Ethical considerations, such as fairness, accountability, and transparency, also play a significant role in discussions around AI literacy as the impact of AI on society continues to expand (Chai et al., 2020; Wang et al., 2023).

We have not found relevant literature on AI literacy in military contexts. While AI literacy is valuable across societal and professional domains, military students will need to understand and use AI both in educational settings and in operations when they return to practice. Future studies should dive deeper in the particularities of AI’s role, limitations, and utility in higher military education.

Insights From Observations, Interviews, and Surveys

The insights rest on our own observations and semi-structured interviews supported by illustrative qualitative data from the survey. The survey data was designed with a 7-point Likert-scale with response options ranging from 1 (“strongly disagree”) to 7 (“strongly agree”). The survey was administered immediately after the experiment. Questions were organized around four main themes: VR; AI literacy (including questions on attitudes and trust); perceived learning; and understanding of hybrid war. Items measuring attitudes to AI were taken from the AIAS-4 questionnaire (Grassini, 2023), while the questions on trust were taken from the TXAI (Perrig et al., 2023). Items relating to AI literacy were taken from the AI literacy Scale (Wang et al., 2023). Tables 2, 3, 4, 5, 6, 7, 8, 9 below present bar charts relating to students’ response patterns; being merely illustrative, they should not be interpreted as statistical data.

Table 2

Table 3

Table 4

Table 5

Table 6

Table 7

Table 8

Table 9

Our informants were all male master’s degree students of similar age with experience from more than two decades of military service in various functions (see Table 2).

Presence

VR’s ability to create a sense of presence is widely held to be its principal added value, both for entertainment and learning. As such, assessing this capacity is, perhaps, kicking a door that is already open. But we consider this as a minimum demand for a VR application; at best, the absence of a developed sense of presence signifies little added value of learning in VR. Following the precepts of CAMIL, technical factors of immersion, control, and representational fidelity play a crucial role in evoking a sense of presence.

In support of CAMIL, several students did experience a feeling of being present in the VE. According to one student, “once you put on the goggles, you are in another world. It helps to maintain focus because all other stimuli disappear” (Student 7). Students emphasize the importance of seeing and hearing – along with verbal communication with the AI companion – as positive factors contributing to presence. According to another student: “I have read a thousand books and do not remember a thing! It is much more difficult to not be present when you see and hear things” (Student 1). Several of the students also highlight the possibility of asking the AI companion open questions as a positive factor. This indicates that technical factors, in this case the combination of VR and AI, contribute to a feeling of presence.

Nevertheless, two students observed that the application could have been developed differently to create a greater sense of presence: the possibility to point or draw things in the VE, pointing towards agency, discussed below. One aspect that negatively influenced presence was the friction in the interaction with the AI companion. While verbal communication was one of the mechanisms that led to a high sense of presence, according to many students, the interaction (“clicking”) with text boxes was not engaging.

Moreover, the AI companion exhibited strange behaviors. Most annoyingly, it struggled to understand questions, or responded to the questions of other students altogether. Another aspect with a negative influence on presence is that spending time in the VE is physically tiring, especially for the eyes. Indeed, reading text in the text boxes, it was reported, strained the eyes.

Weak connections between the map and the AI companion, beyond the thematic focus on hybrid war, reduced the sense of presence. According to some students, digital visualizations in the map (the main river rising when the AI companion talked about waterways, for example) would have increased the sense of presence.

For the most part, the students noted a sensation of presence in the VE, but technical issues in interaction and dialogue with the AI companion and the map hampered the experience for many. The few students that showed a high degree of AI literacy had fewer issues with the environment and spent most of the time in the VE extracting information from the map and in deeper dialogue with the AI companion, as intended in the learning design.

The survey supports these findings, as shown in Table 3. The students reported, on average, that they felt present in the VE, and that it responded to their actions. But they felt their interaction in the VE induced a significant sense of exhaustion.

Agency

Agency – the possibility to “do stuff” – is a core tenet of learning in VR, as underlined in CAMIL, in project-based learning (PBL), and simulation approaches. As VR for learning draws on its legacy from the entertainment and gaming world, players’ actions and interactions with the VE create unique experiences that “stick”. Our prototype includes a 3D map and an AI companion which the students can interact with verbally. Unlike other VR experiences with fixed narratives that constrain user interaction, this affords students the opportunity to generate and control their own actions.

According to the feedback, it appears that friction in the interaction with the AI companion impeded the development of a sense of agency on the part of the students. Many emphasized that the AI companion only offered limited contributions to the experience, and for some students the answers were perceived to be scripted. The AI companion is described as “dumb”, and several students observed that it became “less and less intelligent” throughout the learning experience. One of the reasons for this “dumbing-down” is that the fictional scenario underpinning the LLM is limited, providing few links between various thematic areas.

Several students experienced severe language challenges in their dialogue with the AI companion. It did not always understand the students’ questions, often answering “I do not have any information about that”, or hallucinating an answer to another question altogether. According to one student, the friction in the interaction with the AI companion made it “impossible to have a lively dialogue” (Student 5). Another felt that he did not control his own learning, primarily because he experienced the application to be too pre-programmed.

Conversely, some students enjoyed this way of communicating, being able to have a dialogue with the AI companion and dig into topics they were curious about. The verbal communication and the ability to ask any question were mentioned as engaging. When the AI worked well, the ability to follow up with questions was motivating, as “it was just a normal dialogue”, according to one student (Student 7). Many emphasized that this form of active learning was engaging compared to reading, although learning outcomes related to the topic were harder to identify. During much of the experience, students generated ideas on how to improve the application. This is treated in detail below. In relation to agency, meanwhile, it was mentioned by one student that “soldiers are often practically orientated and would greatly benefit from using VR to understand new threats” (Student 1).

The survey was slightly more positive on agency (see Table 3 above). With the exception of a single student, who gave a low score, participants reported that the VE responded reasonably well. Overall, however, the students state that agency must be enhanced if a sense of presence and their motivation are to be improved. The current prototype allows for only marginal levels of agency, although the most AI literate of the students had few problems navigating the map and extracting information through text and dialogue with the AI companion.

Cognitive load

The CAMIL model set out by Makransky and Petersen (2021) addresses different cognitive and affective factors that can influence learning outcomes. We included four: cognitive load (CL), self-regulation, situational interest, and intrinsic motivation. Due to limited data on the latter three, they are combined into a single section; cognitive load is discussed separately.

Being present in a VE where taking notes is cumbersome due to a surfeit of sensory input, it is likely that one’s brain is quickly clogged with too much information. Again, according to CAMIL, if the information to be processed exceeds the limited capacity of the working memory, the students may experience CL, both intrinsic and extraneous.

Intrinsic CL is influenced by the number of elements in the application; these must be processed simultaneously in the working memory. According to students, the information presented to them in the application was over-finite: “I think the amount of information was limited. It was so limited that I did not benefit from it” (Student 5). This indicates that they did not experience intrinsic CL.

Conversely, extraneous CL is dependent on the design of the learning task and how information is presented to the students. In our case, the friction in the interaction with the AI companion seems to have caused extraneous CL. In other words, the application does not provide too much information; rather, the opposite – the information is presented in a way that makes learning somewhat cumbersome. Some students provided insights that may indicate that they experienced extraneous CL. They missed the opportunity to draw and point in the VE and an introductory brief to the learning experience, without which the learning process lost a degree of structure. The tools that were there to counter potential CL – the text boxes in particular – were both boring to look at and tiring to read, according to several students: the text was considered too distant and was written in capital letters, which was straining to the eyes.

Students spent time discussing the situation in the region, as every student was given somewhat different information. While this process was tiring for many students, a degree of friction – “fighting” to obtain one’s understanding of the situation and to assess it against the understanding of one’s fellow students – serves the process of learning. It was difficult for the students to switch between macro and micro levels of analysis, even though the application allows for such variation in analytical angles. Although the scenario seems quite easy to understand, some elements are similar to the real world and some are fictitious, leading to some students struggling to grasp the strategic picture. In addition, some of the basic info buttons did not work, creating confusion about the big picture.

Finally, some students commented that learning via this application is premised on topical pre-knowledge. As these students had studied hybrid war for two weeks before the test, they were well prepared to understand the purpose, and they were well equipped to avoid cognitive load (CL). One student mentioned that “without knowledge attained earlier in the elective, we would not have a clue what to ask the AI companion about” (Student 3). Consequently, a certain level of pre-learning appears essential for benefiting from this learning design, both to avoid CL and to structure learning in a way that aligns with existing conceptions of hybrid war. This observation also points to the need for mentoring, to learn how to learn from VR/AI companions. Teachers are not made obsolete by this design; rather, they play a crucial role in facilitating pre-learning necessary for students to fully benefit from the learning design.

Overall, it seems like the application was too shallow to create intrinsic CL while simultaneously providing too few tools for managing information in ways that could mitigate extraneous cognitive load. We will return to ideas on how to solve this.

Self-regulated learning, situational interest, and intrinsic motivation

Self-regulation in this context refers to “the ability to manage one’s behavior, so as to withstand impulses, maintain focus, and undertake tasks, even if there are other more enticing alternatives available” (Boyd et al., 2005). As we mentioned above, some students were more able than others to interact with the VE and talk to the AI companion. As one student commented: “I just ended up having a conversation with the AI companion” (Student 7). This indicates a variability in the students’ capacities to regulate their own learning and that some students, finding the AI companion alienating, found it difficult to regulate their behavior towards it. It is true that this application, without a great deal of guiding scaffolding, requires a lot from the student and does not easily facilitate self-regulation. We believe the ability of some students to overcome the friction experienced by many others can best be explained by high levels of AI literacy rather than self-regulation.

According to CAMIL, situational interest is about focused attention and affective response towards stimuli in the VE. While the students clearly displayed an interest in both the technology and the subject matter, we note a particular interested in the former. This may indicate that the application, despite all its flaws, sparked a situational interest in them, perhaps on the grounds of its novelty. Given the serious issues several students noted in their interaction with the AI companion, a negative impact on their situational interest might be expected – but, counterintuitively perhaps, this does not seem to be the case; in fact, the students were very positive towards the application, if largely towards a future iteration of the application or the possibility of bringing VR/AI-technologies into practice in the military profession. As such, it seems that if we failed to permit deeper learning about hybrid war, students became aware of other ways to learn about their profession. This meta-learning perspective, although difficult to plan for, may offer a valuable contribution to modern learning designs.

Finally, students who “engage in a learning activity for the inherent satisfaction it offers, rather than an external award, are driven by intrinsic motivation” (Ryan & Deci, 2000). We have shown above that all our students are true optimists, even when engaging with a crude prototype, pointing to a high degree of motivation to engage with modern technology. We have also shown that some display more interest in technologies than others and seem more literate in their meeting with the application. Again, these observations may point to a high degree of intrinsic motivation, regardless of the type of application.

AI literacy

The integration of the AI companion is intended to engage students on a complex topic. Students reported that interacting with the AI companion increased their sense of presence and agency within the VE. The possibility of dialogue with the AI companion and diving deeper into areas of interest was reported as particularly engaging. This aligns with findings that conversational agents can enhance learner engagement through personalized interactions (Johnson & Lester, 2016). However, this also highlights the critical role of AI literacy in optimizing the learning experience. For students to effectively make the best use of the AI companion’s potential, they must possess the skills to evaluate its outputs, recognize limitations, and frame questions in ways that yield meaningful responses.

Despite acknowledging its potential, students report that the AI companion showed limitations. The database of the AI companion was deemed too shallow, failing to make obvious connections between different societal domains – something essential for understanding hybrid warfare. Moreover, the AI companion lacked transparency. Students were unable to verify the information due to the absence of references. This gap led to skepticism about the companion’s reliability. This points back to the importance of AI literacy, allowing the students to critically evaluate the potential pitfalls of AI-generated information (Long & Magerko, 2020; Ng et al., 2021b). We believe this to be our most significant finding. Our students were indeed able to evaluate the output from the AI companion, despite only having met “him” recently.

The AI also provided answers that were confusing or irrelevant, even if syntactically accurate – a phenomenon often referred to as “hallucination” (Bender et al., 2021; Salvagno et al., 2023). This caused frustration and resistance among students. These issues were reported to disrupt the flow of learning and to diminish trust in the technology, significantly reducing its effectiveness. In this context, AI literacy emerges as a crucial skill, enabling students to identify potential AI hallucinations and mitigate their impact on the learning process.

The students reported a high degree of AI literacy, as indicated in Table 4. Overall, they felt confident in their ability to use and assess various AI-tools, if somewhat uncertain of their own ability to choose the right AI tool for a particular task.

Most informants – apart from Student 1 – held a positive attitude towards what AI can do for them at work and in life, and consider the technology to be on the whole advantageous for humanity (see Table 5).

Nevertheless, as Table 6 indicates, our students do not trust the AI companion, finding it unpredictable and unreliable. They remain wary of it, and skeptical of its ability to perform tasks. Students are somewhat more positive of its potential for decision making.

Learning hybrid war

The survey on understanding hybrid war is more varied, as we can see from Table 7. The application was not effective in helping students understand theories and concepts related to hybrid war. As mentioned in the introduction, this may be an overly optimistic goal given the complexity of the topic and the lack of clear terminology and definitions. Students did not think the learning will make them more proficient in their analysis of hybrid war. Nor were they confident that the application added to their ability to explain the topic to others. As such, the application may fall short of its most important task – but it is important to recall that this is an early prototype. In the current iteration of the prototype, we have improved many of the technical elements that were criticized by the students, and we have already seen improvements in learning outcomes.

Students reported that they understood the red-teaming task that formed the pedagogical framework for exploring the application (see Table 8). While they performed the analysis and discussions based on the task we provided, the experiment never allowed for the planned presentation of the strategy due to time constraints. Consequently, we do not have a complete picture of the results. Students were less convinced of the learning value of the dialogue with the AI companion – and it should be noted that the experiment suffered significant technical difficulties in the first two hours, which probably negatively skewed the result. The students reported the map to have been more meaningful for learning, while remaining skeptical about the value of this learning for real life contexts.

While our students remained highly positive about learning through the combination of VR and AI, the application was less effective in developing students’ capacity to analyze, understand, and communicate matters concerned with hybrid war. Notwithstanding our observation that technical difficulties have skewed the experiment’s results, we also observed the AI companion to have been too crude to permit deeper discussions about the scenario and the topic, even if follow-up study in the weeks after the experiment during the elective course showed that the students had indeed gained knowledge, skills, and competencies about hybrid war.

Pedagogical-technological optimism

In combination, the technical issues and the practical setup of the experiment should have negatively influenced the students’ attitude towards the application. Indeed, some students did portray a high degree of frustration. Surprisingly, however, most of them remained motivated and optimistic towards this way of learning throughout, and even after, the experiment.

Despite being exhausted, the students remained highly positive about using this type of technology in their education (see Table 9).

Overall, the students wanted more of everything and for everything to be a little simpler to understand: more details, more connections, and more clarity. Very few asked for more complexity or more time to think – a finding underlining an inherent preference for immediate solutions over ponderous processes of thought (Kahneman, 2013). Ideally, tools such as this prototype should be able to motivate students in higher education to take the “longer road”.

When asked about their learning experience, the students frequently discussed what could have been better, rather than reflecting on the actual learning experience. This suggests that students are highly optimistic about the technology and its potential as an effective learning tool, both for educational purposes and for real-life use in the military profession. Thus, we include some of these ideas, to demonstrate the types of ideas that emerged. This kind of meta-learning can be valuable both for innovation and for self-reflection: consideration of how one learns about complex issues and how that process can be supported.

Primarily, students wanted more content in the application and more opportunities to interact with the VE. The map should have been more detailed with more information (topography and visualizations of information layers, for example). Additionally, they wanted to be able to see or experience possible futures: visualization of what may happen in one societal domain because of something that happens in another. Visualization is much preferred over text. Furthermore, they had ideas about the ability to zoom in and out between levels and layers on the map. Concerning the information layers, they wanted the ability to “travel” to one domain on the map and experience a scenario, or talk to people in a specific domain.

There were also concrete ideas about how the AI companion should be improved. First, students believe it should be more intelligent. According to Student 7, “the application will gain learning value the moment it is possible to have a lively dialogue.” Students also indicated that they wanted to know where the AI companion was sourcing its information. The LLM draws from a fictional scenario, and the students expressed a wish to validate the accuracy of the answers they got, which points to a high degree of AI literacy. Moreover, they wanted a clear connection between the AI companion and the map. Other ideas included the ability of the AI companion to provoke and to test assumptions about hybrid war, not simply to act as a guide throughout the learning experience.

Students also offered ideas about how the learning experience could be better designed. One of these was the inclusion of an introductory brief, permitting an overview of the scenario and suggesting the direction of the learning to follow. In addition, they felt that there was a lack of opportunity to take notes and sort the information inside the VE, which we treated as part of the CL analysis. The ability to filter information to adjust the complexity level was also mentioned as an idea for further development. This feature would be especially useful if the current prototype contained more information.

Findings from the survey suggest that the technology rather than the content were considered the most rewarding. Students were positive towards the opportunity to learn about emerging technology and to become aware of its future potential. As such, despite many weaknesses, being engaged with the application permitted open-ended reflection on the future of learning and military practices.