From Competency Models to Targeted Interventions: Fostering AI Competence as a Future Skill with AICompAss in Heterogeneous Teams – a Design-Based Approach

Future skills and organizational learning

The discussion on future competences has gained increasing relevance in recent years, both in educational policy and in academic research. The growing dynamics of social, technological, and ecological transformation processes drive this development, generating new competence requirements. The ability to deal productively with uncertainty, complexity, and discontinuity is regarded as one of the central challenges of the 21st century (Bundekanzleramt Österreich, 2024, p. 7; Brandhofer et al., 2024, p. 5; Vuorikari et al., 2022, p. 1).

In this context, future skills refer to those competences that go beyond disciplinary knowledge and enable individuals to act in complex and dynamic situations in a self-organized, responsible, and creative manner. They are characterized by their multidimensionality, value orientation, and processual nature, combining cognitive, motivational, and social resources into an integrated competence structure (Ehlers, 2020, pp. 44–53).

While future skills are often discussed at the individual level, it is becoming increasingly clear that their sustainable development is closely linked to processes of organizational learning. Competences do not emerge in isolation but within social, structural, and cultural frameworks that enable, support, and guide learning. Organizations therefore assume an active role as learning systems: they create spaces for experience, exchange, and reflection and promote collective knowledge generation, thus becoming key enabling structures for competence development (Ehlers, 2020, pp. 16–26, 159–163; Kauffeld & Rothenbusch, 2023, p. VII, 10).

Organizational learning represents the key mechanism for transforming individual learning processes into collective developmental capacity. It involves the continuous adaptation of routines, decision premises, and structures to changing environmental conditions. Learning organizations are characterized by their systematic evaluation of experiences, the sharing of knowledge, and the derivation of innovation-oriented patterns of action. Team learning, workplace-based learning processes, and communities of practice serve as connecting elements between individual competence and organizational performance (Kauffeld & Rothenbusch, 2023, pp. 7, 200; Rampelt et al., 2025, pp. 9–13; Tippelt & Hippel, 2018, p. 617).

This relationship gains particular significance in the context of digital transformation. Automation, data-based decision-making processes, and artificial intelligence are fundamentally changing the nature of work and learning (Heil et al., 2024, p. 51). Organizations must not only manage technological adjustments but also develop the capability to support their employees in building digital and reflective competencies. The acquisition of future skills thus becomes a collective process in which institutional learning cultures, participatory structures, and continuous professionalization play central roles (Wirtschaftskammer Österreich & Zukunftsinstitut GmbH, 2024, p. 6; Kauffeld & Rothenbusch, 2023, pp. 2, 32). This perspective is also reflected in European competence frameworks such as DigComp (Vuorikari et al., 2022), which conceptualize digital competencies as a key prerequisite for participation and employability. In addition, recent regulatory developments such as the EU AI Act underline the growing importance of AI-related competences in professional and societal contexts.

Future skills can therefore be understood as an overarching framework encompassing transversal digital competences such as digital, data, or AI literacy. The promotion of these competences is not solely the responsibility of individuals but rather an expression of an organization’s maturity and capacity for learning and development. Organizations that strategically embed learning and view it as a shared developmental principle thereby create the foundation for maintaining agency and shaping innovation sustainably in an environment characterized by uncertainty and change.

AI Competence as a transversal competence

The discussion on future skills makes it clear that the issue is not merely an abstract description of future competence requirements but rather their operationalization within concrete, socially and professionally relevant domains. A particularly salient example of this is the field of artificial intelligence. The ability to engage with AI is gaining importance across almost all professions and societal domains, representing a key competence that goes beyond purely technical abilities. In this sense, AI literacy can be understood as a transversal future skill that integrates knowledge, application, reflection, and ethical responsibility in dealing with AI (Ehlers et al., 2024a, pp. 13, 87; Almatrafi et al., 2024, p. 1; Wirtschaftskammer Österreich & Zukunftsinstitut GmbH, 2024, p. 16; Chiu, 2024, p. 34).

In recent years, the concept of AI literacy has become established accordingly. It refers to the ability to understand artificial intelligence, evaluate it critically, apply it practically, and reflect upon it in a socially responsible way. In professional contexts, this term includes tasks such as evaluating AI-generated outputs, integrating AI tools into existing workflows, and making informed decisions based on AI-supported information. AI literacy thus aligns with related concepts such as digital literacy, data literacy, and media literacy, all of which aim to promote reflective participation in technology-mediated environments (Chan & Colloton, 2024a, p. 28; Cousseran et al., 2023, pp. 15ff).

A review study by Almatrafi et al. (2024) identified six core constructs of AI literacy: Recognize emphasizes the ability to identify AI applications; Know & Understand comprises fundamental knowledge of how AI functions; Use & Apply refers to competent handling of AI tools; Evaluate denotes the ability to critically assess AI outputs; Create concerns the development of original AI-related concepts; and Navigate Ethically highlights reflection on social and ethical implications (Almatrafi et al., 2024, pp. 6–8). These core constructs are particularly relevant because they pragmatically represent the central dimensions required for developing actionable competences in dealing with AI in real-world settings.

Although AI literacy is increasingly recognized, current studies show that corresponding competences remain underdeveloped. Many initiatives still focus on technical knowledge, while reflective and ethical dimensions receive little attention (Rampelt et al., 2025, p. 5; Ehlers et al., 2024a, pp. 7, 83; Almatrafi et al., 2024, p. 16; Brandhofer et al., 2024, p. 89; Almatrafi et al., 2024, p. 9; Brandhofer, 2024, p. 93). Furthermore, organizational training initiatives are still weakly developed and often limited to isolated workshops without fostering systematic competence development. This lack of structured competence development is particularly critical in professional environments, where employees are increasingly required to use AI systems responsibly and effectively in their daily work (Rampelt et al., 2025, p. 9; Brandhofer, 2024, p. 93; Laupichler et al., 2022, pp. 10–12).

Ehlers et al. (2024b) have introduced the AIComp model, a competence framework for AI literacy based on the future skills approach, which systematically integrates the core constructs identified by Almatrafi et al. (2024). Recognize and Know & Understand are embedded in critical digital competence, Use & Apply in action-oriented domains such as implementation competence, Evaluate in critical thinking and decision-making competence, Create in creative problem-solving competence, and Navigate Ethically in ethical competence. AIComp thus connects the breadth of the core constructs with an empirically validated competence structure and provides a basis for describing AI-related competences in a way that is applicable to professional practice.

AIComp and AICompAss

Ehlers et al. (2024a, 2024b) developed the AIComp model (Artificial Intelligence Competence Model) and described it in two research reports published in 2024. Developed in a German-speaking research context, the model reflects current discourse on future skills in this region while addressing challenges that are equally relevant across international professional settings. It aims to conceptualize AI competence not merely as technical knowledge or application skills but as a multidimensional construct that integrates cognitive, social, motivational, and value-based dimensions.

In contrast to broader competence frameworks such as DigComp or qualification-oriented models like the European Qualifications Framework (EQF), which provide general reference structures for digital and professional competences, AIComp offers a domain-specific operationalization of AI-related competences. In particular, its emphasis on action-oriented, reflective, and ethically grounded dimensions enables a more differentiated understanding of competence development in the context of artificial intelligence. AIComp thus directly builds upon Ehlers’ (2020) understanding of future skills.

The model differentiates AI competence into twelve competence fields assigned to three overarching domains:

• Work-related competences (activity and implementation competence, system design competence, creative problem-solving competence, critical digital competence),

• Personal development (decision-making competence, self-efficacy, critical thinking, active self-regulation, autonomy), and

• Social and organizational competencies (ethical competence, cooperation competence, communication competence).

This tripartite structure highlights that AI competence cannot be reduced to technical aspects but depends on the integrative interplay of multiple dimensions (Ehlers et al., 2024b, pp. 7–8).

With the validation of the AIComp model, a structural framework is now available. Ehlers et al. (2024a, p. 90) emphasize that the model provides approaches for the creation of training and qualification opportunities. The aim is not only to describe the twelve fields but also to differentiate them into successive levels that make learning and development processes visible. This reflects the underlying future skills logic: competence is dynamic, developable, and expressed in varying degrees.

Against this background, the AICompAss (AI Competence Assessment) self-assessment instrument was developed. It translates the abstract dimensions of the AIComp model into action-oriented scenarios that allow for realistic self-assessment.

Existing assessments for AI literacy proved insufficient for heterogeneous professional contexts, as they were often either too knowledge-based or too generic in their conceptualization. The AI Literacy Test (Hornberger et al., 2023), for example, focuses on the objective assessment of cognitive knowledge using performance-based items, thereby primarily capturing conceptual understanding. In contrast, the MAILS scale (Carolus et al., 2023) adopts a self-report approach and emphasizes metacognitive, motivational, and self-regulatory dimensions of AI literacy. Similarly, the SNAIL scale (Laupichler et al., 2023) provides a differentiated assessment for non-experts, combining self-perceived knowledge, critical evaluation, and practical application.

While these instruments offer valuable contributions and are empirically well-founded, they remain limited in capturing competence as situated, action-oriented performance in context-specific scenarios. In particular, their alignment with competence development approaches based on the future skills paradigm is limited.

AICompAss, by contrast, was explicitly designed to assess competence as the ability to act in realistic, practice-oriented situations. Grounded in the multidimensional structure of the AIComp model, it translates abstract competence dimensions into scenario-based tasks that reflect professional challenges. This enables a closer alignment between competence assessment and the requirements of professional practice, particularly in heterogeneous, non-technical environments.

AICompAss was deliberately designed as a scenario-based instrument. Participants engage with short situations that describe typical challenges in dealing with AI. In this way, competence is not captured as knowledge but as the ability to act. The scenarios are formulated in a general, accessible way, avoid technical jargon, and are transferable to multiple contexts, particularly within higher education administration and non-technical professional environments.

The assessment uses a concealed five-point scale that reflects degrees of agency in action:

1. ‘I do not feel able to perform the task’.

2. ‘I can do it with guidance or by following a checklist’.

3. ‘I can do it independently under familiar conditions’.

4. ‘I can do it independently in a new but similar context’.

5. ‘I can do it independently across different contexts and can explain and further develop it’.

Each of the 12 competence fields contains six scenarios, resulting in a total score range of 6 to 30 points per field. Based on these point ranges, results are categorized into four competence levels:

1. Beginner – possesses initial basic knowledge and experiences high uncertainty (6–12 points).

2. Advanced beginner – can act independently in familiar contexts but remains somewhat insecure (13–18 points).

3. Competent – applies skills reflectively in new contexts with growing confidence (19–24 points).

4. Expert – uses AI in a confident, strategic, and flexible manner, including further development and guidance of others (25–30 points).

These levels build on the further development of the structural model into a competence level model and make learning progressions visible. They are not intended for normative evaluation but serve as an orientation framework for individual and organizational development.

A distinctive feature of AICompAss is its dual function: it provides diagnostic data on competence profiles while simultaneously fostering reflection through its scenarios. Participants engage with application situations and reflect on their possible courses of action. Thus, AICompAss serves not only as an assessment tool but also as a learning stimulus.

Within the scientific discourse, AICompAss can be situated in relation to existing instruments that assess different dimensions of AI literacy. While the AI Literacy Test (Hornberger et al., 2023), MAILS (Carolus et al., 2023), and SNAIL (Laupichler et al., 2023) each contribute valuable perspectives ranging from knowledge-based assessment to self-reported and non-expert-oriented approaches, AICompAss complements these by focusing explicitly on scenario-based, action-oriented competence.

Rather than replacing existing instruments, it extends the assessment landscape by providing a tool that is specifically aligned with competence development in professional contexts. This creates particular value for training design, organizational development, and the practical implementation of AI literacy as a future skill.

This approach is particularly relevant in heterogeneous higher education teams operating in the Third Space. Here, employees with diverse professional backgrounds work together, often without specific technical expertise but with increasing responsibility for handling AI systems. AICompAss makes these differences visible, enables the creation of team profiles, and provides a foundation for tailored measures. Beyond individual benefits, this also generates organizational value: the results can be integrated into staff development and training strategies, offering an empirical basis for the targeted advancement of future skills within higher education institutions.

In summary, AICompAss combines the theoretical foundation provided by the AIComp model with a practical, application-oriented operationalization. It addresses a gap in both research and practice by enabling the assessment of AI competence as situated performance in professional contexts. Moreover, the results can be directly integrated into human resource development and quality strategies, supporting the systematic and sustainable promotion of AI competence as a future skill within organizational learning processes.

Development of the action scenarios

The development of the action scenarios was grounded in the twelve competence fields of the AIComp model (Ehlers et al., 2024b, pp. 7–8), which conceptualize AI competence as a multidimensional and action-oriented construct. To operationalize these competence fields, learning outcomes were formulated based on the revised Bloom’s taxonomy (Anderson & Krathwohl, 2001).

However, in contrast to the original taxonomy, the levels were consolidated into three broader categories—Knowledge & Understanding, Application & Analysis, and Evaluation & Creation—to better reflect the integrated and practice-oriented nature of AI competence. This consolidation allows for a more holistic representation of competence as the ability to act in complex situations, rather than as isolated cognitive processes.

Building on these learning outcomes, scenario-based items were developed to capture competence in contextually embedded action situations. This approach was deliberately chosen instead of traditional knowledge-based or decontextualized survey items. Scenario-based assessment enables the representation of authentic decision-making situations and supports the measurement of perceived action capability in realistic contexts. It thus aligns with the underlying competence model, which emphasizes agency and the ability to act under conditions of uncertainty.

To ensure consistent formulation and alignment with each competence field, the learning outcomes were operationalized using a custom-designed chatbot (Brandstetter & Hochfellner, 2025) under the human-in-the-loop principle and refined through iterative feedback cycles. The resulting scenarios were originally developed in German to ensure linguistic accessibility and contextual relevance for participants from non-technical backgrounds. Consequently, three learning outcomes were generated for each competence field, each serving as the basis for two scenarios (see Table 1). In total, this resulted in 72 action scenarios (12 competence fields × 3 learning outcomes × 2 scenarios).

The scenarios were designed to reflect typical challenges in dealing with AI without using technical jargon or domain-specific terminology. This makes them transferable across different organizational contexts and enables a realistic, action-oriented self-assessment. The complete set of German-language scenarios used in the AICompAss instrument can be provided as supplementary material upon request.

Beyond mere measurement, the scenarios also fulfill a didactic function: they stimulate reflection on action strategies and make implicit assumptions visible. Participants are prompted to make decisions within a hypothetical yet plausible context, thereby initiating a transfer between abstract knowledge and concrete action. This design combines diagnostics with learning opportunities and thus follows the principle of formative competence assessment.

While the scenarios are designed to support reflective engagement, responses are systematically captured using a structured rating scale to ensure comparability across participants. The focus of the assessment lies on perceived action capability in context rather than on factual knowledge alone.

Validation through expert panel

Validation was conducted by an interdisciplinary expert panel (n = 10) comprising professionals from higher education teaching, AI strategy, media didactics, and competence development. The panel was deliberately composed heterogeneously to incorporate technical, didactic, and organizational perspectives. The validation process involved several steps. First, each item was reviewed individually with a focus on clarity, relevance, and alignment with the competence fields. In addition, the Item Content Validity Index (I-CVI) and the Scale Content Validity Index (S-CVI) were applied to systematically ensure content validity (Yusoff, 2019). The combination of qualitative feedback (e.g., linguistic precision, conceptual adequacy of scenarios, clarity of response scales) and quantitative indicators resulted in a consistent and practically relevant instrument foundation. An additional methodological advantage was that repeated rounds of consultation allowed multiple perspectives to be incorporated, enabling continuous refinement. This ensured that AICompAss is both conceptually robust and applicable in everyday organizational practice.

Psychometric foundation and methodological considerations

The evaluation was designed to allow both individual and aggregated competence profiles to be generated. At the item level, responses were converted into point values (a concealed scale ranging from 1 to 5). From these, mean scores were calculated for each competence field, enabling a differentiated representation of proficiency levels along the AIComp model. The results were classified into four competence levels: Beginner, Advanced Beginner, Competent, and Expert. In addition, aggregated profiles were created at the team level to make collective patterns visible. Visualizations such as radar charts were used to illustrate the results. Furthermore, the calculation of the AI Activity Index (KIX), as described by Ehlers et al. (2024a), was included to provide an overall perspective.

The analysis focuses on perceived competence profiles, which are particularly suitable for identifying development needs in organizational contexts and early-stage competence development.

In addition to descriptive evaluation, psychometric procedures such as reliability analyses (e.g., Cronbach’s alpha) and validity testing are beyond the scope of this exploratory study but represent a key focus of subsequent research. Cronbach’s alpha was selected to assess the internal consistency of the competence fields and to enable comparability with established self-assessment instruments. Moreover, factor analyses are conceivable to empirically examine the structure of the competence fields. Content validity has already been ensured through the expert panel and the CVI calculations.

Special methodological attention is required due to the scenario-based nature of the instrument: since participants are not presented with knowledge-based items but with complex action situations, the testing of validity and reliability constitutes an innovative yet challenging process. At the same time, this design opens up opportunities for further research, for example, examining the extent to which scenario-based self-assessments correlate with actual performance in practical settings. Future studies will aim to triangulate self-assessment data with behavioral or performance-based observations to further enhance construct validity.

Pilot implementation in an organisational context

The pilot implementation of the instrument was conducted within a Third-Space department at an Austrian university of applied sciences. In total, the department comprised eleven employees; excluding the department head and the author, nine potential participants remained for the study. This setting was characterized by a high degree of heterogeneity: employees possessed varying levels of prior experience, areas of application, and attitudes toward AI. Consequently, the sample provided a suitable basis for testing the instrument’s practical applicability in a non-technical but increasingly AI-related context.

Recruitment took place through an internal invitation, and data collection was carried out online using a standardized questionnaire containing the scenario-based items. Prior to participation, all participants were informed about the study’s purpose, procedure, and anonymity. Participation was voluntary, and withdrawal was possible at any time. This procedure ensured that the data collection process was ethically sound and formative in nature.

In addition, contextual information, such as professional role, prior use of AI tools, and personal attitude toward digitalization, was collected to facilitate interpretation of the results and to analytically capture the heterogeneity of the sample.

Due to the small and context-specific sample, the findings are exploratory in nature and do not allow for generalizable conclusions.

Ethical framework

Particular attention was given to the ethical framework of the study. Participation was entirely voluntary and not associated with any form of coercion or pressure. All data were anonymized and used exclusively for research purposes. The survey was designed so that the results could not be used for individual performance evaluation but solely for competence development. This approach follows the principle of formative diagnostics, where the aim is not to identify deficits but to make development opportunities visible. Furthermore, deliberate anonymization created an environment that encouraged honest self-assessments and minimized potential effects of social desirability.

According to institutional guidelines, formal ethics approval was not required for this type of study. The study was conducted with the prior knowledge and approval of the department head.

Overall, the methodological approach demonstrates how the AIComp model was operationalized through AICompAss into a practical instrument for assessing AI competence. The design-based research approach enabled close integration of theory, development, and validation. The pilot implementation served to examine the applicability of the instrument within an organizational context and to establish a foundation for further studies with larger samples, more in-depth psychometric testing, and potential performance comparisons.

Individual and aggregated competence profiles

The data collection was conducted in spring 2025 with eight participants working in a department characterized by a Third-Space structure. The response rate of 89% (n = 8) indicates a high level of willingness to engage with the topic. The average completion time across the entire tool was 43 minutes, excluding a few upper outliers. Differences in duration across the competence fields showed that scenarios with reflective and ethical content generally required more time than practice-oriented tasks. This finding suggests that participants spent more time addressing issues related to responsibility, bias, and control than engaging with application-oriented scenarios. It also indicates that dealing with the critical and ethical dimensions of AI requires greater cognitive effort and is more strongly linked to individual values and attitudes. Thus, AICompAss appears to systematically capture not only technical but also ethical and reflective dimensions of AI competence.

The AI Activity Index (KIX), as conceptualized by Ehlers et al. (2024a, p. 57–59), provides a structured approach to capturing AI-related experience by combining two dimensions: frequency of use and type of engagement. The index differentiates between passive, active, and co-creative forms of AI use and relates these to usage frequency (sometimes, regularly, very frequently), resulting in a composite score ranging from 0 to 6 (see Figure 1). Based on this scoring, participants can be grouped into low (2), moderate (3–4), and high (5–6) KIX levels, while non-users are assigned a score of 0.

Figure 1

In this study, the KIX was used as a contextual indicator of participants’ prior experience with AI. The distribution indicated two participants with high, four with moderate, and two with low KIX values: a small group with a high degree of routine in dealing with AI, a majority with an intermediate level of experience, and a few individuals with very limited user experience (see Figure 1). This index proved valuable for examining and interpreting individual competence profiles in light of actual usage practices. At the same time, it clearly demonstrates that usage intensity alone does not provide a sufficient explanation for competence levels; rather, it must be differentiated through scenario-based assessment. This finding indicates that frequency of use alone does not automatically translate into higher competence in professional practice. Rather, the ability to apply AI in a reflective and context-appropriate manner appears to be a distinct competence dimension.

The subsequent analysis of individual profiles illustrates how these KIX-based starting points are reflected in specific competence patterns. The instrument’s added value becomes evident, as it reveals differences that self-assessment using the KIX alone could hardly capture.

Two contrasting cases can be highlighted as examples. One participant with a high KIX achieved levels ranging from Competent to Expert across several competence fields, with particularly strong scores in activity and implementation competence, as well as cooperation competence. This already indicates a high degree of confidence in action. In contrast, a participant with a low KIX remained at the entry level or only slightly above in all twelve competence fields. This, in turn, underlines the correlation between application experience and perceived confidence (see Figure 2). These two extreme cases illustrate, on the one hand, the predictive value of the KIX and, on the other, the existing heterogeneity within small teams as well as the potential of the AICompAss instrument.

Figure 2

In addition, the remaining individual profiles illustrate that there are also mixed forms in which certain competences are well developed, while others appear only weakly pronounced. These observations form a bridge to the subsequent aggregated analysis, which presents the overall distribution across all twelve competence fields. This demonstrates that the contrasting cases are not isolated phenomena but should be interpreted within the broader context of group patterns. A closer examination allows for the identification of both individual developmental trajectories and collective patterns, from which tailored learning pathways can be derived.

When viewed in aggregate, all twelve competence fields showed median values in the range of advanced beginner (see Figure 3). This indicates an overall rather low competence level, even though some individuals clearly exceeded it. From a professional perspective, this suggests that many participants are currently able to engage with AI tools only under familiar conditions but may face challenges when transferring their skills to new or more complex work situations. This is particularly relevant in organizational contexts where AI-related tasks increasingly require flexible and context-sensitive application.

Figure 3

As all scenario responses were rated on the concealed five-point scale (1–5) and converted to summed field scores ranging from 6 to 30, median and interquartile range (IQR) values were examined together to enable a more differentiated interpretation of the results. To interpret these results in a more differentiated way, the interquartile range (IQR) values were examined together with the medians at the individual level. The analysis revealed that in some cases, the degree of dispersion varied considerably. A medium median combined with a low IQR indicates a homogeneous, collectively shared competence level, whereas the same median with a high IQR points to a polarization between more advanced and less experienced individuals (see Figure 4).

Figure 4

Particularly noteworthy were the results for decision-making competence, with an interquartile range (IQR) of 13.0 points, and for activity and implementation competence, with an IQR of 10.5. Such large dispersions indicate strong polarization within the team; some individuals possess relatively high abilities, while others remain in an orientation phase. In a professional context, such disparities may lead to an uneven distribution of AI-related tasks within teams, where more experienced individuals take on more complex or responsibility-intensive activities, while others remain in more passive or supportive roles. In other fields, such as active self-regulation, the dispersion was lower (IQR 5.3), suggesting a homogeneously low starting level. This combined analysis of aggregated competence profiles together with the median and IQR evaluation illustrates that average values alone are insufficient to realistically represent competence distributions within teams.

To provide a more differentiated perspective, a detailed analysis of a single competence field was conducted. Figure 5 presents the distribution of decision-making competence across proficiency levels within the sample as an illustrative example. The results show a heterogeneous distribution, with participants represented across multiple levels of the scale.

Figure 5

This fine-grained analysis complements the aggregated findings by making intra-team differences within a specific competence field visible.

AI competence as a dynamic bundle of action

The results of the pilot implementation demonstrate that AI competence in organisational contexts must not be understood as a static body of knowledge but rather as a dynamic, action-oriented bundle of capabilities. The competence profiles collected through AICompAss reveal clear heterogeneity among participants, which can be attributed to divergent professional roles, varying experiential backgrounds, and differing self-efficacy expectations in dealing with AI. These differences reflect the multi-perspectival reality of organisational learning environments and are made visible in a differentiated way through the scenario-based design of the instrument. At the same time, they highlight the necessity of context-specific development measures. In sum, the findings underscore that AI competence should be conceptualized not as declarative knowledge about AI, but as enacted agency—the capability to act reflectively, responsibly, and creatively in AI-mediated contexts.

It is particularly relevant that the aggregated findings indicate low to medium competence levels in several areas, especially in activity and implementation competence, ethical competence, and active self-regulation. These fields represent central nodes within the AIComp model and are closely interrelated: while activity and implementation competence refer to the ability to transfer AI applications into one’s own work context, ethical competence emphasizes the reflective assessment of responsibility, fairness, and transparency, and active self-regulation denotes the ability to consciously monitor and evaluate AI use (Ehlers et al., 2024b, p. 28).

Compared with the findings of the AI competence study by Ehlers et al. (2024a), a similar pattern emerges, albeit at different levels of self-perception and future projection. In that study, the highest mean values for perceived sovereignty—defined as subjective confidence in action—were observed in instrumental digital competence and learning competence, whereas reflective and cooperative fields such as ethical competence and cooperation competence scored significantly lower (Ehlers et al., 2024a, p. 75). This pattern is mirrored in the present sample: while medium levels were reached in the technical-practical competence fields, the values for reflective and regulatory areas, particularly ethical competence and active self-regulation, tended to be lower.

When considering the future relevance of competences, a clear area of tension becomes apparent. Ehlers et al. (2024a, p. 77) identify decision-making competence and learning competence as the most relevant fields for the future, while ethical competence and critical thinking also exhibit high projected relevance. These findings highlight that precisely those competence areas that currently show low levels of perceived sovereignty are simultaneously regarded as particularly decisive for the future.

The team examined here thus exemplifies in condensed form what is also evident in other datasets: a clear divergence between current confidence in action and anticipated future relevance. Overall, this reflects a general transformation pattern repeatedly described in empirical research on AI competence: the future of work will require not more technical detail knowledge but rather the ability to use AI responsibly, critically, and with a capacity for creative shaping (Cousseran et al., 2023, p. 51; Almatrafi et al., 2024, p. 8; Laupichler et al., 2022, p. 12; Zhang et al., 2025, p. 15).

AICompAss as an operationalization of future skills

The AIComp model by Ehlers et al. (2024b) builds on the Future Skills concept developed by Ehlers (2020) and transfers its underlying assumptions to the domain of artificial intelligence. Within this framework, future skills are defined as learnable, value-based, and developable competencies that enable individuals to act self-organized in dynamic and uncertain contexts. They are not static bodies of knowledge but emergent abilities that evolve through the interaction of knowledge, values, and self-regulation (Ehlers, 2020, p. 57).

AICompAss operationalizes this logic through scenario-based action situations. Participants assess how they would act in typical but context-independent AI-related situations, thereby reflecting on implicit attitudes, values, and decision-making patterns. As a result, competence becomes tangible as experienced agency. In this sense, AICompAss does not represent a conventional testing arrangement but rather a reflective learning impulse that initiates metacognitive engagement with one’s own capacity to act.

Recent reviews and scale analyses show that existing instruments for assessing AI literacy predominantly focus on declarative knowledge, self-perception, and attitudes (Almatrafi et al., 2024, pp. 17f; Lintner, 2024). In most cases, they capture cognitive or perception-based aspects via Likert-type items, while performance-oriented formats—those that examine situational action and contextual transfer—are largely absent. Only a few approaches attempt to depict real action situations (Lintner, 2024, pp. 1, 7).

AICompAss addresses precisely this gap by capturing the extent to which individuals are able to integrate knowledge, values, and motivation in concrete action situations. The instrument thus follows the performative logic inherent in the Future Skills framework. Through its scenario-based structure and concealed scoring mechanism, AICompAss operationalizes this logic empirically by making participants’ learning and reflection processes themselves the subject of measurement. In this way, it contributes to bridging the existing gap between diagnostics and didactics by conceptualizing competence not merely as an object of assessment but as a starting point for learning processes.

Needs analysis and development areas

The needs analysis based on these profiles identified three central clusters:

Taken together, the results depict an overall low baseline level interspersed with individual strengths. A striking feature is the discrepancy between more practice-oriented fields (e.g., implementation competence) and reflective dimensions (e.g., ethics, regulation). While some individuals already act confidently in practical contexts, there is often a lack of ability to critically evaluate and manage AI responsibly. This suggests that developmental measures should follow a dual logic: systematically strengthening foundational and reflective fields while using existing strengths as levers to initiate progress.

Furthermore, the findings indicate that AICompAss functions not only as a diagnostic instrument but also as a basis for strategic human resource development by making developmental priorities visible. In a broader context, AICompAss could thus contribute to the professionalization of university staff and be integrated into long-term development programs.

Competence development as a collective process of organisational learning

Interpreting the results in the context of organizational learning processes reveals that competence development in heterogeneous teams is not only an individual but, above all, a collective achievement. According to Kauffeld and Rothenbusch (2023, pp. 9f), competences do not arise exclusively within individuals but in social interactions supported by shared values, routines, and spaces for reflection. Learning processes are therefore inseparably linked to organizational culture and structure.

AICompAss can thus be understood as a structuring element within organizational learning processes. It creates transparency regarding existing competences and development areas, thereby promoting targeted reflection and collective advancement within the team. When teams discuss their results together, communicative spaces emerge in which learning becomes visible and organizational knowledge can be shared. In this way, the diagnostic assessment itself becomes a didactic intervention.

Such a perspective aligns with Ehlers’s (2020) understanding of future skills as the collective capacity of an organization to reorient itself in a changing environment. Competence development is understood here as an integral component of organizational learning maturity, reflecting an organization’s ability to absorb irritations, share knowledge, and develop new logics of action. Consequently, AICompAss is not merely a tool for measuring individual abilities but an initiator of reflection and learning processes at the organizational level.

Design-based research as a circular learning principle

The design-based research approach proved particularly suitable for this study, as it links theory development, instrument design, and application within an iterative process. It allows scientific models such as AIComp to be transferred into real organizational contexts and empirically tested there before being theoretically reflected upon again (McKenney & Reeves, 2021, p. 82). The research process thus follows the same logic that also underpins the future skills philosophy: knowledge emerges through cyclical feedback between action and reflection (Ehlers, 2020, p. 213).

In this sense, AICompAss is conceived not merely as a diagnostic procedure but as a didactically integrable instrument that initiates and structures learning processes. Repeated application in organizational settings enables continuous development, both of the instrument itself and of the learning culture. The design-based approach thus combines empirical knowledge generation with the practical testing of learning innovations, establishing a reciprocal relationship between research and organizational learning.

This logic is also reflected in the methodological design of AICompAss: as a scenario-based self-assessment with a concealed scoring structure, it stimulates not only evaluation but also self-reflection. The focus on realistic action situations turns data collection itself into a learning opportunity in which knowledge, values, and intentions for action are interwoven. AICompAss therefore functions as a reflexive research instrument that integrates diagnostic and didactic functions, understanding learning and assessment not as separate but as complementary processes.

Practice-oriented measure development: Didactically grounded promotion of AI-related action competence

Based on the competence profiles collected with AICompAss, an exemplary training measure was developed to address the specific developmental needs of a heterogeneous team working in a Third-Space context at an Austrian university of applied sciences. The aggregated results revealed pronounced differences in three competence fields of the AIComp model—activity and implementation competence, ethical competence, and active self-regulation—which were identified as key entry points for fostering AI competence. The objective was to design a training program that, grounded in this empirical baseline, enables action-oriented, reflective, and responsible use of AI and guides participants toward the “Competent” level within the AICompAss framework.

The concept was modeled according to the AI Course Design Planning Framework (Schleiss et al., 2023), which integrates macro- and meso-didactic perspectives and thus allows for context-sensitive implementation within organizational learning processes. This ensures that the training is not perceived as an isolated continuing education activity but as part of a strategic competence development initiative.

The training focuses on three learning domains derived from the AICompAss data and the competence definitions of the AIComp model:

1. Understanding and shaping AI in one’s own context – Participants identify areas of application for generative AI in their work environment, reflect on potential and risk, and develop initial ideas for use. The goal is to translate abstract AI concepts into concrete, work-related actions.

2. Acting ethically and responsibly – Participants apply ethical principles to AI-related scenarios and assess them in terms of fairness, responsibility, and transparency. In particular, topics such as bias, explainability, and trustworthiness are addressed.

3. Actively managing AI use – Participants learn to apply generative AI tools purposefully, critically evaluate their outputs, and develop personal strategies for steering their use. This strengthens the ability to use AI reflectively and autonomously across different contexts.

Didactically, the training follows the phase-oriented structure of the RITA model (Activate Resources – Process Information – Initiate Transfer – Evaluate) by Schubiger (2022) in combination with the 4MAT logic (Why? – What? – How? – What if?) by McCarthy (2000). This structure provides the learning psychology framework to link knowledge acquisition, application, and reflection into a coherent learning process. The central idea is to use activity and implementation competence as an anchoring field from which reflective and regulatory competences, particularly ethical judgment and active self-regulation, are systematically developed.

The training is deliberately practice-oriented: scenario-based tasks, peer discussions, and workplace-related reflections ensure a direct connection to real work processes. In this way, AI literacy is experienced not as technical knowledge but as reflective, responsibility-oriented action competence that emerges and becomes effective within organizational learning contexts.

In addition to the formal training concept, the results also revealed complementary opportunities for competence development. Particularly in fields with high variance, such as decision-making competence, mentoring formats appear suitable: individuals with higher levels of competence can deliberately share their experiences, thereby supporting the stabilization of collective learning processes (see Figure 5). This corresponds to the principle of organizational learning described by Kauffeld and Rothenbusch (2023, p. 9), according to which implicit knowledge becomes explicit and collectively usable through social learning formats.

Furthermore, peer learning and collegial consultation can be used as low-threshold formats to balance differences in competence levels and to integrate learning processes into everyday work. In this way, the team’s heterogeneity is not perceived as a deficit but utilized as a resource.

These examples illustrate how empirically derived competence data can serve as a basis for developing didactically grounded, context-sensitive, and evidence-based approaches to competence development. In doing so, a central aim of the AIComp model is fulfilled: to describe competence development not in abstract terms but to translate it into concrete, future-oriented learning processes that deliberately foster future skills within organizational contexts.

From diagnosis to intervention: Heterogeneity as a starting point for didactically grounded competence development

A central finding of the pilot implementation was the pronounced heterogeneity within the examined team. This diversity should not be regarded merely as a diagnostic challenge but as a developmental resource. While differences in competence levels reveal the diagnostic potential of AICompAss, they simultaneously provide a starting point for a developmental logic, enabling learning through exchange, perspective-taking, and social negotiation. In this sense, heterogeneity becomes a visible expression of organizational learning potential. Different competence levels generate variance and create spaces for resonance where new perspectives emerge and implicit knowledge is socially shared. Learning organizations are characterized by their ability to actively harness this diversity to promote collective reflection and knowledge exchange (Ehlers et al., 2024b, p. 17; Tippelt & Schmidt-Hertha, 2018, p. 317; Kauffeld & Rothenbusch, 2023, pp. 173, 200). AICompAss supports this process by making individual differences visible, thereby providing an empirical foundation for team-based learning and development measures.

Based on the collected competence profiles, two complementary directions for development can be distinguished. On the one hand, AICompAss enables the derivation of didactically structured training measures that systematically plan learning processes and align them with specific competence fields. In this study, this was realized through the combination of the RITA model (Schubiger, 2022) and McCarthy’s 4MAT logic (2000). Both models structure learning processes along distinct phases and learning pathways, from activating prior resources to processing information, applying knowledge, and engaging in reflection and evaluation, thus addressing different learning styles and starting points. This combination creates a modular structure that renders learning processes both coherent and adaptive.

On the other hand, AICompAss also opens possibilities for fostering informal and emergent forms of learning that are not designed as formal training but arise organically from team interaction. These include peer learning, collegial consultation, and mentoring formats in which individuals with higher proficiency in certain competence fields share their experiences, thereby multiplying organisationalorganizational knowledge. In this way, heterogeneity is productively utilized.

AICompAss thus expands the perspective on competence development beyond the traditional training context. It functions not only as a diagnostic instrument but also as a catalyst for learning processes in which formal and informal development formats are interwoven. The results of this research demonstrate that the use of AICompAss empirically supports these theoretical assumptions: applying the instrument not only enhances the visibility of individual competence profiles but simultaneously initiates collective learning and reflection processes within the team. This makes it evident that the use of AICompAss itself constitutes a part of organizational learning processes and renders the transition from diagnosis to intervention practically tangible.