Skip to main content
Have a personal or library account? Click to login
Evaluating Impact on Co-Researchers: Instruments for Participatory Projects and Insights from Practice with Citizens, Artists, and Artificial Intelligence Experts Cover

Evaluating Impact on Co-Researchers: Instruments for Participatory Projects and Insights from Practice with Citizens, Artists, and Artificial Intelligence Experts

Open Access
|Nov 2026
Read online

Full article

Introduction

Impact as a concept has been widely discussed within the research ecosystem for the past 15 to 20 years (Bornmann 2013), during which the emphasis has shifted from scientific or academic impacts (effects that research has on the research system) towards societal impacts (Smith et al. 2020). The term societal impact has as yet no consensus definition, but, broadly, it refers to the effects or changes that research has on society (see for example, Bornmann 2013; Reale et al. 2018; Reed 2025). However, the type of “research,” the “effects” or “changes” that are valued, and what constitutes “society” vary considerably depending on context (Pedersen et al. 2020). Furthermore, definitions of impact are usually accompanied by the expectation that it will be or has been demonstrated (Donovan 2011). Approaches to capturing impact range from strict quantitative or statistical measures to qualitative data or even arts-based methods (e.g., based on embodiment or aesthetics), or a mix thereof (Reed 2025).

Participatory research approaches are also affected by this discourse, with the shift towards societal impact facilitating the inclusion of new actors within the research system. At the same time, these approaches seek increased legitimacy by demonstrating their own impact (Wiek et al. 2014). While most concepts of impact concentrate purely on output-related outcomes, within participatory approaches, impact is also strongly focused on process-related outcomes. In other words, in addition to the collaboratively created output, which should affect wider society, the collaborative research process itself impacts researchers and everyone involved (Staley 2015; Gönner et al. 2023).

Multiple approaches to including actors from outside academia within research teams exist (Schrögel 2025). Our case study drew on the project How to Explain AI, funded through the Ludwig Boltzmann Gesellschaft’s Open Innovation in Science (LBG OIS) Impact Lab, where OIS signifies a process of managing knowledge flows and collaborations between and beyond disciplines and organizations for all stages of research (Beck et al. 2022). However, for the purposes of this article and to discuss the relationship between impact and the involvement of non-academics in research, we use the term participatory research instead. Participatory research is an umbrella term for collaborative research approaches that engage societal groups outside academia in research processes (Vaughn and Jacquez 2020).

In evaluating the project, it was important to consider approaches that fit the specific characteristics of participatory research. Authors in the literature on citizen science and participatory research evaluation have emphasized that conventional approaches to assessing impact that focus on predefined outputs and quantitative indicators do not fully capture the complexity of collaborative research processes (Staley 2015; Kieslinger et al. 2018). They advocate context-sensitive evaluation frameworks that take into account impacts on science (and the scientific system), participants (meaning anyone involved within the research process), and wider society (Shirk et al. 2012; Kieslinger et al. 2018) and that consider a wide array of domains, such as science and technology, society, economy, environment, and governance or policy (Wehn et al. 2021; Passani et al. 2022).

The literature further highlights the importance of formative and adaptive approaches to evaluation that accompany projects throughout their lifecycle. Iterative reflection, feedback loops, and the inclusion of multiple stakeholder perspectives are considered particularly important in participatory settings, where goals and forms of engagement may evolve over time (Kieslinger et al. 2018; Passani et al. 2022). We therefore considered a strictly number-driven or statistical evaluation of impact to be too narrow and not capable of meaningfully capturing process-related outcomes (beyond output-related effects) for those involved. Consequently, we concentrated on qualitative evidence and a process-oriented, modular approach. The funding framework included an impact support program accompanied by an impact expert. Within this program, we found the proposed OIS Impact Model (LBG OIS Center 2021), an impact planning and evaluation tool, to be suitable for evaluating the outcomes of our participatory process.

The OIS Impact Model was co-created by the LBG OIS Center together with researchers, practitioners, citizens, funders, politicians, and the media, with the aim to map out pathways to impact for projects in which researchers and non-researchers collaborate (Wailzer and Soyer 2022). Its structure is based on a theory of change (ToC), a commonly used tool in impact evaluation that describes how and under which assumptions a project is expected to contribute to societal change (see for example, Newcomer et al. 2015; Breuer et al. 2016; Reed et al. 2018).

The OIS Impact Model aligns with central developments in the literature on evaluating participatory research by moving beyond narrowly output-oriented understandings of impact and explicitly integrating process-related dimensions. Similar to existing frameworks in participatory research evaluation, it assumes that collaborative research produces impact not only through scientific outputs, but also through changes in awareness, competencies, relationships, and behaviors among participants. In this sense, the model responds to calls for broader, holistic, and formative evaluation approaches that take into account learning processes, empowerment, mutual exchange, and the co-production of knowledge (Shirk et al. 2012; Kieslinger et al. 2018; Schäfer et al. 2025). At the same time, the OIS Impact Model further develops these approaches by operationalizing participatory impact pathways in a particularly practice-oriented and modular way. While many existing frameworks remain relatively abstract or primarily conceptual, the OIS Impact Model links specific quality criteria and outcomes to concrete actor groups and corresponding evaluation methods. The model therefore combines elements from established participatory impact evaluation literature with a pragmatic tool for ongoing reflection within participatory research contexts.

The OIS Impact Model approach suggests pathways from designing collaborative research (activity) via the collaborative process itself (seen as the output in that case) to its short-, medium-, and long-term outcomes. Short-term outcomes are changes in awareness and competencies of the persons involved; medium-term outcomes are changes in behavior; and long-term outcomes are changes in circumstances. The approach culminates in two impacts, defined as (i) more societally relevant research and (ii) a higher likelihood of uptake of the collaboratively produced research. All elements include suggestions on how to evaluate these quality criteria and outcomes that apply to different actor groups: co-researchers, (professional) researchers, practitioners, and policymakers. Note that the model is not intended to be used in its entirety; rather, a relevant subset of aspects is to be defined at the beginning of a project. The evaluation approach uses a combination of various empirical social-science methods, namely surveys, semi-structured interview guides, focus groups, team reflections, and desk research. These methods can be used individually or in combination for summative or formative evaluation at various time points throughout the project.

Participatory Process of the Project How to Explain AI

The project How to Explain AI, on which this article is based, aimed to explore public understanding of artificial intelligence (AI) and to develop an art-based science communication intervention to raise AI literacy among non-experts (Long and Magerko 2020; Almatrafi et al. 2024). In a twelve-month participatory process, we involved 20 co-researchers to contribute their perspectives on what pressing topics should be addressed by the intervention and on suitable art-based formats and their implementation. The key issues we jointly addressed were: What questions do citizens have about AI in relation to their everyday lives? How should we present answers to these questions such that they generate interest and are relatable for the widest possible audience?

Participants

We use the term “co-researchers” to refer to three groups of persons with diverse backgrounds: citizen scientists, artists, and AI experts. Nine citizen scientists contributed their everyday perspectives on AI across multiple stages by formulating topics, questions, and concerns about AI, co-creating the educational intervention, and joining a collaborative review workshop to align outcomes with earlier project insights. They were also encouraged to act as multipliers by sharing the project outcomes within their social environments, and their contributions were acknowledged in all publications on the topic. We use the term citizen scientists in accordance with the Ten Principles of Citizen Science (ECSA 2015) and the ECSA Characteristics of Citizen Science (Haklay et al. 2020).

In addition, seven artists from a range of artistic disciplines introduced new creative approaches to knowledge transfer. Four AI experts with a professional involvement with AI provided a realistic view of the current technological capabilities of AI, including advantages and risks. Where differentiation from the co-researchers is needed, the academic researchers affiliated with the university who led the participatory process are referred to as the “core research team” in this article.

Participation was voluntary, and calls for participation were shared by various organizations and networks, mostly with a local focus because the project involved in-person meetings.

Rotating workshops with individual co-researcher groups

Figure 1 provides an overview of the project timeline. The upper section illustrates the topic-related activities that advanced the participatory research process, including knowledge generation and the co-creative development of an educational intervention to foster AI literacy. The lower section presents activities related to planning and evaluating the impact of the participatory process, which are the focus of this article.

Figure 1

Project and evaluation activities schedule. WS: workshop.

We began the participatory process by working separately with the three groups of co-researchers to identify their priority AI topics and generate innovative ideas for communicating them, conducting a series of rotating workshops with the three groups (see “Activities related to the research topic” in Figure 1). Over two months, three workshops per group helped to specify (i) the topics to be addressed by the intervention and (ii) key elements of the intervention’s art-based format (Meyer et al. 2023).

Mixed-group ideas workshop

The next step was to convene a mixed-group workshop with all interested co-researchers (12 in total) in order to develop concrete ideas for the educational intervention. This group took the collaboratively generated ideas, and using a jointly developed set of criteria, selected three to be taken forward. This step ensured meaningful co-researcher involvement while preserving flexibility for the core research team to make the ultimate decision given practical and resource considerations. The project leads made the final selection—an AI-themed song—as it met the key criteria of dissemination potential, transferability, and creativity while remaining feasible within the project’s timeframe and budget, and chose a singer-songwriter to implement it.

Song review workshop

Once a first draft of the song (about two-thirds complete) was available, we held a review workshop where the singer-songwriter presented the interim version that we discussed with representatives of the co-researchers. All participants from the earlier ideas workshop were invited, of whom six attended. Their feedback was incorporated into the final version of the song (Robopsychology Lab 2023). All co-researchers were invited to appear in the accompanying music video.

Public event

The participatory process concluded with a public evening event at which the song—as the main output of the process—was officially released. The event included presentations on the participatory process and a panel discussion featuring co-researchers and the singer-songwriter. Thus, co-researchers were involved throughout the process, from the initial development of content and concepts through implementation to public presentation and launch of the song as the project’s final AI literacy intervention (Robopsychology Lab 2023).

Characteristics of the participatory process

This participatory process had several characteristics that, while not uncommon individually, in combination set it apart from many other citizen science projects and were relevant to the impact evaluation described in the following sections:

  • The output format was intentionally left open, which led to diverse expectations and ideas among participants.

  • The process brought together individuals with diverse backgrounds in communication and work experiences, culture, and interests.

  • Participation was possible exclusively through in-person workshops.

  • Co-creation—along with collection of knowledge and perspectives—was a central emphasis.

  • Separate- and mixed-group workshops were employed, starting with three distinct co-researcher groups that were then merged.

  • Responsibility for the ultimate intervention decision remained with the core research team.

Building on these characteristics, we present a transferable, modular evaluation framework that comprises hands-on methods and instruments that are particularly suitable for process-oriented projects with strong collaboration among the participants and that aim primarily to assess impact on (non-academic) co-researchers. Drawing on the collaborative development of a creative intervention to foster public AI literacy as our example setting, we report methodological lessons on the feasibility and usefulness of the instruments applied and offer practical guidance for applying this evaluation toolkit in other contexts.

Methods for Evaluating Impact

We employed several approaches that fostered self-reflection within the core academic research team and utilized complementary assessment tools to determine the extent to which the intended forms of impact, that is, (i) creating a shared vision of the project outcome among all involved, (ii) fostering engagement with the research project beyond the project activities, and (iii) empowering the co-researchers to critically reflect on and interact with AI in everyday contexts, were achieved.

To ensure ongoing reflection and enable responsiveness to emerging insights and potential needs for adaptive refinements, we deliberately chose not to limit impact evaluation to a final summative assessment at the end of the project, but instead embedded it as a continuous practice within the research process. We distinguish between methods for setting up the reflective process and instruments for impact evaluation.

Setting up reflective processes

Theory of change workshops

Within the core academic research team, we conducted a series of ToC workshops facilitated by an external moderator. Drawing on a Logic Model Template (LBG OIS Center 2023), these workshops provided a basis for vision building, systematic reflection, and focused discussions within the project team on project goals (see Supplemental File 1: ToC).

In the impact evaluation literature, a distinction is usually made between a logic model and a ToC. A logic model is typically a simpler and more linear representation of how a project or program would result in certain expected outcomes and to which impact goals these outcomes would contribute. A ToC, on the other hand, is a more complex and detailed representation of the different pathways to impact needed to achieve a certain impact. It shows how these pathways intersect or potentially create barriers and makes explicit the assumptions underlying the expected changes (Newcomer et al. 2015; Breuer et al. 2016; Reed et al. 2021). Even though the first ToC workshop drew on a logic model template as a basis, the subsequent workshops served to develop the model further (according to Mahve-Beydokhti and Wailzer 2026). In particular, the team and the moderator looked more closely at the different pathways to impact for each actor group, how they interact, and under what conditions they would realistically lead to further changes, while reflecting on the socio-political environment in which the project was situated. Therefore, we will refer to the model as a ToC rather than a logic model.

While the ToC process also encompasses broader levels of impact, including those on society and the research community, we defined several objectives that specifically addressed the anticipated impact on the co-researchers. For example, one central goal was to enhance co-researcher self-efficacy by strengthening their capacity to reflect on the topic of AI and thus support a direct life impact that extended beyond their involvement in the project. Throughout the process, we reflected on informal indicators of progress, such as co-researchers’ voluntary attendance at conferences and public events, and sharing of personal accounts of how the project affected their daily lives, for example, by motivating them to use newly gained perspectives in conversations with their social environment. In addition to these anecdotal indications of impact, the goals developed in the ToC workshops closely aligned with the project’s predefined impact aspects, which were subsequently examined in a systematic evaluation process.

OIS Impact Model and evaluation of impact aspects

We used the OIS Impact Model (LBG OIS Center 2021) as a framework for reflecting on core questions relevant to the participatory process, including: (i) What kind of impact on the co-researchers are we aiming to achieve and why? (ii) How can we know whether participation in the project has led to a positive and long-term impact? Guided by these considerations, we then defined a subset of impact aspects from the OIS Impact Model that the project should meet. Among this subset, particular attention was directed towards the impact aspects of “eye level” (translated literally from the German “(auf) Augenhöhe”, which has the figurative meaning of “(on an) equal footing”), “continuous involvement,” and “shared vision.” These were specifically prioritized as they directly corresponded to the project’s stated key goals for co-researcher impact, namely enabling sustained involvement and supporting the development of a shared vision. Table 1 lists the aspects that we considered particularly important for the impact on co-researchers. It also shows that each aspect can be assessed with multiple evaluation instruments and, conversely, that each instrument described below can inform on several aspects. We utilized and adapted tools from the OIS Reflection Instruments for our evaluation purposes (LBG OIS Center 2022).

Table 1

Overview of the selected impact aspects and evaluation methods that explicitly focus on the impact on co-researchers. The aspects are named in accordance with the OIS Impact Model (LBG OIS Center 2021).

ASPECTDESCRIPTIONEVALUATION INSTRUMENT
1. Eye levelCo-researchers are involved as equals, which enables participation, co-design, and balance of interests among all participants
  • Research diaries

  • Qualitative interviews

  • Participation check

  • Anecdotal feedback per mail or orally

2. Emotional benefitsParticipation in the project yields emotional meaning and value for the co-researchers
  • Participation check

  • Qualitative interviews

3. Continuous involvementCo-researchers are involved continuously and are satisfied with their participation
  • Participation check

  • Qualitative interviews

  • Research diaries

  • Theory of change reflection

4. Self-efficacyCo-researchers feel increasingly capable and confident to actively engage in reflecting on and discussing the topic
  • Qualitative interviews

  • Theory of change reflection

  • Anecdotal feedback per mail or orally

  • Attendance at AI-related public events and conferences

5. Shared visionResearch priorities, processes, and outcomes are discussed and identified collaboratively to foster a shared sense of project direction and purpose among co-researchers
  • Qualitative interviews

  • Pre-project dialogue

  • Pre-project reflection form

6. Direct life impact on co-researchersParticipation in the project yields practical use and meaning for the co-researchers
  • Research diaries

  • Theory of change reflection

  • Anecdotal feedback per mail or verbally

  • Discussion rounds in workshops

Instruments for impact evaluation

Research diary

For overarching self-reflection, we used a research diary that allowed structured consideration of the participatory dynamics and reflection on whether the predefined impact aspects appeared to be met (see Supplemental File 2: Diary DE and Supplemental File 3: Diary EN). At the end of each month and after every workshop, all members of the core academic research team documented key events and considered how co-researcher involvement had been ensured. The diaries also provided space to assess whether interactions with co-researchers seemed to occur on an equal footing. In addition, they were used to record other observations relevant to the impact aspects, such as anecdotal accounts of how the project and related experiences affected co-researchers’ personal lives, for example, by increasing their willingness to discuss AI with others or awareness of AI in everyday contexts.

Pre-project introductory dialogue

All individuals who had expressed an interest in participating as co-researchers were initially contacted by telephone or video call by a member of the core academic research team. This allowed us to establish an early connection, explore first-hand the potential co-researcher motivations and communication styles, and assess whether they were willing and able to express themselves at the level necessary for productive engagement with other group members (see Supplemental File 4: Dialogue DE and Supplemental File 5: Dialogue EN).

Pre-project reflection form

In addition to this personal dialogue, co-researchers were encouraged to take some time to reflect on their reasons for participating, their anticipated role in the project, and their expectations for the collaborative process, and to capture their thoughts in writing using a pre-project reflection sheet provided before the participatory process. The sheet comprised seven open questions that addressed general expectations, personal motivations, and intended goals for their involvement. The co-researchers were also asked what they hoped to take away from the project and which aspects they considered particularly important for the process and the group to acknowledge. Finally, they were given the opportunity to share any additional thoughts they wished to communicate to the core academic research team (see Supplemental File 6: Pre-Form DE and Supplemental File 7: Pre-Form EN).

Both pre-project interactions aimed to build early involvement and connection as peers. They laid early foundations for a shared vision of the participatory process and offered a basis on which to proceed with the impact evaluation.

Participation check surveys

To evaluate whether the workshops were contributing towards fulfillment of the selected impact aspects and to identify potential problems early on, we used the participation check instrument from the OIS Reflection Instruments (LBG OIS Center 2022). This brief online questionnaire offered a quantitative overview and was administered at the end of each workshop. The co-researchers accessed it via link or QR code and were provided with a tablet if needed. Five items assessed emotional value, namely, whether the workshop was perceived as motivating, connecting, meaningful, exciting, and pleasant. They further rated whether the participatory process occurred on an equal footing, that is, whether they (i) were able to influence content and result of the workshops, (ii) were taken seriously when voicing an opinion, and (iii) experienced an atmosphere that allowed opposing views. In the second project phase, after the joint intervention workshop, three additional items evaluated overall satisfaction with the intervention and whether the co-researchers felt involved and heard in the creation and decision-making processes. All items used a five-point scale. Given the small number of participants, questionnaire data were analyzed descriptively and served primarily as an exploratory overview for the core research team. Finally, three open-ended questions captured positive and negative aspects of the workshops and offered space for co-researchers to share further comments or wishes (see Supplemental File 8: Check DE and Supplemental File 9: Check EN).

Qualitative interviews

We conducted two rounds of qualitative, semi-structured interviews to assess the impact aspects of shared vision, emotional value, connection on an equal footing, and continuous involvement in a personal setting. The first round took place approximately two weeks after the intervention workshop and focused on co-researcher satisfaction with the decision-making process regarding the intervention idea (see Supplemental File 10: Interview 1 DE and Supplemental File 11: Interview 2 DE). This complemented the immediate quantitative participation check, allowed time for reflection, and offered an additional opportunity to voice perspectives or grievances. Particular emphasis was placed on the aspect of a shared vision, as we examined the extent to which the co-researchers’ initial expectations and visions aligned with the intervention. These interviews were analyzed using thematic clustering, with findings informing subsequent intervention development. The second round, carried out after a process review meeting with the co-researchers and the singer-songwriter, explored views on the song as the intervention, including whether this format and the specific song aligned with their personal project vision and aims, and which aspects they wished to see more strongly reflected in the lyrics. We also discussed experiences of the review meeting, particularly whether the co-researchers felt heard and valued (see Supplemental File 12: Interview 2 DE and Supplemental File 13: Interview 3 EN).

Results and Discussion

In this section, we report observations and reflections on how the methods and instruments that we used performed in practice, highlighting strengths, limitations, feasibility, and participant burden. We do not present substantive interview and survey results (e.g., scale item means), as this case study focused on the applicability of the evaluation toolkit and its suitability for continuous, process-oriented impact assessment.

Tool-based observations and reflections

Reflections on the OIS Impact Model

We found that the OIS Impact Model provides a valuable anchor point for participatory projects. The detailed descriptions of each aspect and the concrete guidance for their evaluation may prove particularly useful for researchers with limited prior experience in participatory projects. However, applying the model requires both interpretative judgement and time, especially when it is used continuously throughout a project rather than only retrospectively. Unlike a checklist or linear framework, the OIS model is not designed to be followed step by step from beginning to end. Instead, it offers a broad set of impact aspects for consideration from which to select and combine to suit a specific context and project design.

Research diaries

Completed on a regular basis, the research diaries supported the core academic research team’s continuous engagement with the impact aspects, prompting deliberate reflection on how these were addressed throughout the project. They also provided an important space for documenting more anecdotal observations and interactions that did not readily fit into other instruments, but offered meaningful insights into aspects such as continuous involvement and development of a shared vision. In this sense, the diaries functioned as both a reflective and a documentation tool for the team. Importantly, they also illustrate why impact evaluation was approached as an ongoing practice rather than a purely summative assessment at the end of the project. Iterative use of the diaries enabled learning from one another’s reflections over time and allowed incorporation of emerging insights into the ongoing process. Reflections during the ToC workshop highlighted that the diaries also played a role in sustaining motivation over time, for example, by making ongoing progress more tangible and by offering space to address topics that might otherwise have remained undocumented.

Pre-project interactions with the co-researchers

Given the project’s participatory character, communication on an equal footing was defined as a key impact goal. Rather than relying solely on email exchanges, we thus made an effort to initiate direct personal contact with the co-researchers prior to the project start, as we assumed that this would facilitate early connection. From our perspective, this approach proved effective, as these early personal exchanges contributed to the emergence of a shared vision for the collaborative work and allowed us to gain insights into the expectations that co-researchers brought to the project. This reflection is contingent on project conditions and scale: While such early personal exchange is difficult to realize in projects that involve a large number of participants, it played an important role in our project with a small core group.

Participation check

Overall, the instrument proved useful as a recurring point of reference for monitoring co-researcher experiences throughout the project and for identifying general trends regarding what worked well and where adjustments were needed. While the items used (standardized or adapted versions) supported this form of brief, low-threshold feedback, opportunities for more detailed reflection emerged primarily in responses to open-ended questions, which allowed co-researchers to articulate considerations not readily expressed through Likert-style item ratings. At the same time, it became clear that certain expectation-related dynamics remain difficult to capture by questionnaires alone. For example, in one case, a co-researcher expressed disappointment at the end of the first workshop phase, stating that they had “expected more impressive results from a scientific project.” Though this is only a single instance, it is noteworthy that the gap between expectations and what is feasible within the scope of such projects became visible only through personal dialogue and was not revealed in the participation checks. This underscores the importance of continuous direct interaction in participatory research.

Qualitative interviews

The interviews proved to be an important complement to the participation checks, as they enabled personal exchange between the core academic research team and co-researchers on an equal footing and surfaced nuances that were missed by standardized quantitative instruments. In particular, they made visible how project intentions and choices were interpreted by the co-researchers. For instance, one co-researcher described in an interview a perceived “blurring” of the roles of citizen scientists, experts, and artists towards the end of the process and expressed a preference for clearer role distinctions. While convergence of roles had been an intentional element in the final phase (i.e., starting with the mixed-group ideas workshop), focused on jointly developing the intervention, this tension between core academic researcher intention and co-researcher preference became apparent only in the interview setting. Similarly, the interviews brought forward a wish for more exchange and input across groups during the workshop phase—feedback that went beyond what was articulated in the participation checks and pointed to experiential aspects of collaboration that are difficult to assess using brief survey instruments. Although we shared updates between co-researcher groups throughout, it was only in the qualitative interviews that this desire for increased cross-group interaction was revealed, which suggests that future projects could consider more opportunities for such exchange from the outset, for example, in the form of an initial joint introductory meeting.

General observations and reflections

Impact planning for a project with an open outcome

Conceptualizing impact orientation and integrating it into a participatory project from the beginning is not a trivial task. Designing pathways to impact within a theory of change involves creating logical, intertwined sequences of effects that detail how a program or project relates to wider societal goals. However, at the very beginning of a participatory project, the output of the project may not be clear because this will be the result of the co-creation process. Thus, designing pathways to impact when the project’s results are unknown seems paradoxical.

However, this does not mean that thinking about impact at this stage is inappropriate. On the contrary, it may support thinking in navigating open-ended processes. Open and participatory projects are not free of constraints, because they exist within a context and have objectives that contribute to long-term societal goals. Defining these long-term impacts explicitly from the start helps to facilitate the co-creation process such that the co-created output will be aligned with these impacts.

Additionally, the development of pathways to impact is not a one-time endeavor at the start of a project after which they are set in stone. It should be seen as a continuous reflection of steering the project in accordance with its societal goals. While the first pathways to impact mapped out within the ToC from activities to outputs and then outcomes may be vaguer at the beginning, they help to create a common understanding and to gain a sense of what elements are important to the project. The more advanced the co-creation process and the more concrete the output, the easier it becomes to adapt and refine the pathways to impact.

Evaluation as a basis for iterative project adaption

The continuous evaluation process informed the ongoing development and adaptation of the project. For instance, we had initially planned shorter workshops for the experts than for the other two groups, as we primarily understood their role as contributing knowledge to the project. However, the workshops also proved valuable to the experts, particularly by creating space for exchange and shared reflection. In response to co-researchers’ feedback and their explicit wish for more discussion time, we extended the duration of subsequent workshops. In addition, several co-researchers emphasized in the qualitative interviews that, in their view, an intervention should always include a physical or embodied component. Although the song remained the central intervention format, these reflections were revisited later in the project and informed the development of an interactive exhibition booth with multiple activities designed to actively engage visitors in reflecting on AI (Bauer and Meyer 2025; Meyer et al. 2024). Finally, ongoing evaluation helped to identify which impact aspects mattered most in our project context and should be given particular focus.

Definition and assessment of impact aspects

We defined a broad set of core impact aspects at the outset of the project. However, in the course of the participatory process, two of these aspects took on particular importance for understanding impact on the co-researchers: connection on an equal footing and development of a shared vision. This reprioritization was not the result of an explicit redefinition, but rather emerged naturally through our ongoing reflection. Reflections documented in the ToC workshops also suggested that the initial set of aspects may have been overly extensive, which highlights the importance of prioritization when working with impact frameworks in participatory settings.

A recurring observation throughout the project concerned the tension between maintaining low-threshold engagement on an equal footing and pursuing a systematic and standardized assessment of impact. Implementing formalized evaluation procedures risks positioning co-researchers as “study subjects,” thereby potentially introducing additional burdens on participants and contravening the principles of citizen science that emphasize their role as active peers and collaborators. One implication of this tension concerned the evaluation of self-efficacy, which was defined as an impact aspect but was not assessed by explicit, standardized measures. We deliberately avoided introducing such measures to anchor the framing of the interactional dynamic in collaboration rather than assessment. We thus relied on our observations of group dynamics, contributions during workshops, and insights obtained through dialogue to evaluate impact gained in this area. Additional tensions between the impact aspects were voiced in the research diaries, for example, in relation to balancing (i) adherence to workshop schedules and progress towards concrete outcomes with (ii) the aim of fostering communication on an equal footing. This occasionally required adopting a more directive moderator role rather than participating as a group member in discussions.

Taken together, these reflections underscore that even when explicit impact aspects are defined in advance, they cannot always be fully realized simultaneously in participatory processes, and that impact evaluation in such contexts needs to take these inherent trade-offs into account.

Evaluation as a contributor to impact

Importantly, the evaluation process not only was a means of assessing impact but itself also contributed to the project’s impact. This was particularly evident in the pre-project interactions with co-researchers, which served primarily to support early involvement, but also functioned as an initial evaluation of the impact aspect of shared vision. More broadly, the distinction between evaluating and creating impact was not always clear-cut. However, in participatory projects such entanglement may be productive. For instance, we used qualitative interviews to examine whether aspects such as shared vision, self-efficacy, and direct life impact were being addressed, but the interviews also contributed to these aspects by giving co-researchers further opportunities to articulate their perspectives and remain involved in the project. The evaluation therefore enabled systematic assessment and at the same time directly supported the forms of impact it sought to create.

Direct life impact enabling co-researchers to act as multipliers

A broader learning emerging from the project concerns how impact on co-researchers may unfold when engaging with highly technical and complex topics such as AI. In such contexts, individuals may initially be hesitant to interact or participate actively, often perceiving the subject matter as inaccessible or “not for them.” Against this backdrop, impact thus initially manifests on an individual level, beginning with the co-researchers themselves. Supporting co-researchers in reflecting on AI and its presence in their everyday lives can therefore be understood as a meaningful form of impact in its own right, particularly where it contributes to increased confidence, self-efficacy, and a sense of agency in relation to the topic. At the same time, this individual-level impact may extend beyond the immediate participatory setting, as co-researchers can act as multipliers who carry these reflections into other aspects of their social life. In this vein, impact on the co-researchers may be seen as a prerequisite for wider dissemination and resonance. For instance, experts reported that engaging with citizen scientists’ lay perspectives, uncertainties, and questions about AI gave them a renewed understanding of their own field and helped them to recognize how they could communicate their work more effectively to non-expert audiences. Informal moments, such as conversations during breaks, exchanges embedded within workshop activities, and insights from the qualitative interviews demonstrated an increasing willingness among co-researchers to engage with the topic in their social lives, beyond their direct participation in the project. An example of increased co-researcher self-efficacy in engaging with AI beyond the project context emerged in email exchanges with the core research team, where some co-researchers mentioned that they had used AI services for the first time to help formulate their messages.

Period of impact evaluation

The assessment of impact on the co-researchers focused on the participatory phase of the project and thus concluded with its end. Though the chosen Logic Model Template prompts reflection on anticipated long-term changes if the impact goals are to be achieved, it does not specify a temporal horizon or concrete time points for revisiting these assumptions over time. As a result, how impact on co-researchers may develop post-project remains an area for exploration in future appropriately scoped projects.

Accountability for steering impact

The question of who to involve in actually defining and steering impact remains. In the project How to Explain AI, the co-researchers did not co-design the impact evaluation. While participatory research changes power structures within the research system by integrating diverse types of knowledges, it does not remove power asymmetries. It remains the professional researchers who decide whether to include co-researchers to co-design the pathways to impact. This constitutes a limitation of our study and may have constrained the range of perspectives captured, particularly regarding impacts experienced and perceived. At the same time, pressure to formalize and demonstrate impact typically sits—as in our case—with the members of the core academic research team, not the co-researchers. The imperative to systematize and capture impact therefore remains with the professional researchers. This becomes particularly evident in the growing numbers of impact case studies, in which researchers systematically document and reflect on how their projects have generated impact across multiple levels—an approach we also adopted for the present project (Bauer and Meyer 2025). This concentration of responsibility should, however, spark discussions about who could and should be part of defining impact and how it is evaluated, and about the circumstances required in terms of policy, funding, and research priorities and frameworks to achieve that.

Practical Guidance

From the experience gained by applying the instruments described, we derived recommendations that we share here as practical guidance.

We offer universal recommendations that apply to participatory projects regardless of the type of citizen science implementation. However, we also provide recommendations that are particularly relevant for open-outcome-oriented projects that involve intense collaboration with and among co-researchers. Examples include art and science projects, co-design projects in the field of human–computer interaction, and co-creation projects for urban development. For these cases, formative impact evaluation is especially important. Modular instruments and regular measurements can help teams to respond adaptively to an open and evolving project trajectory. Highly collaborative processes with open output pose particular challenges in clarifying expectations, defining roles, recognizing contributions, and balancing power between co-researchers and core academic researchers. The following best-practice recommendations synthesize the key lessons learned from the project.

  • Initiate impact planning and definition of evaluation aspects at the very beginning of the project.

    In line with existing literature (Kieslinger et al. 2018), setting up the impact planning and evaluation at the start or even before collaboration with the co-researchers is highly beneficial. This approach offers a foundational framework for more detailed planning and serves as a practical guideline for project coordination.

  • Focus on a small set of impact aspects and prioritize what is truly relevant.

    In our project, two aspects proved particularly important for generating impact: connection on an equal footing and the development of a shared vision. Rather than addressing many impact aspects superficially, projects should thus define a smaller set of core aspects and pursue them systematically.

  • Select evaluation instruments that can be applied repeatedly, while keeping the burden on co-researchers within reasonable limits.

    In line with the emphasis of ECSA characteristics on active participant involvement (ECSA 2015), impact evaluation should avoid reducing co-researchers to research subjects who are repeatedly assessed. Instead, evaluation tools should support reflection without becoming overly burdensome. Future projects could also involve co-researchers directly in designing and conducting the impact evaluation itself.

  • Consider and define, at project start, how insights from informal interactions will be systematically captured and integrated as evaluative data within impact assessment frameworks.

    Existing work argues for moving beyond fixed, mainly quantitative impact measures for impact evaluation (When et al. 2021). In our project, it became increasingly clear that informal exchanges offered a rich source of impact evidence, which underscores the need to plan their systematic documentation (e.g., via notes in diaries) from the outset.

  • Analyze evaluation results continuously and use the insights to guide project management.

    Our work demonstrated that impact assessment can directly inform and improve project management when conducted formatively throughout the project, rather than summatively after its completion. This aligns with the literature, which emphasizes that ongoing evaluation enables flexibility and helps to address undesirable project developments as they emerge (Kieslinger et al. 2018).

  • Be aware of your specific role as a core academic researcher (project lead) and its implications.

    Core researchers should consider how their positioning shapes the participatory process and its outcomes. The choice between participating actively and taking a more facilitative or moderating role influences group dynamics and contributions, and should therefore be reflected explicitly and repeatedly.

Conclusion

Overall, we found that the methods and instruments used for continuous evaluation in this process-oriented participatory project with an open outcome were effective. Building on these experiences, this case study provides hands-on guidance and a transferable toolkit for assessing impact on co-researchers, which is an area of growing importance as participatory research projects are increasingly expected to reflect on their impact on the individuals involved. Our theory of change—established at the outset, revised after a major milestone, and revisited at the end—provided overall guidance for impact planning and evaluation as well as for project management. The OIS Impact Model offers a modular, flexible framework that articulates key aspects for successful co-research projects and suggests tools and templates that can be adapted to specific needs. This formative evaluation approach supports impact planning and assessment, and enables management of project activities that is continuous and iterative. Hence, our approach is specifically suited to projects with open outputs and intense collaboration, such as art-based and co-creation projects.

Future research could examine the transferability of these findings to other contexts and further develop the tools. While this article has focused on the impact on co-researchers, future work could investigate the impact on the core research team and their evaluation in greater detail.

Supplementary Files

Supplemental File 1

ToC. Screenshot Theory of Change. DOI: https://doi.org/10.5334/cstp.983.s1

Supplemental File 2

Diary DE. Research Diary Template (German Original). DOI: https://doi.org/10.5334/cstp.983.s2

Supplemental File 3

Diary EN. Research Diary Template (English Translation). DOI: https://doi.org/10.5334/cstp.983.s3

Supplemental File 4

Dialogue DE. Pre-Project Introductory Dialogue (German Original). DOI: https://doi.org/10.5334/cstp.983.s4

Supplemental File 5

Dialogue EN. Pre-Project Introductory Dialogue (English Translation). DOI: https://doi.org/10.5334/cstp.983.s5

Supplemental File 6

Pre-Form DE. Pre-Project Reflection Form (German Original). DOI: https://doi.org/10.5334/cstp.983.s6

Supplemental File 7

Pre-Form EN. Pre-Project Reflection Form (English Translation). DOI: https://doi.org/10.5334/cstp.983.s7

Supplemental File 8

Check DE. Participation Check Survey (German Original). DOI: https://doi.org/10.5334/cstp.983.s8

Supplemental File 9

Check EN. Participation Check Survey (English Translation). DOI: https://doi.org/10.5334/cstp.983.s9

Supplemental File 10

Interview 1 DE. Guideline First Qualitative Interview (German Original). DOI: https://doi.org/10.5334/cstp.983.s10

Supplemental File 11

Interview 1 EN. Guideline First Qualitative Interview (English Translation). DOI: https://doi.org/10.5334/cstp.983.s11

Supplemental File 12

Interview 2 DE. Guideline Second Qualitative Interview (German Original). DOI: https://doi.org/10.5334/cstp.983.s12

Supplemental File 13

Interview 2 EN. Guideline Second Qualitative Interview (English Translation). DOI: https://doi.org/10.5334/cstp.983.s13

Ethics and Consent

It was not possible to acquire determination of oversight because the institutional ethics committee was established after the research and does not review studies conducted prior to its formation. However, we have adhered to ethical guidelines: This study complied with the Declaration of Helsinki. All persons involved were ≥18 years and provided written informed consent after an in-person explanation. Participation in surveys and interviews was voluntary and unpaid. Names of the co-researchers were published on the project website only with explicit consent.

Acknowledgements

We thank all participating co-researchers and our former team members Benedikt Leichtmann and Thomas Meneweger for their contributions to the project. A full list of persons involved in the participatory process is available on the project website (Robopsychology Lab 2023).

Data Accessibility Statement

Supplementary materials include templates (research diary, pre-project introductory dialogue, interview guidelines, participation check) to enhance transparency and reproducibility of the research design and participatory process. Research data are available from the authors on request.

Author Contributions

Kathrin Meyer, member of the core academic research team at Johannes Kepler University (JKU) Robopsychology Lab, served as organizational project manager, co-led project design and implementation, coordinated the participatory process, and facilitated workshops. She led the article’s concept and writing and finalized the manuscript. CRediT: Conceptualization; Project administration; Methodology; Investigation (workshops); Data curation; Visualization; Funding acquisition; Writing – original draft; Writing – review & editing.

Lara Bauer, member of the core academic research team at JKU Robopsychology Lab, documented the project’s processes and results, conducted qualitative interviews and synthesized impact-evaluation findings. She co-developed this article, and drafted methods and results sections. CRediT: Methodology; Investigation (workshops); Data curation; Writing – original draft; Writing – review & editing.

Mathieu Mahve-Beydokhti, Program Manager Impact at the LBG OIS Center, funding agency of the underlying project, served as an external impact-planning consultant. He led the ToC workshops with the core academic research team in which the theory of change was developed and refined, thereby accompanying the project throughout its duration. For this article, he contributed the theoretical background on impact and an external perspective on project specifics. CRediT: Methodology; Resources; Writing – original draft; Writing – review & editing.

Martina Mara, head of JKU Robopsychology Lab, served as PI, leading fundraising, project conception, and scientific oversight. She was heavily involved in co-creation and intervention implementation and revised this manuscript. CRediT: Conceptualization; Funding acquisition; Resources; Supervision; Writing – review & editing.

DOI: https://doi.org/10.5334/cstp.983 | Journal eISSN: 2057-4991
Language: English
Page range: 43 - 43
Submitted on: Feb 13, 2026
Accepted on: Sep 28, 2026
Published on: Nov 12, 2026
Published by: Ubiquity Press
In partnership with: Paradigm Publishing Services

© 2026 Kathrin Meyer, Lara Bauer, Mathieu Mahve-Beydokhti, Martina Mara, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.