Introduction
Citizen science has become an umbrella term for a range of data and knowledge co-production activities that involve different constellations of scientists, members of the public, decision makers, industry, and other interested parties and actors (IPAs) (Wehn 2022).1 It can entail the co-production of scientific knowledge with volunteers (Shirk et al. 2012); the pursuit of civic agendas related to local environmental concerns (Irwin 1995); or the creation of learning opportunities and awareness raising (Schäfer and Kieslinger 2016). While there are disciplinary distinctions and ambiguities surrounding various citizen science practices (Fraisl et al. 2026), citizen science is best defined in context (Haklay et al. 2021; Gharesifard et al. 2019). Overall, citizen science has emerged as a powerful approach to address complex environmental, social, and scientific challenges (Ceccaroni and Piera 2017; Fritz et al. 2019). As citizen science initiatives proliferate globally (Wehn et al. 2025, MacPhail and Colla 2020, Strasser et al. 2023, Sprinks et al. 2025), ranging from small grassroots projects to large-scale programs, there is a clear need to better understand and demonstrate their impacts beyond immediate outputs. Yet evaluating these impacts remains surprisingly challenging.
The evaluation landscape reveals a persistent tension: Most citizen science initiatives are designed with explicit expected impacts in mind. However, the actual achievement of outcomes and impacts often extends beyond project timelines, may encompass unforeseen outcomes (both positive and negative), and spans multiple domains: societal, scientific, environmental, economic, and governance. Traditional evaluation approaches have inherent limitations because they either prioritize single-domain outcomes (such as data quality alone) or rely on linear causal frameworks that cannot capture the complexity of participatory processes, where goals and activities evolve iteratively (Kullenberg and Kasperowski 2016; Wehn et al. 2021a,b).
The need for more robust impact assessment and evaluation frameworks is increasingly recognized within the citizen science community (Davis et al. 2022). While diverse evaluation instruments and impact models exist (from domain-specific indicators to participatory pathway mapping), implementation remains fragmented (Schäfer et al. 2021). Many initiatives struggle with impact assessment and evaluation because of tradeoffs between what frameworks intend to measure and what can realistically be implemented within project constraints (Strasser et al. 2019). Furthermore, existing frameworks often fail to involve the people and institutions most affected by citizen science activities in defining and prioritizing the impacts that matter most to them (Passani et al. 2022). Other approaches are applicable only from the start of an initiative or are focused on science-driven citizen science initiatives, excluding ongoing and bottom-up citizen science initiatives.
The MICS project (European Union Horizon 2020 Measuring the Impact of Citizen Science, 2019–2022) addressed these challenges by developing and testing the Impact Journey (IJ) Approach: a participatory evaluation framework specifically designed to capture the complexity of citizen science impacts across multiple domains and in new as well as ongoing citizen science initiatives. Rather than treating evaluation as an ex-post activity, the IJ Approach positions it as an integral intervention for co-creating impact with IPAs (including project leads, participants, community partners, and beneficiaries). As such, it aligns with the tradition of a well-established body of evaluation literature, such as collaborative approaches to evaluation (Cousins and Whitmore 1998) and empowerment evaluation (Fetterman 1996) that have been practiced in a wide range of sectors to support participants in evaluating and improving their practice while fostering their agency. By combining domain-specific impact indicators with collaborative pathway mapping, the IJ Approach intends to enable participants in citizen science initiatives to visualize causal links between their activities, outputs, outcomes, and impacts; to identify unexpected changes; and to monitor and tailor progress iteratively throughout and beyond project implementation.
Since the MICS project finished, several similar participatory evaluation frameworks for citizen science have been published. Of particular note are the frameworks developed in the projects ACTION (Participatory science toolkit against pollution) and CoACT (Co-designing Citizen Social Science for Collective Action) (Passani et al. 2022; Kieslinger et al. 2022). For example, the framework by Passani et al. (2022) also considers impacts across multiple domains, draws on a pool of standard indicators, and involves citizen scientists in the process of evaluation. The co-evaluation approach developed by Kieslinger et al. (2022) emphasizes a higher degree of participation, with greater involvement from IPAs throughout all stages of the process and fewer pre-determined indicators or methods. At the same time, recent research has also emphasized that participatory evaluation frameworks are not culturally or politically neutral. Calyx and Finlay (2022), writing from an Australian and Indigenous-informed perspective, highlight the need to examine researchers’ positionality, language, institutional framing, and control over the evaluation process. They argue for approaches grounded in reflexivity, relationality, reciprocity, representation, and rights, particularly where frameworks developed in dominant institutional contexts are applied to diverse communities. The MICS IJ Approach combines a relatively extensive indicator framework with a structured participatory process. Compared with Passani et al. (2022), it relies on a broader and more formalized set of indicators, while compared with Kieslinger et al. (2022), it gives the research team a stronger role in framing, facilitation, and synthesis. This approach supports consistency and cross-case comparability, while leaving scope to consider how researcher input, participant ownership, and culturally and contextually diverse understandings of impact are balanced in practice.
This article critically reflects on the IJ Approach as both an evaluative tool and a method for participatory learning and adaptation. We present findings from applying the IJ Approach across five environmental citizen science initiatives operating in the United Kingdom, Italy, Hungary, and Romania in diverse contexts representing different levels of project maturity, governance models, and socio-ecological conditions. These case studies were selected to test whether the IJ Approach could accommodate this diversity and generate meaningful, comparable insights about citizen science impacts. Follow-up with the citizen science initiatives three years after the end of the MICS project also provides a rare insight into the long-term outcomes from the evaluation process and into the reasons for apparent lack of follow-through with the full approach during and after the MICS project lifetime.
Conceptual Approach
Theoretical foundation: from logic models to the Impact Journey Approach
Fully evaluating the impacts of individual citizen science initiatives requires an approach that moves beyond linear causality models to capture the complex, nonlinear pathways through which activities generate outcomes across multiple domains. Participatory evaluation has often drawn on logic models: frameworks that articulate underlying assumptions about how activities are expected to generate outputs and outcomes over short-, medium-, and long-term horizons (Gugiu and Rodríguez-Campos 2007). Logic models (also known as results chains or logical framework approaches) have proven valuable in citizen science contexts to demonstrate program effectiveness. However, existing logic model implementations in citizen science tend toward static, ex ante or retrospective application and often fail to engage participants meaningfully in the evaluation process itself (Skarlatidou and Haklay 2021). Closely related approaches, such as theory-of-change (ToC) frameworks (e.g., Van Es et al. 2015), present richer versions of the logic models that—in addition to specifying objectives, inputs, activities, outputs—help to identify, document, and monitor impact pathways. This is done through articulation of intermediate steps, assumptions, hypotheses, and assumed causal relations in the project design and in how envisaged changes may be achieved.
The IJ Approach builds on these conceptual foundations, particularly within participatory evaluation, but reconceptualizes evaluation as an intervention rather than solely an assessment mechanism. The IJ Approach treats evaluation as an integral part of collaborative learning and adaptation, positioning it alongside project activities as a vehicle for co-creating impact with IPAs in citizen science. This philosophy aligns with broader movements in participatory research and responsible research and innovation that emphasize shared decision-making in defining what impacts matter and how they should be measured (Kieslinger et al. 2022). The approach recognizes that citizen science initiatives operate within specific socio-ecological and governance contexts and that meaningful evaluation must be responsive to these contexts while remaining comparable across diverse initiatives. Participatory evaluation can strengthen relevance, learning, and ownership, while also raising questions of representation, power, and evaluator roles (Cousins and Chouinard 2012; Apgar and Allen 2021). The IJ Approach reflects these strengths and tensions through structured involvement of citizen scientists and other IPAs, and through facilitation.
Strategic thinking about impact and desired pathways of change should ideally be incorporated at early stages of planning a citizen science initiative. Nevertheless, applying the IJ Approach to understand the impact pathways of ongoing citizen science projects can function as a checks-and-balances mechanism that helps reflect on assumptions, strategies, and functioning of these initiatives (Van Es et al. 2015). Reflecting on assumptions made and decisions taken for achieving certain outcomes and impacts can provide ongoing initiatives with an opportunity to adapt their planning and implementation, and as a result, improve impact monitoring, evaluation, and learning processes.
The five impact domains
Central to the IJ Approach is the recognition that citizen science initiatives can generate impacts that extend far beyond contributions to scientific knowledge production via data collection. Drawing on a comprehensive literature review, Wehn et al. (2021b) identified five distinct yet interlinked impact domains of citizen science initiatives: (1) society, encompassing changes in both individual and collective (societal) values, understanding, action, and well-being (including relationships); (2) science and technology, including advances in scientific knowledge, methodological innovation, technological artefacts and changes in the capacity of the research system itself; (3) environment, that is, measurable changes in the state of environmental systems; (4) economy, covering economic benefits such as job creation and business development; and (5) governance, reflecting changes in decision-making processes, public policy, and institutional frameworks. This five-domain schema enables comparison across citizen science initiatives while avoiding the pitfalls of single-issue evaluations that overlook the systemic, interconnected nature of impacts. Each domain accommodates context-specific outcomes while maintaining a consistent conceptual framework. The combination of the ToC methodology with the five impact domains for the IJ Approach is illustrated in Figure 1.

Figure 1
Theory of change for citizen science initiatives.
Source: Wehn et al. (2020), compiled from Van Es et al. (2015).
Core components: indicators, pathways, and co-design
The IJ Approach comprises three integrated components that operate iteratively throughout the implementation of a citizen science initiative and beyond:
Domain-specific indicators: Indicators encompass outputs (concrete products such as datasets, publications, or educational materials), outcomes (observable changes resulting from outputs in the short term), and impacts (structural or systemic changes in the longer term). The IJ Approach draws on existing research and best practice that was combined in the MICS set of domain-specific indicators (see https://about.mics.tools/indicators), which can be adapted to individual initiative contexts. The MICS indicator library (see https://about.mics.tools/guidance/measure/indicators) comprises 83 indicators across the five domains. These indicators provide operational definitions of impact (concrete markers against which to assess progress). From the broad set of MICS indicators, locally relevant indicators can be selected by IPAs, ensuring relevance and feasibility while maintaining some basis for cross-case comparability.
Collaborative pathway mapping: Building on the ToC methodology, the IJ Approach employs pathway mapping as a participatory process. Through facilitated workshops and iterative dialogue, IPAs collectively develop visual representations of how their activities are hypothesized to generate outputs, outcomes, and impacts within and across the five impact domains. These pathways capture causal assumptions, identify enablers and barriers, and acknowledge external influences and unintended consequences. Pathway maps serve simultaneously as planning documents and evaluation tools, evolving as implementation progresses and assumptions are tested against reality.
Participatory learning and adaptation: The IJ Approach treats evaluation findings not as end-point judgments but as generative material for ongoing learning. Structured reflection on pathways (examining which assumptions are held, where unexpected impacts emerged, and why) feeds directly back into project design and execution. This creates a feedback loop in which evaluation serves as a vehicle for adaptive management and collective learning among diverse IPA groups.
Implementation process
The IJ Approach is operationalized through four main steps (see Figure 2).

Figure 2
Steps of the Impact Journey Approach for citizen science initiatives.
Step 1: Situation and context analysis involves detailed documentation of the facilitator of each citizen science initiative’s history, governance, IPA landscape, and existing impact monitoring systems. This step establishes baseline understanding and identifies the range of potential actors and beneficiaries whose voices must be included in subsequent participatory evaluation work.
Step 2: Impact pathways of the citizen science initiative and validation bring together IPAs in collaborative workshops. Using participatory tools and facilitated dialogue, this step generates a collectively-owned impact journey for the citizen science initiative. Participants articulate their theories about how activities will generate change and the assumptions that underpin them. Outputs include visual pathway maps and documented assumptions.
Step 3: The Impact monitoring scheme consists of negotiated indicator sets: Interactions are designed to help IPAs select indicators that best capture progress towards the outcomes and impacts, and define the details for monitoring prioritized indicators.
Step 4: Impact monitoring, evaluation, and learning (MEL) extends throughout and beyond implementation of a citizen science initiative. Regular monitoring against agreed indicators, periodic reflection workshops, and ongoing dialogue among IPAs generate evidence about whether hypothesized pathways are being realized. Ethnographic observation and supplementary data collection provide contextual understanding of process-level changes and unexpected outcomes that quantitative indicators alone might miss. Learnings and insights feed back into ongoing activities of the citizen science initiative.
The IJ Approach thus brings together established evaluation approaches (ToC) with participatory principles, iterative learning cycles, and multi-domain frameworks to create a pragmatic yet theoretically grounded approach to citizen science evaluation. Its emphasis on collaborative definition of impacts, together with predetermined metrics, addresses a longstanding criticism in the field: that evaluation frameworks often fail to measure what matters most to the people and communities conducting and participating in citizen science (Skarlatidou and Haklay 2021). By positioning evaluation as a participatory intervention, the IJ Approach aims to generate both robust evidence of impacts and meaningful engagement around questions of value and change.
Methods
The critical evaluation of the IJ Approach is based on a deductive comparative case study strategy, comparing the experiences with the implementation of the IJ Approach in the cases. The research team’s characteristics and values are summarized in the Supplementary File 1: Positionality statement.
Case-study sites—Impact Journey Approach implementation in the cases during project lifetime
The IJ Approach was applied in five cases incorporating citizen science within nature-based solutions (NBS) across Europe (two sites in the UK, and one site each in Hungary, Italy, and Romania; see Figure 3). The sites explored the applicability of the IJ Approach in regions with differing needs, contexts, and various levels of citizen science activity and purposes regarding NBS. For example, in the UK and Italy case studies, citizen science initiatives already existed that helped with monitoring of NBS. In Eastern European case studies, there were no existing structured citizen science activities, so the IJ Approach was embedded in efforts by the MICS project to co-design citizen science initiatives to monitor existing NBS (Romania) and to make the case for NBS (Hungary). A detailed description of the case study sites is available in MICS project Deliverable D4.5 (Wheatland et al. 2021).

Figure 3
Map showing location and details of the MICS case study sites.
Source: based on Wheatland et al. (2021). NBS: nature-based solutions.
The MICS IJ Approach was developed in 2020, and the case studies were guided remotely in applying the approach by social science researchers from the MICS project team over a period of approximately 18 months between 2021 and 2022. Specifically, each case was led by a MICS partner who facilitated the interactions among the case IPAs, that is, participants in the citizen science initiative and IPAs from relevant authorities.2 The implementation of the IJ Approach in the first case study (Outfall Safari, UK) served as a pilot; based on this experience, the approach (including workshop formats and preparation of inputs) was simplified before its application in the subsequent four case studies. Since the end of the MICS project in July 2022, no formal support on the IJ Approach has been provided by the former MICS consortium to the citizen science initiatives.
The timeline of the MICS project activities (January 2019—June 2022) was significantly disrupted by the emergence of the COVID-19 pandemic in 2020 and the continuing restrictions through 2021 and 2022. More specifically, the co-design and running of citizen science activities in each of the MICS case study sites was delayed. As a knock-on effect, this also delayed the IJ Approach in all cases. Furthermore, the timing and format of workshops was limited by social-distancing restrictions throughout the project. Overall, this meant that it was not possible for the MICS project team to guide the case studies through all four steps of the IJ Approach, and most reached only the third step.
Post-project enquiry among interested parties and actors (IPAs)
To assess longer-term use of the IJ Approach, the authors conducted a post-project enquiry three years after completion of the MICS project. In December 2025, the MICS research team contacted representatives of the citizen science initiatives to gather insights into the benefits the IJ Approach had provided over the longer term. This consisted of open-ended questions, allowing the case-study participants to reflect on their own personal experience and provide specific examples (see Supplementary File 2: Post-project enquiry questions). The questions were emailed in December 2025 to contacts from each of the MICS case-study sites. Where required, follow-up emails were sent in January 2026. In situations in which the participant was uncomfortable responding via email, the enquiries were carried out by phone call, with the researcher writing notes to capture the responses to each question.
Data sources and analysis
All research involving human subjects was carried out in line with the MICS project ethics procedure, and informed consent was obtained from all participants, both during the MICS project and for the post-project enquiry among case IPAs (see details in the Ethics and Consent section).
The critical reflection on the IJ Approach draws on a range of sources, namely the outputs and responses generated by the IPAs in each case study as well as the data set created by the MICS teams supporting each case study (see Supplementary File 3: Summary of data sources).
A tool for cross-case comparison was created in Excel by two authors. This matrix consisted of the steps of the IJ Approach and how each step was applied in each case, distinguishing the actual function of the step, case enablers and constraints, and the overarching result (whether the IJ step was achieved) (see Supplementary File 4: Implementation of the IJ Approach in 5 cases). For each of the four steps of the IJ Approach, the authors reviewed the case data sources (see Supplementary File 3: Summary of data sources) to write a summary that was validated by the case facilitators.
This enabled the comparison of the actual implementation of each step to its intended function and the identification of recurring patterns and context-dependent divergences (relating to resource demands, facilitation expertise, participant time, and analytical effort required in the cases to implement the IJ Approach).
Results
The following sub-sections summarize the main reflections on how each step of the MICS IJ Approach functioned as an evaluation tool, and the similarities or differences within each of the five case study sites. (Exemplary workshop designs and slides are included in Supplementary File 5: Summary of Outfall Safari Impact.)
Step 1: Situation and context analysis. This step was intended to provide an understanding of the historical setting of the citizen science initiative (especially for ongoing initiatives) and of the governance context, and to identify all relevant IPAs that should be invited to participate in the IJ Approach for this citizen science initiative. It was implemented in the cases as intended, namely as a structured desk review and IPA consultation exercise, tailored to each case. Information about the social, institutional, economic, environmental, and technological contextual settings and all IPAs connected to the citizen science initiative were extracted from the project websites and any published material available. This information was then validated by the main contact at the citizen science initiative. For the Romanian, Hungarian, and Italian cases, the main contacts were within the MICS project. Since Outfall Safari and Riverfly (both in the UK) were citizen science initiatives managed by organizations external to the MICS project, completion of the context analysis was dependent on good relationships with the respective coordinators to provide or validate this information. As an example, for Outfall Safari, this step mapped the political and regulatory landscape—including the Water Framework Directive, the partnership roles of Thames Water, the Environment Agency, and the Zoological Society of London, the absence of a standardized monitoring methodology of pollution outfalls, and the undocumented nature of much of London’s aging sewage network. It also traced the project’s origins back to the River Crane pilot (2014–2016). For the cases in Italy, Romania, and Hungary, the context analysis was blended into the co-design of these citizen science initiatives. Rather than reviewing an established program, the MICS team had to first co-design citizen science activities with local communities, and the situation and context analysis informed this process.
Step 1 played a foundational role in shaping the participatory process. It served to identify who needed to be involved in the Impact Journey process, and how. In the Outfall Safari case, step 1 helped to identify that citizen scientists had limited opportunity to engage with project coordinators or the regulatory agencies using their data. This insight directly led to the methodological decision by the MICS research team to run separate workshops (in step 2) for citizen scientists and project coordinators. For the cases in Italy, Hungary, and Romania, this step clarified that the governance contexts were characterized by prior participatory activities with mixed experiences, especially among civil society participants. It surfaced distinct barriers to participation in the MICS activities: lack of trust, community skepticism about NBS, communication failures in prior schemes, and limited understanding of citizen science.
Step 2: Impact pathways of the citizen science initiative and validation. This step was designed to provide a space for collaboration among IPAs via facilitated, participatory workshops to co-create a shared impact journey for the citizen science initiative, mapping how activities are expected to drive change, clarifying underlying assumptions, and producing visual pathway maps with documented insights.
Building on the findings from Step 1, the design of Step 2 was adapted to the unique contexts of each case. Accordingly, the format of activities were tailored in terms of the IPAs’ interests, prior relations, and motivations. The first round of separate workshops for citizen scientists and project coordinators in the Outfall Safari case surfaced the starkly different perspectives of each group, proving a valuable methodological choice. Outfall Safari was the first case study, and the IJ map was built entirely from scratch with IPAs (see Figure 4), as there was no pre-existing template to draw on. The process was highly participatory and valuable for surfacing divergent perspectives: Citizen scientists emphasized environmental and personal knowledge impacts, while project coordinators emphasized governance and data utility. However, it was resource-intensive and required multiple workshops (online, due to the COVID pandemic). The time required by the MICS team to process the almost 200 resultant comments was significant (see Figure 5). The model used for all subsequent case studies was that a draft IJ map was prepared by the MICS team in advance and validated in a workshop with IPAs.

Figure 4
Activities and outputs of the Impact Journey workshop in the Outfall Safari case study (UK). (a) Diagrammatic representation of how the activities were organized during the workshops; participants were asked to complete the tasks from right to left, effectively working back in time, first considering the long- and short-term impacts of the citizen science activities associated with their project (Activities 1 and 2), then what citizen science or other activities are required to achieve them (Activity 3). Participants were also asked to consider what assumptions were being made that the activities would be successful, for example, for the activity to “engage with local citizens,” the assumption would be that citizens are interested and willing to be involved. (b) Virtual whiteboard with comments from Outfall Safari IPAs as an example.
Source: Wheatland et al. (2021).

Figure 5
Steps taken by the MICS team to generate the draft Impact Journey map for the Outfall Safari case study (UK).
Source: Wheatland et al. (2021).
Overall, the simplified, team-led drafting of the IJ map was more efficient than open co-creation with IPAs in the Outfall Safari case over a series of workshops, for both the MICS team and the IPAs. However, the circulation of the draft IJ map prior to the workshop with IPAs, as in the Riverfly case, was counterproductive, as some participants were confused why they were being asked to validate something that had already been created. In the remaining three cases, the draft IJ map was presented and explained during the workshop rather than circulated in advance, to allow the facilitator to ease into detailed discussions about impacts of the citizen science initiative. IPAs generally found the IJ map easy to digest; this helped to open a dialogue about the impacts of their citizen science initiative. It is noteworthy that across cases, citizen participants tended to emphasize short-term learning benefits and environmental outcomes, whereas project coordinators in affiliated organizations (e.g., in NGOs and authorities) offered a broader range of comments across societal, governance, and institutional dimensions.
The implementation of step 2 faced several constraints, most notably those related to COVID-19. These disruptions were particularly acute in the Outfall Safari case, where all workshops had to be conducted online. In the other cases, a mix of face-to-face and online activities were possible, but these were competing for availability and attention of IPAs owing to intermittent COVID-19 disruptions. In the Hungary and Romania cases, the workshop was run back-to-back with a training on citizen science monitoring to foster participation by citizens. The face-to-face format with printed materials and sticky notes was specifically praised by IPAs for enabling engagement of citizens who were less comfortable with digital tools. The need to translate materials from/to English (the project language, due to the structural conditions and practical demands of working in an international project team) to case languages imposed further constraints, requiring additional time. Nevertheless, step 2 was completed in all five cases as intended, providing a clear learning trajectory on how to undertake this step. The open co-creation of the IJ map in the Outfall Safari case was rich but unwieldy; this was changed into a repeatable, efficient process. Nevertheless, the shift toward MICS team-led drafting of the IJ map created some tensions between quality and common understanding and ownership by the IPAs.
Step 3: Impact monitoring scheme. The design logic of this step was to arrive at a negotiated set of indicators and monitoring agreements to track progress towards outcomes and impacts. Constructing the impact monitoring scheme (IMS) was the most problematic element of the entire IJ process for Outfall Safari, and this experience shaped how the IMS development was approached in all subsequent case studies. During workshop #4 with Outfall Safari, IPAs were asked to complete a structured table identifying, for each prioritized impact, the indicator, method, frequency, responsibility, and feasibility. This activity was described as “abstract” by participants—particularly citizen scientists—who struggled with the concept of thinking about impact measurement in such a structured way. The MICS team had to take a substantially larger role in guiding discussions during the IMS activities than had been anticipated. The draft IMS produced by the three breakout groups required considerable post-workshop editing: Comments had to be consolidated and reclassified under appropriate column headings, and gaps were filled by the MICS team. De facto, the finalized IMS was developed by the MICS team considering the comments of participants as opposed to being developed by participants themselves.
For the subsequent four cases, the IMS activities were not included in the workshop at all, focusing instead solely on the IJ map validation and impact prioritization. The IMS was then constructed entirely by the MICS team following the respective workshop (see Supplementary File 6: IMS Outfall Safari), based on the impacts that IPAs had voted to prioritize. This approach reduced the cognitive burden on participants, while still ensuring co-production of the final IMS for their case. In Creek Rakos (Hungary) and Marzenego River (Italy), voting on priority impacts was done via email after the workshop. This worked well in the Marzenego River case (15/18 responses) but less so in Creek Rakos (4/25 responses).
This step of the IJ Approach was fully completed for two of the five cases (Outfall Safari and Marzenego River) during the lifetime of the MICS project, with Outfall Safari going through two formal versions and a partial implementation trial. For the IMS, relevant indicators for prioritized impacts were selected from the broader MICS indicator library. Governance-related indicators (i.e., tracking policy change, political pressure, inter-institutional collaboration) were consistently the hardest to operationalize and frequently received low feasibility ratings or were deferred during the implementation trial. By contrast, indicators based on existing data collection (survey counts, volunteer numbers, annual reporting) were more readily embedded. The IMS of the Outfall Safari case is provided in the Supplementary File 6: IMS Outfall Safari.
Step 4: Impact monitoring, evaluation, and learning. This step is intended as a recurrent and long-term process in which IPAs continue to hold dedicated workshops to review evidence as to whether the intended impact pathways are being achieved, take into account new developments and unexpected changes, and adapt ongoing activities of the citizen science initiative accordingly. None of the five case studies had reached a stage of sustained implementation of this step by the time the MICS project finished, owing to the delays encountered from the COVID-19 pandemic. Therefore, the analysis of this step draws on three sources: the experience with the process of building towards this step (the impact MEL of the citizen science initiatives), its partial implementation by Outfall Safari in spring 2022, and the results of the post-project enquiry in late 2025 and early 2026.
As an established citizen science initiative operating since 2016, Outfall Safari could already demonstrate a body of impact evidence before the MICS IJ Approach was implemented. However, existing metrics were focused on the environment domain, were short-term in nature, and did not constitute an assessment of the overall impact of the initiative. The trial of the Outfall Safari IMS implementation showed mixed results. A questionnaire to capture enhanced citizen science knowledge was met with a very low response rate. Information on prioritized pollution outfalls was available as before (reported annually), but permission could not be obtained for the release of data about annual remediation actions. Attributing any specific remediation actions by regulators to reports by Outfall Safari citizen scientists also remained an unresolved challenge as retrospective data did not allow this crucial impact pathway to be traced. The monitoring of these indicators and other governance indicators relating to increased political pressure, to improved decision-making, and to improved policy or legislation, were all deferred. The Outfall Safari coordinator acknowledged the value of the indicators but was limited by the time required for additional data collection and further relationship-building with key IPAs. In contrast, indicators drawing on existing data, such as volunteer numbers and survey counts, could be activated relatively quickly. A key learning from this trial was, therefore, that the IMS helped to organize existing data from operationally simpler indicators, but new and complex indicators—for example, those related to governance and policy—faced structural and institutional barriers that could not be resolved within the MICS project timeframe.
The results of the post-project enquiry indicated the difficulty in sustaining the IJ Approach over the long-term. It was not possible to get a response from either of the UK cases because of staff turnover in the citizen science initiatives, but 20 responses were collected from across the IPAs in the other three cases. None of the three responding cases were still actively using the MICS IJ Approach, and most structured activities appear to have stopped shortly after the end of the MICS project. However, many of the respondents said that the bringing together of IPAs as part of the IJ Approach had had long-term benefits and that participating in the process had shaped the way they now viewed and planned for impacts in other contexts.
Discussion
We applied the IJ Approach in five diverse cases (rather than a homogeneous set of cases) during a period characterized by the COVID 19 pandemic. While these experiences do not provide the basis for generalizable results per se, they provide valuable insights into the complexity of citizen science impacts and practical and conceptual challenges of participatory evaluation of citizen science impacts.
At a content level, the IJ Approach helped to effectively illuminate the perceived causal links between strategies, outcomes, and impacts in each case. The environment domain was anticipated to dominate in all cases, given the monitoring focus, but the IJ workshops consistently revealed a richer picture.3 Society and governance impacts were significant in nearly every case, and without the IJ workshops, these would likely have been underestimated. Citizen scientists tended to prioritize personal development and environmental outcomes, while project managers and authorities placed greater weight on governance, data utility, and institutional change. Across all five case studies, the IJ maps organized strategies into two or three recurring pathways. Most identified versions of fostering stakeholder engagement (building the relationships and networks needed to sustain citizen science and translate activities into wider change) and from data to action (ensuring that citizen science monitoring generated evidence that could be used to drive environmental and policy change) as their core strategic pillars, with some cases adding a third around driving future citizen science initiative development (disseminating the tailored citizen science approach to new areas and organizations, expanding the geographic reach of proven approaches). The relative weight placed on each strategy varied by regional context. In the UK cases, where the citizen science initiatives Outfall Safari and Riverfly were already well established, strategies focused more on deepening impact measurement, expanding reach, and using evidence to influence governance. In the Marzenego case (Italy), the strategy placed particular emphasis on school engagement as a vehicle for both monitoring and long-term culture change. In Creek Rakos (Hungary), it was heavily oriented toward awareness raising and building public and political support for restoration, given that no NBS had yet been implemented. In the Carasuhat Wetland (Romania), it necessarily included trust-building as a foundational step before any monitoring or impact work could proceed.
A notable cross-case observation was the enabling role of the IJ workshops. In the MICS cases, the IJ workshops were often the only, or one of the few, occasions in which citizens and scientific experts interacted directly with public authorities. By opening dialogue between citizen scientists and key IPAs, the workshops themselves can be a strategic intervention that can serve to extend beyond impact prioritization and planning to co-creating future impact. Relational outcomes of these social interactions were cited as a key long-term benefit of the IJ Approach in the post-project enquiry. These findings are consistent with the participatory evaluation research literature, which has shown that evaluation processes can generate value beyond the production of findings, including dialogue, shared understanding, organizational learning, and strengthened relationships among participants (Cousins and Chouinard 2012), especially transformative types of participatory evaluation (Cousins and Whitmore 1998; Cousins and Alborhamy 2025).
Arriving at a shared understanding of the means for possible transformation in each case posed specific facilitation challenges. Discussions about outcomes and impacts of citizen science initiatives can seem abstract or “academic,” particularly for citizen scientists, reducing their ability or willingness to engage in the very activity that is intended to be done with them, not about them. This was the case, even though the MICS team translated academic terms into more accessible language (for example, referring to short- and long-term impacts rather than explaining the conceptual distinction between outcomes and impacts). In principle, involving IPAs in the drafting of the IJ map can help develop familiarity with both the concepts and content of the IJ map. In the MICS cases, we found that starting a workshop with a draft IJ map was better than either starting from a “blank page” or circulating a draft IJ map beforehand. Sufficient time during the workshop is required to work through the components of the IJ map and explore linkages. In several MICS cases, holding such discussions back-to-back with hands-on citizen science (training) activities achieved effective participation. Similarly, the IMS construction process has to ensure that the selection of citizen science impact indicators is based not only on what would be ideal to measure, but on commitment from IPAs to generate and/or release the relevant data.
The full IJ Approach represents a substantial commitment in terms of facilitation, participant time, and analytical efforts at a level not replicable in all contexts. The evolution of the IJ Approach across subsequent MICS case studies reflects our efforts to retain the core value of participatory evaluation while reducing its demands. The key efficiency gains achieved were the pre-drafting of IJ maps and impact monitoring schemes by the MICS team, the reduction to single validation workshops where possible, the removal of in-workshop content-development activities, and the use of email voting to capture prioritization preferences without extending workshop length. These adaptations meaningfully reduced participant time and facilitation complexity, at the cost of tradeoffs, as the Riverfly experience demonstrated. By preparing the initial IJ maps and monitoring schemes, the MICS team inadvertently retained greater control over the framing of impact pathways, the use of evaluation concepts, and the synthesis of participants’ contributions. This created a trade-off between sharing control of technical decision-making and the feasibility of deeper participation, two key dimensions that characterize participatory evaluation (Cousins and Whitmore 1998). Despite these adaptations, it also remains unclear whether smaller citizen science initiatives are able to apply the IJ Approach without hands-on guidance or support from experienced facilitators, as no evidence was obtained that any of the MICS cases continued to apply the IJ Approach in the long term, beyond the project. From a participatory evaluation perspective, this is not surprising: Participatory evaluation does not occur by chance, but requires expertise and dedicated effort to bring together and maintain in dialogue relevant IPA groups (Apgar and Allen 2021). This suggests robust and longer-term facilitation (external to the CS initiative) of participatory evaluation is necessary for the implementation of the IJ Approach.
An inherent strength of the IJ Approach is its flexibility. It can be applied to both existing and new citizen science initiatives; for example, the approach can be used at the beginning of a citizen science project to help plan activities in order to ensure that they engender the desired impacts, or it may be used as an evaluative tool for an ongoing citizen science initiative to create an overview of its impact journey, take stock, and make strategic choices for achieving (additional) outcomes. Workshops may be run with all IPAs present or with different IPA groups separately. An IJ map may be drafted from scratch, including IPA input, or may be prepared by the citizen science initiative coordinator and later validated through the involvement of IPAs. Different steps of the IJ Approach can be emphasized based on the interests and motivations of the IPAs or the aims of the coordinator of the citizen science initiative. For example, in some cases, the main reason may be to generate a robust IJ map. In others, the primary motivation may be its use as a collaboration tool to bring IPAs together and provide feedback on the scope of the impacts of the citizen science initiative and the overall effect that the activities of the citizen scientists are having. The elements of the IJ Approach tested in the MICS cases can be drawn upon selectively (as a toolbox) or as a complete set. Nevertheless, rather than being merely a toolbox, we argue that the IJ Approach provides the basis for meaningful and continued engagement of citizen scientists and other IPAs around questions of value, purpose, and pathways for change.
Conclusion
This paper has presented a critical reflection on the IJ Approach developed and tested by the MICS project. Our findings confirm the diversity and context-dependence of citizen science impacts, and by implication, the need for achieving impacts with citizen science via distinct and tailored strategies (Wehn et al. 2021b; Davis et al. 2022; Kieslinger et al. 2022; Pateman and West 2023).
Evaluation frameworks of citizen science outcomes and impacts frequently leave out those people and institutions most affected (Passani et al. 2022). However, in line with previous studies (Passani et al. 2022; Kieslinger et al. 2022), our work has demonstrated that involving citizen scientists in participatory evaluation is not straightforward. The citizen scientists were often more interested in hands-on environmental monitoring and in creating change—arguably the most important outcome of transformative participatory evaluation (Cousins and Alborhamy 2025—than in identifying and tracking their journey towards impact. Institutional actors typically shied away from committing to changes in data-use pathways, structured data flows, and exchange mechanisms, with impact journeys remaining largely aspirational. Moreover, participatory evaluation approaches such as the IJ Approach demand considerable resources and expertise—which many, if not most, citizen science initiatives neither possess nor have access to. This highlights the need for participatory evaluation approaches and support that can further strengthen the intervention element of evaluation, co-creating impact with IPAs. These should provide more leverage for fostering institutional willingness and strengthen the capacity of institutional actors to formally embed, value, and act upon the knowledge being generated by the citizen science initiatives. We acknowledge that pointing to the need for external facilitation expertise to support participatory evaluation inevitably raises issues with resourcing such activities.
A key strength of the IJ Approach developed and tested by MICS is its flexibility, which means it may prove useful, in part or as a whole, for participatory evaluation for existing or new citizen science initiatives. By fostering a shared understanding (and illumination of different perspectives) of the impact goals of a citizen science initiative, the IJ map provides a critical foundation for holistic and effective impact evaluation and transformative change over time.
A promising avenue for future research and practice is to explore the integration of citizen science into formally recognised participatory governance processes, such as the Italian River Contracts. Their long-term and open-ended nature, supported by formal collaboration among public authorities and IPAs, may provide favourable conditions for embedding citizen science within sustained processes of knowledge co-creation and environmental governance. In this context, the IJ Approach could support the assessment of the outcomes of specific citizen science activities and their contribution to broader participatory governance processes, while facilitating learning, continuous improvement, and the longer-term institutionalization of citizen science in freshwater ecosystem management. Moreover, testing the IJ Approach outside the EU’s shared regulatory and institutional context remains an important direction for future research, and would help establish whether our findings generalize beyond European governance settings specifically.
Applying the IJ Approach challenged the MICS team’s initial assumptions about what counts as impact and how unexpected impacts should be evaluated. IPAs emphasized different combinations of social, environmental, economic, and governance outputs, outcomes, and impacts (Wheatland et al. 2021), demonstrating that impact could not be defined by researchers or citizen science coordinators alone. The process also required the team to adapt terminology, facilitation styles, and workshop formats to different languages, levels of experience, and local relationships, as is recommended best practice in participatory evaluation (Apgar and Allen 2021). These adaptations made the researchers more aware that their own framing and methodological choices may have influenced which perspectives were brought forward and how the resulting knowledge was shaped. In this way, the case studies did not simply test the IJ Approach; they also refined the researchers’ understanding of collaborative evaluation as an iterative process of mutual learning (Cousins and Alborhamy 2025).
Supplementary Files
Supplementary File 4
Implementation of the IJ Approach in 5 cases. DOI: https://doi.org/10.5334/cstp.1025.s4
Notes
[1] The term IPA encompasses stakeholders (i.e., those affected by, interested in, and/or able to affect a project) and actors (persons, groups, or organizations acting within a system of interest) (based on Reed et al., 2009; Newton and Elliott, 2016). We use “IPAs” instead of “stakeholders” because the latter foregrounds interest and affectedness while leaving agency implicit, whereas in citizen science the parties concerned do not merely have a stake in the initiative but constitute it. IPAs also keep visible a distinction that recurs throughout our cases: Those who act within the initiative and those with an interest in its results were frequently not the same people, and their expectations of impact differed accordingly.
Ethics and Consent
This study presents results from human subject research generated during the MICS project lifetime and via the post-project enquiry (described in the Methods section). The MICS project had ethical oversight by an ethics committee of the European Commission during the proposal evaluation. During the implementation of the MICS project and the work in the MICS case studies reported on in this paper, research involving human subjects was carried out in accordance with the MICS project ethics procedure (described in the report “D6.1 Ethics requirements: Participant informed-consent procedures” (Wehn, Williams and Ceccaroni, 2019). This detailed the procedures and criteria used to identify/recruit research participants and the informed consent procedures to be implemented for the participation of humans. Informed consent was obtained from all participants, both during the MICS project and for the post-project enquiry among cases IPAs. Templates of the informed consent forms and information sheets (in language and terms intelligible to the participants) were kept on file by the respective case studies leads.
Acknowledgements
We thank the citizen science practitioners involved in the five MICS case studies who contributed to this work.
Author Contributions
Uta Wehn – Conceptualization, Methodology, Visualization, Investigation Analysis, Writing – original draft.
Bruna Gumiero – Investigation, Analysis, Writing – review & editing.
Balazs Kozak – Investigation, Analysis, Writing – review & editing.
Hannah Joyce – Investigation, Analysis, Writing – review & editing.
John Wheatland – Investigation, Analysis, Writing – review & editing.
Albert Scrieciu – Investigation, Analysis, Writing – review & editing.
Stephen Parkinson – Methodology, Investigation, Analysis, Writing – original draft.
Mohammad Gharesifard – Methodology, Investigation, Writing – review & editing.
Luigi Ceccaroni – Writing – original draft, Visualization, Writing – review & editing.