Introduction
Evaluation and impact assessment have become central concerns in citizen science, as participatory research approaches increasingly seek to demonstrate their value for science, society, and socio-ecological systems. Over the past decade, a growing body of work has proposed frameworks, principles, and tools to support the systematic evaluation of citizen science processes and outcomes. Among these contributions, the citizen science evaluation framework (Kieslinger et al. 2017, 2018)—referred to as CSEF in the following—has been widely referenced and applied across diverse contexts.
The CSEF has been disseminated through multiple publication formats. In addition to its inclusion in an edited book volume (Kieslinger et al. 2018), which is the basis for this study, there was an open-access version as pre-print (Kieslinger et al. 2017) with 3,931 views and 3,697 downloads (May 2026). This open archive version is further linked through a Zenodo record, which has been accessed 1,361 times and downloaded 947 times (May 2026).
Its sustained visibility suggests that it has played a formative role in shaping how evaluation and impact assessment are conceptualized and operationalized within the field, and underscores the importance of examining not only how often it is cited or downloaded, but how it is interpreted and used.
While numerous publications cite the CSEF, less is known about how it has been taken up in practice. This article addresses this gap by presenting a systematic analysis of how the book chapter on the CSEF has been referenced, used, and transformed. Through a structured review of publications that cite or substantively engage with the framework, we examine, first, which aspects of the framework are explicitly mentioned or discussed by other scholars. Second, we analyse which components of the CSEF are practically applied in evaluation and impact assessment activities, shedding light on how conceptual elements are translated into operational practices. Third, we explore how the CSEF and its indicators have informed the development of subsequent frameworks, guidelines, and evaluation toolkits, thereby tracing its influence beyond direct citation.
Building on our literature review, we identify core considerations that we didn’t include in the original framework and suggests an adapted and context-sensitive approach via pathways to evaluation and impact assessment. In this way, this article contributes both a reflective analysis of framework uptake and a forward-looking perspective on how evaluation frameworks can remain responsive to changing practices and expectations.
State of the Art: Evaluation and Impact Assessment in Citizen Science
The CSEF is a framework that contains process- and outcome-based indicators of evaluation in citizen science across three dimensions—the dimension of individual participants, and the scientific and the socio-economic levels. In literature, evaluation is commonly understood as the systematic assessment of project processes and outcomes against explicit or implicit criteria, serving both formative and summative purposes (Weiss 1998). Citizen science is framed as generating impacts across multiple domains—science and technology, society, environment, economy, and governance (Schaefer et al. 2021; Wehn et al. 2021). Impact assessment thus extends beyond project-internal goals to wider contributions to sustainability (e.g., United Nations Sustainable Development Goals [SDGs]), public participation, political impacts, and structural change in the science-society relationship (Liu et al. 2023; D’Andrea et al. 2022; Turrini et al. 2018). Within citizen science, evaluation has increasingly been seen as a learning process that supports self-reflection, adaptive management, and the iterative improvement of participatory research practices (Davis et al. 2022; Schaefer et al. 2021).
Next to the CSEF, several influential frameworks now shape the field (Wehn et al. 2021; Passani et al. 2020), and more recent contributions extend this conceptual work by translating frameworks and principles into operational tools and infrastructures (Ceccaroni et al. 2022), which are discussed in more detail later in this section. We also found in the literature the development of validated scales for outcomes on involved participants, such as, for example, interest in science, self-efficacy, motivation, behavior, attitudes, or skills (Phillips et al. 2018), all of which allow for an in-depth evaluation of indicators suggested by the CSEF.
In parallel, institutional-level initiatives such as TIME4CS foreground evaluation as part of broader organizational change processes. Rather than focusing exclusively on project-level outcomes, these approaches emphasize evaluation capacities within research-performing organizations, including governance structures, incentive systems, and support mechanisms, as summarized by one of the authors in Schaefer et al. (2023). This marks an important expansion of evaluation beyond individual projects towards the conditions under which citizen science can be sustained and scaled.
Scientific outcomes of citizen science activities have most frequently been assessed so far, and traditional metrics such as data quality, publications, and contributions to research agendas are increasingly amended by broader scientific impacts, including methodological innovation, increased openness of research processes, and changes in relationships between science and society (Bonn et al. 2018; Kasperowski et al. 2017). Citizen science is also increasingly studied as an object of research in its own right, with methodological and epistemological questions becoming a growing area of scholarly attention (Schaefer et al. 2021).
Amongst participant-level outcomes, learning outcomes are the ones most frequently evaluated. Reviews consistently show that content knowledge gains are the most reported effects of participation, while more complex outcomes—such as changes in attitudes, behavior, and self-efficacy—are less frequently and less systematically assessed (Aristeidou and Herodotou 2020; Peter et al. 2019). To close this gap, Somerwill and Wehn (2022) reviewed recent scales and identified five validated approaches for evaluating changes in environmental attitudes, behavior, and knowledge that can be directly adopted in citizen science.
Evaluation of impacts on socio-ecological systems and governance structures remains comparatively underdeveloped (Schaefer et al. 2021) but has gained renewed attention through links to the SDGs. Recent analyses suggest that citizen science can contribute to monitoring a substantial proportion of SDG indicators, particularly where community-generated data complement official statistics (Fraisl et al. 2025). Liu et al. (2023) investigated how urban citizen science contributes to SDGs and highlighted the need to align citizen science indicators with SDG metrics to mainstream citizen science contributions.
Complementary to standardized scales and classical data collection instruments like surveys, interviews, and focus groups, new innovative, playful instruments aim to improve response rates and align evaluation with citizen science’s participatory ethos (Allf et al. 2025). Recent project-level work also emphasizes inductive and qualitative approaches integrated into structured frameworks (Janssen et al. 2025), illustrating the need for mixed-method and flexible evaluation designs to capture complex, emergent outcomes.
Across the literature, several unresolved tensions persist. These include the balance between standardization and contextual flexibility. There is progress towards common domains, indicators, and validated scales, but strong calls remain for context-sensitive, mixed-method, and participatory approaches that can capture emergent and unanticipated outcomes (Janssen et al. 2025; D’Andrea et al. 2022). There is the difficulty of evidencing long-term and systemic impacts, where longitudinal studies are still rare (Aristeidou and Herodotou 2020; Peter et al. 2019). Assessment guidelines are applied unevenly across the impact domains, with economic and governance aspects (Wehn et al. 2021) or unintended impacts (Walker et al. 2020) being seldom systematically assessed. Current debates suggest that future advances in evaluation and impact assessment will depend less on the creation of new frameworks and more on the critical adaptation, combination, and contextualization of existing approaches (Wehn et al. 2021).
Methodology
This study applies a qualitative interpretative review design to analyze how the CSEF has been taken up, referenced, adapted, and further developed within the citizen science community. The methodological approach combines systematic identification of formal citations with purposive inclusion of additional relevant materials known to the authors, reflecting the heterogeneous ways in which frameworks circulate in academic and practice-oriented contexts.
Positionality statement
We want to make transparent our relationship to the object of analysis and the situated perspectives from which this work emerges. Such transparency is particularly important for this publication, as it examines the uptake, interpretation, and transformation of an evaluation and impact assessment framework that we ourselves co-authored.
We were two of the co-authors of the original citizen science evaluation framework, published in 2017 as an open archive pre-print (Kieslinger et al. 2017) and 2018 as a book chapter (Kieslinger et al. 2018), alongside several other contributors. We wish to explicitly acknowledge the intellectual labor of our co-authors, whose critical engagement substantially shaped the structuring of the CSEF. With temporal distance, it is neither possible nor productive to disentangle individual contributions in detail. However, the framework can be traced back to earlier evaluation work developed in 2015 in the context of a commissioned project for the Austrian Ministry of Education and Science (Kieslinger et al. 2015). As such, we have been closely associated with successive iterations of the CSEF over nearly a decade and have developed a strong relationship to it.
Over time, external feedback has indicated that many actors within the European citizen science community perceive us as the principal initiators or custodians of the framework. While we recognize and value this attribution, we also emphasize that the CSEF emerged through collaborative processes and benefited from diverse perspectives. At the same time, our long-standing engagement with the framework has prompted us to critically examine how it has travelled and been referenced, adapted, and used within the citizen science community.
It is also important to situate ourselves in terms of our broader research context. We are two female researchers embedded in European-funded research environments and have been actively involved in multiple European research projects. Our professional experiences, networks, and epistemic orientations are shaped by this context. Consequently, our analysis reflects a predominantly European research and funding perspective. We acknowledge that this positionality may limit our sensitivity to global, ethical, and socioculturally diverse interpretations and uses of evaluation frameworks in citizen science. We therefore present this work as a situated analysis, informed by our specific trajectories and locations within the field, rather than as a universal or exhaustive account.
Data sources
The primary data source consists of publications that explicitly reference the 2018 published CSEF (Kieslinger et al. 2018), which was chosen for this study because the 2017 published CSEF (Kieslinger et al. 2017) is only a pre-print. Google Scholar was selected as the main data source for the screening of references due to its broad coverage of peer-reviewed literature, conference proceedings, reports, and other scholarly outputs relevant to citizen science. At the time of analysis (December 2025), this search resulted in 125 entries in Google Scholar that cite the Kieslinger et al. (2018) article.
To complement this citation-based corpus, based on the authors’ knowledge of the uptake of the CSEF in European research projects, four additional publications were added: two of our own project reports of the CAPTOR (Collective Awareness Platform for Tropospheric Ozone Pollution) Schaefer et al. 2019) and the CoAct projects (Kieslinger et al. 2022), presenting the results of applying the CSEF in a mid-term and end-term self-assessment; one project report describing the evaluation frameworks of the MICS (Measuring the Impact of Citizen Science) project (Ceccaroni et al. 2022); and one project report describing the evaluation framework of the ACTION project (Passani et al. 2020), the latter two of which are strongly based on the CSEF introduced here.
Further instances of uptake exist beyond the scope of this study, particularly in informal settings, unpublished materials, or internal project documentation. These forms of use are not comprehensively captured here, but their existence is recognized as part of the broader diffusion of the CSEF.
Data analysis
The combined data sources resulted in a synthesis corpus of 129 publications, which were screened and analyzed. From the 129, three publications had to be excluded because they were written in Chinese, Estonian, and Indonesian. Eight publications had double entries, which means that they were listed two times with slightly different titles or in different formats, while being the same article. One article listed Kieslinger et al. (2018) in its references but no related content was found in the content of the article.
The 117 remaining articles were thoroughly studied to address the first two objectives of the study: identifying how the framework has been referenced and how its concepts have been interpreted and used across different contexts. All publications in this corpus were subjected to a qualitative screening process in which the two authors individually examined the nature of the reference, the evaluation dimensions addressed, and the role of the framework within the respective work (e.g. conceptual framing, methodological guidance, or evaluative benchmark) and discussed the results of this screening to come up with a common understanding of the aspects previously mentioned.
From this initial corpus, a subset of 17 publications were selected for a second, more in-depth round of analysis to address the third objective of the study, focusing on transformation, adaptation, and extension of the framework. These publications were selected because of their depth of engagement with the framework, which went beyond referencing it or using selected indicators for assessing citizen science activities. These articles showed evidence of modification or reinterpretation of the CSEF’s components and contributed to emerging evaluation practices in citizen science. After selection, these publications were analyzed to identify recurring patterns and innovations in the use of the framework.
The publication selection process is summarised in the following flowchart (Figure 1).

Figure 1
Flowchart of the publication selection.
Development of framework adaptations
Based on the findings from the qualitative analysis, suggestions for adapting and extending the original framework were developed. Rather than proposing an entirely new framework, the adaptations aim to respond to empirical evidence of how the CSEF has been appropriated in practice and to address gaps or limitations revealed through its uptake.
Results
The reviewed literature demonstrates that the CSEF has been taken up in diverse and uneven ways, ranging from critical engagement and selective adoption to partial integration and inspiration for new evaluative approaches. Across the corpus, most publications refer to the CSEF as a point of orientation within the state of the art, while a smaller but analytically significant subset engages more deeply with its structure, dimensions, or indicators. Following the qualitative analysis described above, the resonance of the framework was mapped into three main groups: 1) those that make purely a reference to the framework, either in the introduction, the state-of-the-art, or the discussion; 2) those who apply the CSEF or parts of the framework, such as specific indicators; and 3) those who adapt the framework and expand it or make suggestions for adaptations. The authors discussed the attribution of each paper to one of these groups and assigned it to the groups based on mutual agreement only. Each paper is attributed to only one of the three groups. Figure 2 shows the three main groups and the main usage of the CSEF in each of these.

Figure 2
Reference, application, and uptake of the citizen science evaluation framework CSEF (Kieslinger et al. 2018).
In Table 1 we show the distribution of these three groups amongst the 117 analysed publications.
Table 1
Number of publications belonging to one of the three identified groups.
| IDENTIFIED GROUPS | NUMBER OF PUBLICATIONS |
|---|---|
| Reference the framework | 77 |
| Apply and operationalize the framework | 23 |
| Adapt and expand the framework | 17 |
References to the evaluation framework
As shown above, 77 publications referenced to Kieslinger et al. (2018). Amongst those, we find many publications that refer to the CSEF’s three citizen science impact dimensions in their state-of-the-art descriptions, and introduce the CSEF as an instrument to evaluate citizen science. The authors of these 77 publications discuss potential benefits of citizen science activities for science, individuals, and socio-economic systems in general terms and point to the CSEF for reference. Others point to one specific dimension of the CSEF or they highlight specific outcomes mentioned in the framework. The most referenced dimension of impact is the participant dimension, where scholars refer, for example, to the increased ownership amongst participants, learning and science literacy, the importance of communication, and feedback loops to participants. We also find publications where the authors build their argument for discussion on very specific claims found in Kieslinger et al. (2018), such as the call for a more systematized evaluation, the need for more approved instruments to evaluate citizen science, or clear indicators that are required and/or currently elaborated. Finally, scholars refer to the CSEF in their reasoning to distinguish between process-based and impact-based evaluation or to stress the importance of not only measuring project goals but also participants’ goals.
Operationalization and uptake of the framework to evaluate citizen science activities
Amongst the 23 works that refer to the practical uptake and use of the CSEF we see a clear tendency towards a partial uptake, for example, the application of selected indicators. Nonetheless, three articles refer to having used the complete CSEF to assess their citizen science initiatives. Heinisch (2021) used the entire CSEF along the three dimensions and both process and outcome indicators to self-assess a citizen science project related to linguistics that investigates German use in Austria with the help of the citizen scientists involved. The self-assessment resulted in several identified strengths of the project and potential areas for improvement. Unterfrauner et al. (2021) introduce activities to evaluate four citizen science activities within the REINFORCE (Research Infrastructures for Citizens in Europe) project, and refer to future use of the CSEF for internal self-assessment, where they planned to use the whole set of indicators to reflect on the process and outcomes of their project. Three years later, Angelidakis et al. (2024) introduce the four REINFORCE citizen science activities together with strengths, weaknesses, and generated impacts as a result of this CSEF-based self-assessment.
Most of the screened articles implemented the CSEF partially, meaning they individually selected indicators from the process and outcome dimension. While some authors operationalize nearly the complete CSEF, others select a few indicators of one dimension only, partly combining them with indicators from other frameworks. Wu et al. (2022) used a slightly adapted version of the CSEF to evaluate 15 water-related citizen science projects in China. For their evaluation, they used both process and impact indicators in all three dimensions, but reduced the number of indicators per dimension. They concluded that projects should perform well in at least two of the three dimensions to be sustainable. Also, Sona (2021) operationalized nearly the complete CSEF to compare two humanistic citizen science activities, slightly adapting the framework to their purpose by grouping different indicators in a lower number of sub-categories. Zhang et al. (2023) used the CSEF to collect feedback on a seagrass monitoring project in China. They selected process and impact indicators from all three dimensions. The authors conclude that evaluation was especially useful for them to get feedback on the process, while measuring the impact was perceived as being more challenging.
Others used the CSEF as a source to develop process-related evaluation instruments. Hughes et al. (2022), for instance, merged the ECSA 10 Principles of Citizen Science (ECSA 2015) into six principles, and used indicators from the CSEF to assess the adherence to these principles in the Grizzly Bear Citizen Science project. The indicators used in their study are mainly process related and cover all three dimensions. Howells et al. (2024) state that they will use a combination of the ECSA 10 Principles with the CSEF to do a process evaluation of the Rapid Eczema Trials. Seus et al. (2023) created a toolbox for embedding responsible research and innovation practices in shared Green Deal activities, related to, for example, clean energy, circular economy, sustainable food, and climate action. They based this toolbox on several existing frameworks, one of which is the CSEF, where we find several process-related indicators of the CSEF integrated. Kocman et al. (2020) used only the participant dimension from the CSEF to assess the citizen science activities on the topic of noise exposure and health at a primary school in Ljubljana, Slovenia. They applied all process-based indicators in the form of a self-assessment and chose selected criteria from the CSEF to reflect on participants’ impact.
In contrast, Hoedoafia et al. (2024) created an evaluation framework to self-assess their radon research project that focuses more on impact assessment than on process evaluation. They used the three outcome dimensions of the CSEF and added one more dimension on learning gains for researchers. The process evaluation is covered via one dimension for process evaluation. These five dimensions, together with the detailed indicators per dimension, largely reflect the work of the CSEF, slightly adapted and amended with questions that specifically focus on learning gains for researchers.
Several scholars used selected indicators of the CSEF as a basis for the development of concrete evaluation instruments. Schaaf et al. (2024) created an online questionnaire to collect feedback from participants of citizen science activities related to rare diseases. They used the impact questions of the participant dimension of the CSEF as inspiration for the feedback questionnaire, which covered indicators like satisfaction with the organized workshop, transparency of communication, opportunities for participation, suggestions for collaboration, perceived influence on the implementation, and learning effects. Also, Andresz et al. (2023), Weisberg et al. (2023), and Kox et al. (2021) were inspired by the CSEF when developing questionnaires to investigate the impact of their citizen science projects on participants, like knowledge, attitudes, behavior, ownership, motivation, and engagement.
Next to these references from international scholars, we, the authors, have also performed evaluations and impact assessment of citizen science activities based on the original CSEF. In CAPTOR, a citizen science project concerned with air quality in three European testbed regions, we transferred the CSEF into a self-assessment tool in the form of an online questionnaire where items had to be rated by the project team. This served as the basis for reflective discussions amongst core project actors in the middle and at the end of the project. The discussion and agreement about the ratings as well as the answers to the open questions helped the project to identify strengths and shortcomings and document project evolution along the three dimensions (Schaefer et al. 2019). A similar self-assessment process was applied in three citizen social science initiatives within the context of the CoAct project (Kieslinger et al. 2022). Visualizations (Figure 3) of the self-assessment outcomes helped in both projects to identify specific focus areas and strengths of each initiative, whether it was driven by, for example, the need for social change or the generation of scientific outputs.

Figure 3
Visualization of self-assessment in the CAPTOR (Collective Awareness Platform for Tropospheric Ozone Pollution) project (Schaefer et al. 2019).
Adaptation, expansion of the framework in other frameworks, guidelines and evaluation toolkits
From those 17 authors who suggest an adaptation or expansion of the CSEF, one publication stands out for explicitly challenging the framework from an Indigenous and decolonial perspective, expressing a critical reinterpretation and normative critique. Calyx and Finlay (2022) critique the language of the CSEF for reproducing colonial assumptions. From an Australian Indigenous standpoint, they argue that evaluation frameworks for participatory science must explicitly incorporate principles of Indigenous Data Sovereignty and reframe participation as partnership rather than extraction. Their critique focuses less on the CSEF’s structure than on its terminology and underlying epistemic assumptions, proposing alternative formulations that foreground recovery, trust-building, and long-term relationships. The authors suggest, for instance, to use the expression “partnerships or prospective partners” instead of “target groups.” While they do not present an alternative framework themselves, their contribution highlights limitations of the CSEF when applied across sociocultural contexts shaped by colonial histories, and signals the need for greater reflexivity regarding language, power, and data governance.
A second group of publications engages with the CSEF in a selective and complementary manner. Hjelm (2023) integrates the CSEF with two other models (Woods et al. 2020; Phillips et al. 2018) to derive eight analytical categories tailored to the assessment of learning outcomes and impacts in microplastics-focused citizen science. So, this newly generated framework focuses on learning outcomes and impacts only, and took dimensions from the three frameworks. It functions as a source of open-ended evaluative questions that are translated into interview protocols, illustrating its use as a flexible heuristic rather than a fixed indicator set. Gignac et al. (2021) developed an impact assessment tool as part of the European InSPIRES project that works with self-evaluation, mostly quantitative questionnaires for four user groups, who are invited to assess their citizen science project at different time points. The questionnaires contain several items of the original CSEF, but they are structured along five different dimensions: knowledge democracy, citizen-led research, participatory dynamics, transformative change, and integrity. Thus, the indicators of the CSEF are combined in new dimensions and amended with additional items.
The impact assessment report of the YouCount project, which brings together young people and relevant actors to co-design solutions to today’s societal challenges (Lorenz et al. 2023), explicitly adopts the three core dimensions, but departs from the original logic model to better fit youth-focused and social science–oriented contexts. They present a newly developed framework, where its strong conceptual resemblance illustrates how the CSEF has been appropriated and reshaped to meet project-specific needs. The authors added, for example, metrics for “gender equality and sensitivity” and “youth friendly academic careers” to the scientific dimension. Soria Aguirre et al. (2024) also kept the three dimensions of the CSEF but focused on the process indicators to develop a framework to evaluate citizen science projects in nature conservation. They offer a set of quantitative indicators that helps examine the overall project design and setup. To fit the strong data-driven nature of conservation projects, the scientific dimension has been amended with, for instance, several questions related to data quality, data management, and technology used.
Another finding across the reviewed literature is the development of other comprehensive frameworks that were inspired by the original CSEF. Amongst European-funded projects, MICS and ACTION are the most prominent examples. MICS, like InSPIRES, also introduces five dimensions, although different ones: science and technology, society, governance, ecology, and economy. MICS created a set of more than 200 closed questions for project owners’ self-assessment. These 200 questions amend the indicators of the CSEF with additional ones from other frameworks and provide a fixed set of potential answers to choose from for each question. Likewise, the ACTION project has chosen a five-dimensional approach in its impact assessment framework (Passani et al. 2020), including scientific, social, political, environmental, and ecological dimensions. In these five dimensions, they used a large majority of the impact indicators from the CSEF and amended them with additional indicators on social inclusion, research community empowerment, and self-governance, to name a few. For each indicator, the authors elaborated on how to measure these, making use of quantifiable project outputs and self-reported outcomes.
The reviewed literature also shows the emergence of guiding principles or propositions for impact assessment frameworks. Wehn et al. (2021) conducted literature research on impact assessment in citizen science and demonstrated the impact along the same five domains as for the MICS project. In the science domain, they cite indicators of the CSEF. In the other domains, they propose a collection of indicators identified in the researched literature. As a result of their research work, they propose five guiding principles of evaluation that stress aspects like the variety of purposes of citizen science impact assessment, the non-linear conceptualization of impact journeys, and the consideration of context and specific project goals. Gerke et al. (2023) defined 11 propositions that a generic framework for research impact assessment would need to consider. They stress aspects of co-production, the participatory work with the theory of change (Anderson 2004), and the integrative aspects of a generic framework. The authors highlight the importance of a process-based evaluation to better understand how impact is generated. Additional studies on transformative impacts and on citizen science hubs introduce quantitative indicator frameworks that mirror the original dimensions of the CSEF while adding a distinct domain for institutional or infrastructural support (Skaržauskienė et al. 2024).
Discussion
Overall, the synthesis shows that the CSEF by Kieslinger et al. (2018) has been influential primarily as a structuring reference rather than as a prescriptive instrument. Its three-dimensional logic has been widely echoed, its indicators selectively reused, and its conceptual assumptions both extended and contested.
Across nearly all reviewed publications, there is strong convergence around the need for process-based evaluation alongside outcome and impact assessment. Contributions from both citizen science–specific and broader research impact assessment literature emphasize that understanding how impacts are generated requires attention to participation processes, power relations, learning trajectories, and institutional arrangements. At the same time, several authors stress the importance of contextualizing impact assessment relative to project goals and environments, cautioning against absolute or decontextualized impact metrics (Lorenz et al. 2023; Wehn et al. 2021).
Additionally, the investigation shows that especially the measuring of impact on participants and societal actors is perceived as important by scholars (Jenkins et al. 2025; Vercammen et al. 2020). The operationalization of indicators from the participant dimension of the CSEF was widely applied and helps to identify where the involvement of citizens in the research process makes a difference in terms of changed knowledge, behavior, or ownership.
We find several extensive evaluation frameworks and approaches that have been developed based on the CSEF and provide detailed operational guidance. In some cases, the level of granularity these general frameworks offer may exceed the projects’ capacities and resources for evaluation. Thus, the literature reflects an unresolved tension between comprehensiveness and usability, particularly when frameworks are expected to function across heterogeneous project contexts. And still, we find the call for more structured evaluation of citizen science impacts in the literature (Palma et al. 2024).
What we have clearly seen is that evaluation frameworks’ need to be flexible, modular, and adaptable, which was the intention of the CSEF. A model needs to be open enough to be customizable to the features of each citizen science initiative; observation areas should be adapted or eliminated if not relevant, or others should be added (e.g., D’Andrea et al. 2022). Comprehensive models need to be adaptable to different sociocultural contexts, scientific disciplines, areas of reach, and conditions where they want to create change. We have seen that core considerations such as aspects of social inclusion, power relations indicators (Lorenz et al. 2023; Passani et al. 2020), the impact of citizen science on scientists and their professional and personal development (Hoedoafia et al. 2024), and more detailed reflections on data management and data quality (Soria Aguirre et al. 2024) were not included in the original framework. In addition, we have learned that the language used in these frameworks is not universal and terminology may include different epistemic assumptions.
Another topic that has not been explicitly addressed by any of the developed frameworks, although it is currently widely discussed within and of high importance for the citizen science community, is artificial intelligence. We think that critically assessing how emerging technologies such as artificial intelligence can be useful and harmful for participatory research is of high importance for the future.
Together, the findings from our analysis provide a grounded basis for proposing targeted adaptations that respond to actual patterns of use, critique, and extension observed across nearly a decade of engagement.
Across the analysis there was a clear emphasis on process and feasibility. While policy makers and funders like to see the impact generated to justify their return on public investment, project implementors are keen on having tools and methods at hand for their planning and implementation. Following on from these insights and keeping the required flexibility and possibility for contextualization, we suggest agile pathways to evaluation and impact assessment alongside a theory of change (Anderson 2004). The evaluation process (as shown in Figure 4) should start with a thorough reflection on what should be achieved by the citizen science activity, what changes it wants to bring, and for whom: Is the focus, for example, on the learning aspects, policy change, environmental conservation, or specific innovations for a certain group or community of involved actors? Ideally, all actors are already represented during this definition process. Thus, in this first step, a project planning team may select and specify outcome indicators from the three dimensions of the CSEF that are expected to lead to the desired impact. Once there is an understanding of what changes are aimed for, the preconditions leading to this impact need to be defined. They are often referred to as project outputs and can be defined in terms of key performance indicators (KPIs).

Figure 4
Theory of change thinking adapted to designing impact measurement.
The interventions in a citizen science project are typically the activities of the research process itself that leads to the defined outputs, outcomes, and expected impacts. Process indicators from the CSEF can support the setup of an adaptive and well-structured research process. Assumptions refer to underlying beliefs and previous experiences that explain the connections between the preconditions for long-term change and the expectations about how and why the proposed interventions are appropriate to bring this change. Alongside and based on these core elements of a theory of change, we can select the measures and indicators to provide evidence for the expected change. Applying this approach allows defining flexible impact pathways that connect outputs and outcomes across domains according to the different project contexts.
When applying such theory-of-change thinking to the planning of the evaluation and impact assessment, the resulting evaluation approach is ideally specifically tailored to the needs of the project and the actors involved. As we already suggested in the original CSEF, not every dimension will have the same priority in each project, and thus the involved actors and especially those mostly concerned and affected by the envisioned change should have a strong say in this process early on (Kieslinger et al. 2022). We are better off with modular evaluation frameworks that can be adapted to the specific needs of each project than with one rigid framework that claims to cover everything.
We off a concrete example to make such an approach more tangible: Let’s imagine that a team of researchers in health policy and communication, together with schoolteachers and students, design a citizen science project to explore the impact of social media health influencers on adolescents’ health and food consumption behavior. From the onset, they define their focus as gaining scientifically valid data about adolescents’ influencer communication and deriving insights on how this communication impacts young people’s behavior. Thus, they define how they will monitor social media and communication behavior and work with young people on the content interpretation. In terms of evaluation and impact assessment, the team decides to measure specifically the participant dimension as they aim to find individual behavior patterns and want to assess how the continuous monitoring of social media communication and their participation in content analysis may change the behavior of the adolescents participating in the study. In addition, the impact assessment may add a gender dimension and intersectional aspects to assess possible differences in impact across gender, and intersectional factors that may influence any expected changes. Helpful indicators for such a specific impact assessment may then be found in existing frameworks (e.g., in Passani et al. [2020] or Lorenz et al. [2023]).
Conclusions
The literature review around the CSEF by Kieslinger et al. (2018) identified three different contexts in which the evaluation framework was applied by other scholars: (1) general reference to aspects of evaluation in citizen science, the proposed impact domains and indicators, (2) operationalization of parts of the CSEF in process and impact evaluation of citizen science activities, and (3) work on complementary evaluation frameworks and guidelines. The wide variety of operationalization and partial adaptations of the CSEF stresses the importance of its modular character as well as its continuous need for adaptation and critical reflection. Thus, the strengths of the CSEF relate to its comprehensiveness and multi-dimensionality; limitations are the requested context-sensitivity, resource intensiveness of implementation, and limited indicators related to, for example, data quality, social inclusion, or capturing power and equity dimensions. Subsequent work has expanded on specific parts of the framework, for example, by examining economic and policy outcomes of citizen science projects in more depth and with more formal evaluation designs. Our analysis, suggests defining flexible impact pathways based on the theory of change that allow citizen science initiatives to context-specifically navigate through comprehensive evaluation frameworks.
There are clear limitations in this study. Most importantly, we looked only at the publications found in Google Scholar. While this open database included important references, we are aware that there are many project reports and other grey literature that we did not consider, especially those that applied the original framework.
Acknowledgements
We would like to thank especially the co-authors of Kieslinger et al. (2017) and Kieslinger et al. (2018), Heigl Florian, Dörler Daniel, Richter Anett, and Bonn Aletta, for their contributions to the original CSEF as well as all members of the ECSA working group on impact assessment for the continuous exchange and critical reflections on impact assessment in citizen science.
Author Contributions
Both authors contributed equally to all aspects of the manuscript.