Introduction
Citizen Science (CS) is an emerging area of research and practice where volunteers collaborate with professional scientists by taking part in the research process while enhancing their scientific literacy and generating new knowledge (Turrini et al. 2018; Bela et al. 2016; Bonney 2021). CS projects are becoming an important component of biodiversity and nature conservation research in multiple ways (Peter et al. 2021; Suškevičs et al. 2021). International nature conservation–related citizen science (NCCS) projects such as iNaturalist (Mesaglio and Callaghan 2021) and eBird (Sullivan et al. 2017) are extending the evidence-based needed for policy-making by enriching global biodiversity databases (Chandler et al. 2017; Bonney 2021). One such example is the Global Biodiversity Information Facility (GBIF), which compiles all observations about species occurrences worldwide; at least 50% of their inputs come from NCCS initiatives (Chandler et al. 2017; Miller 2022). The long temporal scope, broad taxonomic coverage, and wide geographic range of the NCCS projects’ datasets are assisting in filling the large-scale data gaps that scientists have identified in the literature (Danielsen et al. 2014; Pocock et al. 2015). Volunteers who engage in NCCS projects generally participate in tracking, monitoring, or documenting biological data (Theobald et al. 2015). The new knowledge generated commonly relates to long-term datasets on phenology, species occurrence, distribution, or abundance. The new data also helps to inform policy and decision-makers as well as to achieve biodiversity conservation goals (Theobald et al. 2015; Chandler et al. 2017; McKinley et al. 2017). Participation in NCCS projects has shown corollary effects on the development of volunteers, which are mainly linked to an increase in scientific and conservation knowledge, raising nature conservation awareness, promoting environmental education, and influencing attitude and behavioral change (Jordan et al. 2016; Bela et al. 2016; Bonney 2021; Santori et al. 2021).
In Europe, CS has predominantly contributed to the life sciences (Hecker, Garbe, and Bonn 2018). However, while NCCSs are well known in Western Europe, they are only now gaining recognition in Central and Eastern Europe (Vohland et al. 2021). In 2018, the European Commission released an inventory of 503 NCCS projects from the European Union that provided policy support, contributed to science, and encouraged meaningful citizen engagement (Bio Innovation Service 2018). Central and Eastern European countries were underrepresented in this document. CS research in Hungary is still scarce (Vohland et al. 2021) despite the fact that several NCCS projects are in operation. The Common Bird Monitoring Program of the Hungarian Ornithological and Nature Conservation Society, also known as BirdLife Hungary (MME), has been active since the late 1990s (Szép and Gibbons 2000), and the WildWatcher (Vadonleső) has been operating for more than a decade (Vadonleső Group et al. 2018, 2019). Since the 2020s, CS initiatives have been supported by the National Research, Development and Innovation Office’s policies, growing expert networks, and funding schemes (Czeglédi 2022).
Evaluation of CS projects is highlighted in one of the principles (principle 9) of The European Citizen Science Association (ECSA) as a core requirement (Robinson et al. 2018). Previous studies have adopted frameworks to evaluate CS projects along specific dimensions including their scientific impacts (D’Andrea et al. 2022), learning-education outcomes (Phillips et al. 2018), the role of CS on conservation (Blaney et al. 2016; Yang et al. 2019), and the quality of participation (Shirk et al. 2012). Several dimensions (social, scientific, and socio-ecological/-economic) are integrated into the open framework of Kieslinger et al. (2018), which can be used for a variety of CS projects from different objectives and fields.
Recommendations for CS evaluation in the literature encourage CS frameworks to be developed for specific fields (Davis et al. 2022).
Most of the evaluation frameworks that specifically focus on NCCS projects usually cover only one dimension such as data quality (Kosmala et al. 2016), learning outcomes (Phillips et al. 2018), ecological literacy (Jordan et al. 2016), pro-environmental behavior (Pocock et al. 2023), or the role of CS in conservation (Yang et al. 2019). Only a few frameworks focus on NCSS projects from a multidimensional perspective. Turbé et al. (2019) assessed European NCCS projects that already have policy relevance by considering citizen scientist, science, socio-economic, and policy dimensions. Price-Jones et al. (2022) evaluated 103 NCCS projects specifically targeting invasive species in Europe. They assessed the projects considering different variables such as data quality, data management, and participants’ engagement.
However, frameworks that would enable the comparison of NCCS projects in more dimensions are still needed (Bonney 2021; Somerwill and Wehn 2022).
Therefore, based on the CS literature, we propose an evaluation framework for NCCS projects in three dimensions: science, nature conservation, and participants’ development. We then apply our framework to assess the performance of eight Hungarian NCCS projects.
Research and Design Methods
Conceptual framework
The evaluation framework shown in Figure 1 was developed to assess NCCS projects, incorporating relevant CS literature that covers different evaluation frameworks and recommendations for the advancement of CS projects (see references in Table 1). We address three dimensions that seem very relevant to NCCS projects: (science, nature conservation, and participants’ development). The science dimension was chosen because there is a discussion in the CS literature about the scientific value of CS outputs (Balázs et al. 2021). While our evaluation framework targets NCCS projects, the primary goal of a nature conservation–related CS project is to contribute to the conservation of natural resources, ecosystems, or species (McKinley et al. 2017; Suškevičs et al. 2021). That is why the nature conservation dimension was added. CS projects are based on the work of volunteers, and their participation might have added value to their personal development, which is starting to be noticed in some studies (Turrini et al. 2018; Phillips et al. 2018; Santori et al. 2021). This is the reason for the choice of the participants’ development dimension. Table 1, which shows the criteria used for assessing NCCS projects, is based on the evaluation framework shown in Figure 1.
Table 1
The criteria for the assessment of nature conservation–related citizen science (NCCS) projects in three dimensions.
| COMPONENT | CRITERIA FOR THE ASSESSMENT | REFERENCES | QUESTION | NO = 0, YES = 1 |
|---|---|---|---|---|
| Science dimension | ||||
| Methodology | Research question | McKinley et al. 2017. | Was a scientific question formulated before starting the project? | |
| Methodology for data collection | Kosmala et al. 2016; Stevenson, Merrill, and Burn 2021. | When recording an observation, is supporting information (e.g., a photograph, videos) mandatory as part of data collection methodology? | ||
| Technology use | Serrano-Sanz et al. 2014; Kosmala et al. 2016; Freitag, Meyer, and Whiteman 2016 | Does the project require the use of technology that supports data collection and data quality? (e.g., smartphones, tablets, sensors) | ||
| Data quality | Data validation | Wiggins et al. 2011; Freitag, Meyer, and Whiteman 2016; Balázs et al. 2021 | Are protocols for data collection tested and validated before starting the project, in order to ensure data quality? | |
| Wiggins et al. 2011; 2013; Kosmala et al. 2016; Freitag, Meyer, and Whiteman 2016; McKinley et al. 2017 | Do they have an effective strategy for data quality assessment? (e.g., specialized professionals involved, request of more information to participants, site visits) | |||
| Institutional affiliation | Freitag, Meyer, and Whiteman 2016; Turbé et al. 2019 | Is the project formally connected to a larger scientific or conservation organization? | ||
| Data management practices | Long term datasets | McKinley et al. 2017 | Does the project have long-term datasets or plan to run for the long term? | |
| Open data | Murray-Rust 2008; McKinley et al. 2017 | Is there any summary of data collected released for the public? (e.g., summaries in form of maps, charts) | ||
| Wiggins et al. 2013; Freitag, Meyer, and Whiteman 2016 | Are data stored in suitable repositories? (e.g., the General Data Protection Regulation [GDPR] considered) | |||
| Murray-Rust 2008 | Are complete or partial datasets publicly available for download? | |||
| Publications | Freitag, Meyer, and Whiteman 2016; Bio Innovation Service 2018; McKinley et al. 2017; Turbé et al. 2019 | Do they have publications in peer-reviewed academic journals or are they planning to? (e.g., collected data used in publications and citations in peer-reviewed academic journals or scientific books) | ||
| Nature conservation dimension | ||||
| Environmental monitoring | Data use in species/Ecosystem monitoring | McKinley et al. 2017 | Have data collected been used for species/ecosystem monitoring? | |
| Environmental management | Data use in species/Ecosystem management | McKinley et al. 2017; Maynard et al. 2020 | Have data collected already been used for species/ecosystem management? | |
| Conservation policy-making | Data use in conservation policy making | McKinley et al. 2017; Turbé et al. 2019; Suškevičs et al. 2021 | Have data impacted local, national, regional or European Union policy making? | |
| Participants’ development dimension | ||||
| Training | Forms of training provided | Turbé et al. 2019; Phillips et al. 2018 | Is personal training provided to the participants? | |
| Turbé et al. 2019 | Is any other form of training/knowledge transfer provided to the participants? (e.g., written guidelines, videos, informative phone application or website) | |||
| Additional learning support | Educational events | Serrano-Sanz et al. 2014 | Are events to support learning and promote environmental education organized? | |
| Communication for gaining knowledge | Brossard, Lewenstein, and Bonney 2005 | Are there ways to contact experts who are available for participants to answer questions and solve problems? (e.g., via email, social media, app chat) | ||
| Measurement strategies for participants’ personal development | Measurement of knowledge gained/reinforced | Maynard et al. 2020; Santori et al. 2021; Brossard, Lewenstein, and Bonney 2005; Jordan et al. 2011; Phillips et al. 2018 | Is participants’ knowledge (gained/reinforced) measured? (e.g., species recognition, knowledge about the species ecology, conservation knowledge) | |
| Measurement of skills – gained/reinforced | Maynard et al. 2020; Santori et al. 2021; Phillips et al. 2018 | Are the participants’ skills gained/reinforced measured? (e.g., using equipment, data collection protocol) | ||
| Measurement of attitude/behavioral change | Somerwill and Wehn 2022; Maynard et al. 2020; Santori et al. 2021; Brossard, Lewenstein, and Bonney 2005; Jordan et al. 2011; Phillips et al. 2018 | Are participants’ attitude/behavioral change measured? | ||
| Strategies for reinforcing and increasing knowledge through feedback | Cox et al. 2015; Ceccaroni and Piera 2017; Tang et al. 2021. | Is any feedback that reinforces and increases knowledge provided to participants? (e.g., feedback in person, through email, or app) | ||
Figure 1
The proposed evaluation framework (dimensions and components).
We compiled a comprehensive list of criteria that were suggested in the literature for each dimension. No criteria were disregarded, yet several were merged. The questions for each criterion of assessment were designed by the authors, ensuring that they are clear, concise, and effective in gathering information that can be answered through yes/no responses.
Selection of nature conservation–related citizen science projects for the assessment
Eight Hungarian NCCS projects were chosen for analysis in a three-step process from November 2021 to January 2022: 1) We conducted a desk study to explore past and ongoing NCCS projects, using information from official web pages of CS associations and European Union (EU) CS platform repositories (e.g., the EU-funded online platform eu-citizen.science co-developed and run by ECSA). Also, we explored the websites of universities and scientific institutes from Hungary; some of these sites guided us to the NCCS project websites. We built an initial list of fifteen potential projects. 2) To include NCCS projects in our final list, we applied the following selection criteria: a) NCCS projects started in Hungary, b) the aim of the project was linked to nature conservation or biodiversity, and c) they had flora or fauna target species. 3) We sent out online invitations to fifteen projects. We received positive responses from nine projects who accepted the invitation to the interviews. We excluded the project “Looking for Cowslips” because it originated in Estonia. We interviewed coordinators from the selected eight NCCS projects, who all became co-authors of this paper.
Data gathering
Interviews
From April to July 2022, six online and two in-person semi-structured interviews (Newing et al. 2011) were conducted with NCCS project coordinators. We aimed to get an in-depth understanding of experts’ experiences running NCCS projects in Hungary. The questions were designed based on our conceptual framework. The interviews were structured in five blocks, 1) project description, 2) science dimension, 3) nature conservation dimension, 4) participants’ development, and 5) outcomes and challenges (see the interview questions in Supplemental File 1). The in-person interviews were recorded using a digital voice recorder (Olympus WS-832), while online interviews were made via the Skype platform with recording. During the interviews, notes were taken, and by utilizing MAXQDA software (VERBI 2021), a literal transcription of every interview was compiled. The interviews lasted an average of 80 minutes; the shortest interview was 60 minutes, and the longest was 120 minutes, which was done in a two-day session because of the time availability of the interviewee.
Data analysis
Following our conceptual framework, the transcripts of the interviews were analyzed with qualitative content analysis, and based on that, the eight Hungarian NCCS projects were evaluated. First, a priori codes were created for the main characteristics of the projects (aim, institution affiliation, target species, periodicity, geographical scope, the task of the participants, number of participants, and number of observations until April 2022. Codes were developed based on the three dimensions and their components (Figure 1), then they were applied to the transcripts of the interviews using the MAXQDA software (VERBI 2021).
A summary table was then created by going through the coded text and based on the main characteristics of the projects (see Supplemental File 2: Table S1). By answering the questions in Table 1 related to the performance criteria, scores were assigned (see Supplemental File 3). After this process, we summed up the scores for each criterion within and across the studied NCCS projects, and we compared the eight projects using the percentage of maximum scores. We did not carry out a more exhaustive quantitative analysis. The scoring was reviewed and accepted by interviewees who are all co-authors of this study. To explain our findings, we employed qualitative data from the content analysis of the interviews. We base our analysis on the interviews showing the state of the NCCS projects in 2022. Yet, there have been changes and further developments in some projects since the interviews were completed, which are reflected in the Discussion.
Results
Overall performance of NCCS projects
The total achieved points of the eight projects are presented in Supplemental File 3: Table S2. The maximum score that could be reached was 22 points. Based on this overview, the Common Bird Monitoring Program scored the best by reaching a total of 17 points, whereas the Arthropods NCCS project had the lowest total performance score of 11 points (Figure 2). The next section contains further information about performance according to each dimension.
Figure 2
Dimension performance of each nature conservation–related citizen science (NCCS) project in 2022. * BeaverMap and MyPond started in 2021.
Performance of the nature conservation–related citizen science projects in each dimension
Figure 2 shows the percentages of scores of NCCS projects achieved per dimension. In the science dimension, all the projects reached 60% or more of the possible maximum scores. In this first dimension, the Common Bird Monitoring Program achieved 90% of the possible maximum scores followed by the Mosquito Monitor and MyPond (82% each). The Common Bird Monitoring Program had the best performance in the nature conservation dimension, along with the Amphibian and Reptile Mapping and WildWatcher, scoring 100% each. The two projects, Arthropods and Butterfly-Net, followed, each demonstrating a 67% performance in the nature conservation dimension. BeaverMap and MyPond received lower scores in nature conservation because they were fairly new projects (began in 2021) at the time of the interview (began in 2022); the same was true for Mosquito Monitor (began in 2019). However, their coordinators intend to extend the use of the project data for nature conservation in the future. The Butterfly-Net effectively represented the participants’ development dimension, achieving 88% of the possible maximum score, while the rest of the seven projects reached less than 50%.
Science dimension and its components
The methodology component includes the research question, methodology for data collection, and technology use. The Common Bird Monitoring Program, Mosquito Monitor, BeaverMap, and MyPond had a scientific question developed prior to project commencement. The methodology for data collection varies among projects. The Common Bird Monitoring Program uses a randomized sampling design, a methodology adopted from the well-recognized Breeding Bird Survey from the UK (Szép and Gibbons 2000). When recording an observation, not all the projects request the mandatory upload of supporting information (e.g., WildWatcher does not request uploaded pictures, whereas in the Amphibian and Reptile Mapping project, uploading pictures is optional, but supporting GPS coordinates and filling out a short questionnaire are mandatory). Every project uses technology to support data collection, including smartphones with specialized applications, GIS map visualization, and websites—with the exception of Butterfly-Net. The data quality component includes data validation and institutional affiliation. The Butterfly-Net and the Common Bird Monitoring Program validated the protocol process for data collection by testing it before the official start of the project. Experts are in charge of data validation in every project, but how they are involved differs. While experts in charge of data validation are employed by the majority of the projects, their participation in the Arthropods NCCS project is entirely voluntary. In the Arthropods project, after applying for the volunteer expert position, they are granted special access to the project platform to review the records. The Arthropods project manager highlighted the lack of taxonomists in the country willing to sign up for the task. Data validation forms that involved requesting more information from participants and site visits were rare. The majority of the NCCS initiatives were affiliated with nongovernmental (NGOs), conservation, or scientific organizations in 2022, except for the Arthropods project, which was launched through private efforts and its coordinator had no intention to pursue an affiliation at that time.
The last component of the science dimension is data management practices, which includes long-term datasets, open data, and publications. All the projects maintain methods of suitable data management in terms of storage and releasing data. Only Arthropods offers open access raw data, but all have a willingness to share datasets on request. Having a platform (e.g., an online website, an app) resulted in the primary strategy to release data through summaries of validated data (e.g., distribution of maps and graphs). For example, on the WildWatcher and Mosquito Monitor websites, observation points can be updated, visualized, and filtered per year or per species using maps. Arthropods as well as Amphibian and Reptile Mapping offer the same, with the extra opportunity to view the recorded points with the added photo provided by the participants. BeaverMap shows the observations of the volunteers on a map that is periodically updated. Project coordinators emphasized that all the data are managed carefully, especially the geolocation of endangered or strictly protected species (Amphibian and Reptile Mapping, BeaverMap, WildWatcher, and Arthropods), as well as personal information of the participants such as email addresses (MyPond, Amphibian and Reptile Mapping, BeaverMap). In the case of BeaverMap, coordinators emphasized that they will not share information about beaver dams until the protection of valuable beaver-made wetlands is guaranteed by regulations.
Writing scientific publications was a common strategy to share data by older projects that have released scientific papers under their project affiliations (e.g., the Common Bird Monitoring Program (Szép and Gibbons 2000; Szép et al. 2012; Szép et al. 2021), the WildWatcher (Vadonleső Group et al. 2018, 2019; Vadonleső Group et al. 2019)). All the other projects were planning to prepare scientific publications, except for the Butterfly-Net due to their short-term running period. Another data management practice was the use of data by other researchers not involved in the programs. Arthropods’ data have been used most widely in scientific papers of other scholars (e.g., Károlyi and Rédei 2017; Merkl, Károlyi, and Korányi 2017; Vétek et al. 2018; Kóbor et al. 2021).
Nature conservation dimension and its components
The environmental monitoring component includes data used in species/ecosystem monitoring. All analyzed projects generated data that was helpful for environmental monitoring, with all of them targeting species, and MyPond specifically focusing on ecosystems. The Arthropods project’s open data led to the identification of new species occurrences in the country e.g., A. heegeri (Károlyi and Rédei 2017), Cybocephalus nipponicus (Merkl, Károlyi, and Korányi 2017) and invasive species such as Acanalonia conica (Kóbor et al. 2021) or Halyomorpha halys (Vétek et al. 2018).
The environmental management component includes data used in species/ecosystem management. In the Mosquito Monitor project, participants catch specimens of invasive mosquitoes, helping in the eradication of these species that are not only causing conservation-related problems but also spread viruses and pathogens that pose a threat to human health. In WildWatcher, the Hedgehog (Erinaceus roumanicus) is the most frequently observed species. Reports from participants about this species’ occurrence outside their distribution range (where food is limited) helped to initiate action for habitat management. Data from Amphibian and Reptile Mapping served to protect the habitat of the fire salamander (Salamandra salamandra) in a district of Budapest. During the operation of Butterfly-Net, observations by students helped to discover a new population of butterflies, which was followed by action for habitat protection by the Őrség National Park Directorate.
The conservation policy component contains the data used in conservation policy; thus older projects perform well in this component. For example, WildWatcher data have been included in the Hungarian Nature Conservation Information System (TIR), and the national report of Article 17 of the EU Habitat Directive. Butterfly-Net data was included in the biodiversity platform called OpenBioMaps (Bán et al. 2022), which is available for decision-makers to support biodiversity and management. Amphibian and Reptile Mapping supplies the European Atlas of Amphibians and Reptiles with data atlases (SEH) (NA2RE) (Sillero et al. 2014), and the project provides regular reporting according to Article 17 of the EU Habitat Directive. The Common Bird Monitoring Program, with the largest ornithological dataset in Hungary, has been providing data for common EU bird monitoring and the annual bird atlas for Hungary for several years (Szép et al. 2021). All these contributions increase the possibility of impacting local, national, regional, or EU policy-making.
Participants’ development and its components
The training component includes the forms of training provided. The Butterfly-Net was a unique initiative that combined personal training (lectures on butterfly biology, and species identification) with other forms of knowledge transfer (e.g., species identification books, phone applications, environmental education events). The other projects utilize non-personal training strategies through their websites, where participants can find descriptive information about instructions for field observation, data collection protocols, and species. The Common Bird Monitoring Program offers extra materials such as a bird-sound guide, maps, and field diaries. MyPond trained people through videos and sampling kits (informative guides with conservation and scientific content, sampling tools for water chemistry and eDNA, data sheets, and user-friendly manuals). Mosquito Monitor also uses videos to show how to trap mosquito specimens.
The additional learning-support component focuses on educational events and communication for knowledge gain. Some projects organize events (e.g., The Mammal of the Year [WildWatcher] or The Amphibian and Reptile of the Year [Amphibian and Reptile Mapping]) that serve to promote environmental education and nature conservation awareness. Social media is used to promote these events. The Common Bird Monitoring Program organizes recognition events where observers gather to receive awards. Participants have established a community of common interests in which they can learn from each other’s experiences. In many projects, experts are freely available for the participants if they have questions or concerns. Volunteers, mainly farmers, commonly contact experts of BeaverMap to find common solutions for beaver-related conflicts. Participants of Amphibian and Reptile Mapping and WildWatcher use social media (Facebook) to communicate with experts and with each other. By exchanging photos and seeking advice on species identification, participants engage in discussions that help to reinforce their knowledge about their observations and ensure species recognition before data recording on the website.
The last component is the measurement strategies for participants’ personal development that involve the measurement of knowledge and skills gained/reinforced, measurement of attitude/behavioral change, and strategies for reinforcing and increasing knowledge through feedback.
Although all eight projects have generated new knowledge that improves the understanding of species in the territory of Hungary, none of them took action to measure the knowledge/skills gained by participants after their contribution. None of the projects measured the attitudes or behavioral changes of participants either. In the specific case of MyPond, coordinators emphasized that even though no behavior/attitude–change survey was used, participants’ willingness to move on to the second phase of MyPond (kitbox distributed for measurements) already demonstrates a good attitude towards involvement. Six projects (the Common Bird Monitoring Program, Amphibian and Reptile Mapping, Butterfly-Net, Arthropods, Mosquito Monitor, and MyPond) provide some form of feedback that helps in reinforcing or increasing the knowledge of the participants (e.g., feedback via email, sending email via post or app). Although project coordinators mentioned that giving feedback encourages participants to continue contributing, no formal measurements have been made to know the impact of it on NCCS projects or volunteers.
Discussion
The evaluation framework
Literature on CS calls for new evaluation frameworks that are beyond the traditional impacts of CS, focus on specific fields, have standardized approaches, and are easy to use (Bonney et al. 2014; Schaefer et al. 2021). We designed our framework for NCCS projects, which includes three dimensions with components and a set of criteria. A scoring system has been developed based on yes and no questions related to each criterion. Other multidimensional frameworks are either not specifically designed for NCCS projects and therefore the nature conservation dimension is missing (e.g., Kieslinger et al. 2018), or in the case of NCCS evaluation frameworks, they focus on partly different dimensions and components (e.g., Turbé et al. 2019; Price-Jones et al. 2022). Kieslinger et al. (2018) proposed a set of supporting questions for the evaluation, but the questions were partly open-ended and did not lead to a scoring system. Turbé et al. (2019) conducted a survey among NCCS projects with policy relevance with closed and open-ended questions. Their evaluation was based on statistical analysis; they did not aim to develop a standardized scoring system either.
Science dimension
It has been demonstrated that NCCS efforts can serve as an essential data source for research when data quality standards are maintained and ensured (Wiggins et al. 2013). For example, eBird, an international CS project with a rigorous data collection and validation protocol, has published at least 90 peer-reviewed scientific studies (Bonney et al. 2014; Kelling et al. 2015). In our study, the Common Bird Monitoring Program in Hungary, which scored the highest in the science dimension, adopted the data collection protocol of the Breeding Bird Survey from the UK. It has consistently maintained high scientific standards for methodology, data quality, and validation, creating one of Hungary’s longest-running (23 years), highest-quality ornithological datasets.
A handbook for CS projects released by the United States Environmental Protection Agency (EPA 2019) emphasizes that data produced by projects in collaboration with professional organizations increases the chance to meet data quality standards and extend scientific outputs. In our study, we found that all projects, with the exception of Arthropods, were linked to a scientific or a conservation institution at the time of the interviews. Now Arthropods is also connected to an NGO (Közösen a Természetért Alapítvány – Together for Nature Foundation), showing that affiliation is becoming important for stable operation.
Turbé et al. (2019) found that projects with open access data typically generate more scientific outputs than those without open data. In our study, only Arthropods had an open access and downloadable data policy, which has led to a long-term dataset of more than 400,000 records. Although all other analyzed projects are prioritizing the scientific dimension, and some newer projects have started to publish their results (e.g., Mosquito Monitor [Garamszegi, Kurucz, and Soltész 2023] and MyPond [Márton et al. 2023; Hamer et al. 2024]), we recognize that there is still a lot of opportunity for developments in terms of data accessibility and data-sharing standards.
Von Gönner et al. (2023) also assessed CS projects in Germany, Austria, and Switzerland from a multidimensional perspective including the science dimension. In line with our findings, they discovered that although data quality assurance is widely applied in most of the projects, not all have open raw data, and scientific publications still need to be released in half of the evaluated projects.
Nature conservation dimension
Citizen research projects all over the world assist natural resource management decisions in international biodiversity monitoring and conservation policy (Chandler et al. 2017; McKinley et al. 2017; Turbé et al. 2019) at all levels—individual, community, national, and international. However, in their study, Runnel et al. (2016) stress that in Central and Eastern European countries, integrating CS in biodiversity monitoring and management is still needed. Based on our findings, all of the analyzed NCCS projects already support monitoring of species or ecosystems (MyPond). Additionally, observations in some evaluated projects are also used for habitat protection (Amphibian and Reptile Mapping, BeaverMap), detecting potential human-wildlife conflicts, (BeaverMap), and invasive species control (Mosquito Monitor).
Some scholars emphasize that data from large-scale and long-term CS projects are commonly used for population management decisions and even for assisting international environmental and conservation policies (Chandler et al. 2017; McKinley et al. 2017). Our findings also show that NCCS projects with long experience (Amphibian and Reptile Mapping, WildWatcher, and Common Bird Monitoring Program) had the highest score in the nature conservation dimension, including contributions to environmental management and policy. At the time of the interviews, most NCCS projects did not use their data in policy-making.
Similar to our findings, Sullivan et al. (2017) found in their evaluation that habitat management and protection, rather than law and policy, were the primary uses of eBird data.
Since the interviews, some additional projects used the NCCS project for conservation goals, indicating rapid development of the field in Hungary. For example, the BeaverMap that was initiated in 2021 already shows an impact on species/ecosystem management. They contributed to the first Hungarian model project that maintained a beaver-made wetland and solved related human-wildlife conflicts. Additionally, CS data allowed for the exploration of high-conservation-value beaver-made wetlands, and in the future, these data will improve the distribution map of the Hungarian beaver population (Juhász and Biró 2024).
Participants’ development dimension
According to Kieslinger et al. (2018), it is an indicator of a successful CS project when the benefits of the project equally serve the goals of the project and the participants. In our study, Butterfly-Net demonstrated the best performance in this dimension even though it was the project with the shortest duration. This project offered in-person training and constant connection with experts. Some authors, including Evans et al. (2005) and Phillips et al. (2018), have highlighted these characteristics as essential for influencing participants’ learning outcomes. Starr et al. (2014) state that personal training is not always cost-efficient. In the case of Butterfly-Net, personal training was possible because of the small number of participants (24). The other projects had much greater numbers of participants (e.g., Amphibian and Reptile Mapping: 23,000, Common Bird Monitoring Program: 10,712), and all continue to increase. Bonney et al. (2009) found that participants acquire confidence in their data collection skills as they work with supporting materials. We found that all projects offered some form of additional learning support through their website (e.g., online guides for supporting data collection, printed material, an app for species recognition).
Somerwill and Wehn (2022) showed in their review that even though a wide range of different validated approaches exist to measure changes in attitude toward the environment, in behavior, and in participant knowledge, only a few projects apply them. We have similar findings because, in the projects evaluated in our study, no assessment has been undertaken on project impact on the knowledge, acquired skills, attitude, or behavior of participants.
Leonard et al. (2023) showed in their study that CS projects in 25 countries evaluated the knowledge and behavioral change of participants. The findings indicated that volunteers exhibited positive attitudes and achieved considerable knowledge acquisition. A key recommendation for the evaluation was to consider demographics. Moving forward, the established NCCS projects in Hungary should enhance their data collection protocols to gather more information about participants, allowing for a deeper exploration of the participants’ development dimension.
Conclusion
In this paper, we developed a conceptual framework to assess the performance of NCCS projects in three dimensions: science, nature conservation, and participants’ development, and applied it to evaluate eight Hungarian NCCS projects. Overall, the investigated initiatives earned high scores in the science dimension, which strengthened the quality of data generated. Hungarian CS initiatives are becoming an effective tool for supporting the monitoring of biodiversity and developing conservation strategies. Since all of them have the vision to expand their impact in the nature conservation dimension, we recommend improving the strategies of data sharing with multiple stakeholders, fulfilling open data requirements, and promoting cooperation between NCCS projects and institutions with experience in informing and implementing conservation policies. All NCCS initiatives could further progress the participants’ development dimension, in terms of evaluation of knowledge gain and changes in attitude and behavior. Our results assist Hungarian NCCS managers and coordinators to understand the strengths of their projects and identify project dimensions that can be improved.
The newly developed evaluation framework can be applied to assess the performance of NCCS initiatives in three dimensions (science, nature conservation, and participants’ development). The main strength of the framework is its simplicity: yes-no questions need to be answered for each indicator. In most cases, with the help of the project coordinators, the questions can be easily answered. The framework’s simplicity and free availability enable its application by diverse end users, including project managers (for self-assessment and planning purposes) and external evaluators who are interested in the evaluation of NCCS projects. Although the evaluation framework was designed based on the Hungarian context, it can be applied to evaluate other NCCS projects as well.
However, the evaluation needs to be repeated from time to time because even since the interviews were conducted (2022), the projects have been further developed. The framework can be improved (e.g., combining composite indices for each dimension, changing the binary scoring system to a 0–3, 0–5 scale, focusing on only achieved results without plans, and altering some criteria, for example, institutional affiliation and extending the knowledge-attitude-behavior part). The framework can be further developed in a participatory way involving coordinators of NCCS projects and experts from other fields (e.g., conservation psychology).
Supplementary Files
The supplementary files for this article can be found as follows:
Supplemental File 2
Table S1. The main characteristics of the nature conservation–related citizen science projects in Hungary. DOI: https://doi.org/10.5334/cstp.762.s2
Supplemental File 3
Table S2. Results of the performance assessment of nature conservation–related citizen science projects (state in 2022). DOI: https://doi.org/10.5334/cstp.762.s3
Ethics and Consent
During the interviews we followed the fundamental ethical principles of the Code of Research Ethics of the Hungarian Academy of Sciences (HAS 2010) and the European Code of Conduct of Scientific Integrity (ALLEA 2017). A signed informed consent including information about data use, confidentiality, anonymity, voluntary participation, permission for recording, and causing no harm to participants was sought from the participants before the interview.
Acknowledgements
We acknowledge the time and help of the participants of all the NCSS projects.
We thank the funding of the eight CS projects involved in this study.
We are thankful to Bálint Balázs for the suggestions during the conceptualization of the study and Jocelyn Weyala Burudi and Yvonne Garcia for the English proofreading.
Funding Information
Johanna Soria was financed by Tempus Public Foundation, Hungary. Stipendium Hungaricum Scholarship (ID number: 2021_422428).
Erika Juhász received support through the National Laboratory for Health Security (RRF-2.3.1-21-2022-00006), HUN-REN Centre for Ecological Research, Budapest, Hungary.
Competing Interests
The authors have no competing interests to declare.
Author contributions
Conceptualization and methodology: SJ, TKE, interviews, analysis: SJ, data provision and validation: VO, BM, JE, SZ, BB, MZ, ST, HB, SI, KB, original draft -writing: SJ, review and editing: TKE, VO, BM, JE, SZ, BB, MZ, ST, HB, SI, KB, CA, BG, supervision: TKE.