Evaluating Citizen Science: Moving beyond Output Measures to Learner Behaviors, Interests, and Motivations

Evaluation of participant outcomes

Despite recognizing the importance of evaluation, the field of citizen science struggles to measure learning and participant impacts, often lacking comprehensive evaluations that align with available theories and tools (Brossard et al. 2005). This inconsistency hampers understanding program effectiveness and reach. Few articles have categorized participant outcomes quantifiably, contributing to challenges in cross-programmatic research (Friedman et al. 2008; National Research Council 2009; Bonney et al. 2009a,b; Jordan et al. 2011; Phillips et al. 2012; Bonney et al. 2016; Bonney et al. 2014). “A Framework for Articulating and Measuring Individual Learning Outcomes from Participation in Citizen Science” (Phillips et al., 2018) addresses this by re-conceptualizing and synthesizing learning outcomes into quantifiable categories, providing standardized vocabulary and evaluation instruments to facilitate research in citizen science (Roche et al. 2020). The framework encompasses content, process, and nature of science knowledge; interest in science and the environment; skills; self-efficacy for science and the environment; behavior and stewardship; and motivation. Its development aims to offer a standardized approach for measuring citizen science learning outcomes.

Common assessment items are crucial for consistent evaluation in citizen science. The Cornell Lab of Ornithology’s Developing, Validating, and Implementing Situated Evaluation Instruments Project (DEVISE) offers practitioners a standardized set of evaluation tools aligned with the Phillips et. al. 2018 framework (https://www.birds.cornell.edu/citizenscience/measuring-outcomes/). These instruments aim to establish uniformity in evaluation methodology across diverse projects (Phillips et al. 2015; Phillips et al. 2017; Porticella et al. 2017a,b). Blue Thumb conducted a needs assessment survey, with staff rating major activities by participant engagement and intended learning outcomes (Olson and Colston 2020). Brossard et al. (2005) recommend focusing on a program’s main desired outcomes rather than attempting to evaluate all listed outcomes. The study identified three major learning outcomes for Blue Thumb’s evaluation: behaviors and stewardship, motivation, and interest. The subsequent section defines each outcome and presents related evaluation tools (Table 1), followed by a literature review of methodologies used in previous evaluations of these outcomes.

Survey design

This study utilized a quantitative survey of participants with open-ended qualitative questions at the end (Appendix A). The survey items were intended to assess the identified participant outcomes determined during the previous focus groups with Blue Thumb Staff (i.e., behavior and stewardship, interest, and motivation). These surveys were modified versions of participant outcomes surveys created by Cornell Lab of Ornithology in their DEVISE project. We chose these because they have been previously tested for validity and used in other evaluation publications (Bonney et al. 2016; Phillips et al. 2015; Phillips et al. 2017; Porticella et al. 2017a,b).

Scales used in this research include: the General Environmental Stewardship Scale (GESS), the Interest in Science and Nature Scale (ISNS), the Motivation for Participation in Citizen Science Scale (MPCS), and the Motivation for Doing and Learning Science Scale (MDLSS).

• The GESS includes 6 different survey items and produces a range of totaled responses from 0 to 42, where higher scores indicate higher levels of pro-environmental behaviors. The GESS had a Cronbach’s alpha score of 0.73 in this study.

• The ISNS scores participants from 1 to 5, where scores closest to 5 indicate higher levels of interest in science. In this study, Cronbach’s alpha indicated strong internal consistency with a value of 0.96.

• The MPCS and MDLSS scores were ranked from 1 to 5. An average of individual response was measured, and scores closest to 5 indicated higher levels of interest. Secondarily, motivations were also classified as either intrinsic or extrinsic on each scale according to the original survey items, and average responses for each type of motivation were calculated. Intrinsic responses were subtracted from extrinsic responses, as suggested by Phillips et al. (2017), to produce a “total response,” where positive scores indicate predominately intrinsic motivations, and negative scores indicate predominately extrinsic motivations. The MPCS exhibited a Cronbach’s alpha score of 0.67 for intrinsic items 0.79 for extrinsic items. The MDLSS Cronbach’s alpha score was 0.71 and extrinsic items had a score of 0.68.

Surveys were adapted for Blue Thumb by aligning terminology and streamlining questions. Participants took less than 15 minutes on average to complete the entire survey. To ensure validity, participants had to answer all scale items, and attention filler questions were included in each scale to ensure careful survey reading before responding.

Qualitative questions at the end of each survey item were designed to provide open-ended responses that probed participants further about each scale/learning outcome. These questions focused on what environmental behaviors each participant engages in, how training/monitoring has changed their interests, their driving motivations for originally signing up for a citizen science program, and what motivations or interests sustain their continued participation in Blue Thumb. Qualitative questions were intended to enhance meaning or give context to the previous quantitative survey results.

Sampling and survey distribution

Blue Thumb volunteers were categorized as “new volunteers” or “experienced volunteers” through purposeful sampling (Palinkas et al. 2019). New volunteers, aged 18 or older, newly joined Blue Thumb, completed a two-day training, and hadn’t started stream monitoring. Experienced volunteers, also 18 or older, completed the same training and had at least three months of active monitoring with data entry at a stream site. Different survey distribution methods were employed based on volunteer category. New volunteers received in-person surveys at the end of Blue Thumb training events. Experienced volunteers had two survey options: a) a digital survey through Qualtrics™ survey software (Qualtrics 2020), or b) hard copies distributed at semi-annual events for those with limited computer access. Experienced volunteers were given additional open-ended reflective questions that were not included in new volunteer surveys.

Data analysis

Quantitative survey data, analyzed using SPSS software, yielded descriptive statistics, and a chi-squared test explored demographic differences based on volunteer experience levels. Significant differences in learning outcomes were assessed between new and experienced volunteers through non-parametric methods (Mann-Whitney U, p < 0.05) due to non-normal data distribution and ordinal data. Post-hoc analyses were conducted for the Motivation for Participation in Citizen Science Scale, comparing intrinsic and extrinsic participation scale items, along with total participation motivation scores. Qualitative data analysis, facilitated by QDA Miner software, employed structural coding to tag responses with “behavior,” “interest,” or “motivation” to identify frequencies. In the final step, quantitative and qualitative data was compared to provide a more comprehensive understanding of our results. This involved comparing survey score items with qualitative comments from individual volunteers to analyze how participants’ qualitative responses explained their scale scores for behavior, interest, and motivation (Creswell et al. 2011).

Demographic summary

The full dataset contained new volunteer (n = 41) and experienced volunteer responses (n = 33), for a total of 74 participants (Table 2). However, not all participants answered every question; thus, sample sizes vary among demographic parameters. Demographically, new volunteers were typically younger and still in college whereas experienced volunteers were older and had already completed college (Table 2). No meaningful difference in gender was noted between experience level, with approximately two-thirds of volunteers identifying as female. Overall, Blue Thumb volunteers are a highly educated group with existing science training. A chi square test of independence showed no significant association between experience level and demographic data.

Comparing new and experienced volunteers

To assess the impact of length of participation from the Likert scale data, Mann-Whitney U tests were conducted in SPSS to compare survey results for desired outcomes (behavior, interest, and motivation) between new (level “N”) and experienced (level “E”) volunteer groups (Table 3). Motivation for Participation in Citizen Science was significantly higher for experienced volunteers compared with new volunteers (p = 0.03). There were no statistically significant differences between new and experienced volunteers for behavior and stewardship, interest in science, and motivation for doing/learning science.

On the Behavior and Stewardship Scale, volunteers averaged a total behavioral score of 29.77 out of 36 possible points (Table 3), indicating high levels of pro-environmental behaviors. On the survey items, volunteers reported that they were currently engaged (averages above a score of 4, “I am currently doing this”) in all the listed pro-environmental actions (Figure 1). Qualitative responses from all volunteers revealed additional categories of environmental behaviors not captured on the survey items, such as sustainable shopping (19%), energy use (15%), alternative transportation (14%), plastic reduction (14%), composting (12%), stream clean ups (9%), and dietary changes (9%).

Figure 1

For the qualitative questions given only to experienced Blue Thumb volunteers, 13 participants indicated a change in either attitude or behavior based on their participation. Several experienced volunteers described how advocacy and education had become new and important behaviors for them. A majority of responses in this category reported participation in additional activities that ranged from educational presentations at regional conferences, virtual environmental webinars, and participation in other citizen science events such as BioBlitz, iNaturalist, and Project WET. A total of four participants mentioned they had not noticed any changes in attitude or behavior since joining Blue Thumb. Volunteers indicate that they already had established pro-environmental attitudes and behaviors before joining Blue Thumb.

On the Interest in Nature Scale, volunteers had a median response score of 4.41 (Table 3) indicating high levels of interest in nature and science. On the survey items, volunteers reported that they were interested (averages approaching a score of 4, “Agree”) in all the listed topics of interest excluding “interested in making tables or reports” (Figure 2). All volunteers were asked which aspects of stream-monitoring they were most interested in within an open-ended survey question. The most reported interests included biological observations of macroinvertebrates (31%) and fish (30%), chemistry (24%), and being in nature (23%).

Figure 2

Experienced volunteer responses described how, if at all, their interest had changed since beginning in Blue Thumb. Slightly over half of experienced volunteers (18 responses) responded to open-ended questions that their perceived level of interest had increased over time. In some cases, these were increases to specific interests (such as more interest in data interpretation and education of others), or increased interest in participation (more dedication to consistent, routine monitoring of a stream). A total of 8 experienced respondents reported no overall change in interest, either indicating that they had high levels of pre-interest before joining Blue Thumb or that there was a consistent level of interest from when they had begun. One experienced volunteer said, “My interests have not changed significantly. I have always cared for every aspect of environmental science.”

Qualitative responses from experienced volunteers also identified barriers or challenges that made it more difficult for them to explore their interests. The largest of these challenges was time: Experienced volunteers felt that though they wanted to engage in topics beyond the required stream monitoring aspect of Blue Thumb, they did not have enough personal time to do so. One volunteer described their barriers to interest as follows: “I would like to be more involved beyond stream monitoring with Blue Thumb but have trouble finding the time.”

Two modified motivational scales were used: the Motivation for Doing and Learning Science Scale (Porticella et al. 2017a,b) and the Motivation for Participation in Citizen Science Scale (Phillips et al. 2017). On the Motivation for Doing and Learning Science Scale, new volunteers’ median response score was 1.50, whereas experienced volunteers’ median response was 1.55, where positive scores indicate intrinsic motivations (Table 3). On the survey items, volunteer responses were above 4 (“I agree”) for intrinsic items, and less than 4 for extrinsic items (Figure 3). Significant differences were found between new and experienced volunteers on the Motivation for Participation in Citizen Science Scale (Table 3). New volunteers’ median response score was 0.61, whereas experienced volunteers’ median response score was 0.95, indicating higher levels of intrinsic motivation in experienced volunteers. The intrinsic motivation score was not significantly different between new and experienced volunteers (p = 0.86), but extrinsic motivations and overall motivations for participation were both statistically significant (p = 0.02 for both) (Figure 4; Table 4).

Figure 3

Figure 4

Qualitative questions related to motivations for participating in Blue Thumb and/or monitoring a stream revealed several different categories of motivations, ranging from interest in learning about water quality to promoting education (Table 5). Volunteers commonly expressed more than one motivation as their reason for participation. Experienced volunteers reported motivation largely attributed to personal feelings or place-based attachments. Volunteers who had continuously monitored at the same site for the duration of their participation mentioned the connection they felt to these places. Many experienced volunteers described how “contributing” to something was associated with positive feelings, or a sense of responsibility and duty. New volunteers reported more motivation for creating/maintaining social relationships between themselves, friends and family, or other Blue Thumb volunteers. Some volunteers, usually new volunteers and younger college students, described participation in Blue Thumb as a way of gaining career-specific skills.

Limitations

A crucial aspect of this study’s findings is the geographic locations of Blue Thumb’s training events. Throughout the research period, many training sessions took place at public educational institutions, ranging from regional colleges to public libraries. These venues primarily attracted the local population, characterized by younger age and higher education demographics. The geographic ties of new participants to program-hosted events may introduce a bias in the volunteer base. Notably, most volunteers at Blue Thumb training events did not represent rural areas or underrepresented groups. This may have made it more likely to capture volunteers with a higher likelihood of having scientific backgrounds and engagement in environmental activities, skewing the overall generalizability of scale items. A directly comparable control group for new and experienced volunteers was unfeasible. The different sampling methodologies potentially introduce sampling bias to the research’s limited sample size (74 participants: 41 new volunteers and 33 experienced volunteers), and could limit the study’s validity despite achieving a relatively high response rate among the 138 active Blue Thumb participants.

Additionally, the scale items used in this study were developed by the Cornell Lab of Ornithology. Items for motivation marked intrinsic or extrinsic, and how to interpret these data, were included in the original scale document. The authors’ use of recommended interpretations from these scales may have limitations, particularly in the calculation of average scores across behavior, interest, and motivation scales. Notable discrepancies arise in how scores were processed: Behavior scores were summed, interest scores averaged, and motivation scores derived by subtracting intrinsic from extrinsic scores, potentially oversimplifying the relationship between these motivations and raising concerns about the validity of findings.