Consistency and Validity of Participatory Science Data: A Comparison of Seasonality Patterns of Northern California and Nevada Birds Across eBird and iNaturalist

The iNaturalist platform

iNat first launched in 2008 as a Masters project by graduate students at UC Berkeley (Seltzer 2024). In 2014, it was incorporated into the California Academy of Sciences (CalAcad), and from 2017 to 2023 was jointly managed by the CalAcad and the National Geographic Society. Since 2023, iNat has operated as a 501c3 nonprofit based in the United States. As of November 2024, iNat numbers more than 4.3 million individual observers who have posted more than 242 million observations of more than 505,000 total species worldwide. The iNat platform allows observers to post observations of any organism and provides AI-based suggestions on organism identification. Evidence of the organism (photo or audio recording) is not required, though it is heavily encouraged (as of November 2024, 97.7% of iNat observations included at least one photo and an additional 0.4% included audio). Once an observation is posted, other iNat observers can confirm or refute a given identification based on the provided evidence, and currently over 408,000 iNat observers have contributed identifications to others’ observations. Observations that meet certain data quality assessment criteria, such as including metadata and community agreement on identification, are deemed “research grade,” and observations not meeting these criteria are deemed “casual” observations. Currently, approximately 56.5% of observations are categorized as research grade. Lastly, iNat observations can include additional fields of data, such as sex, age, and type of evidence (for birds, this can include feather and egg). Track, scat, or other signs of an organism (such as nests) can also be posted as an observation. Each observation also has an open-ended notes section in which observers can document behavioral or ecological information. The inclusion of a photograph in observations also allows for ecological or behavioral data to be gleaned from an observation even if the observer did not note it in the moment.

The eBird platform

eBird was first launched in 2002 as a joint project between the Cornell Lab of Ornithology and the National Audubon Society. Initially, the platform focused on the Western Hemisphere, but was expanded to also include New Zealand in 2008, and the entire globe in 2010 (Sullivan et al. 2009; 2014). Observers can submit observations using the eBird website, a mobile app, or by importing lists from other software. The unit of observation is a checklist: a list of the number of individuals of each species recorded at a location within a fixed period of time. eBird guidelines encourage observers to start a new checklist each time they have traveled 5 miles or stayed at a single fixed location for 3 hours, and to submit checklists of all observed birds whenever possible (eBird Team n.d.). By the end of 2023, more than 930,000 birders had submitted more than 1.6 billion bird observations, and several hundred million observations are now submitted each year (eBird Team 2024). eBird observations are quality-controlled through a combination of automated filtering and human review by regional experts. This ensures the review of observations of unusual species or anomalously high counts, and publicly confirmed rare records are generally supported by attached text, photo, or audio evidence (Sullivan et al. 2014). Age and sex of the birds can be noted in the checklist, as well as codes that indicate confirmed or potential breeding. For each species, observers can also upload images and audio recordings and add open-ended text comments.

Data quality in participatory science

Since its inception, the field of participatory science has faced questions about data quality and reliability (studies relating specifically to iNat and eBird are outlined in more detail in the Supplemental File 1: Appendix A). Some studies have shown comparable precision and variability between volunteer- and professional-collected data (Lewandowski and Specht 2015; Kosmala et al. 2016), while others have reported low concordance among datasets and have cautioned against overstating the reliability of participatory science data (Aceves-Bueno et al. 2017). These concerns are being addressed by an increasingly robust collection of literature outlining best practices in data collection and reporting (e.g., Aceves-Bueno et al. 2017), statistically addressing error and bias (e.g., Bird et al. 2014), aligning data from multiple sources (e.g., Schacher et al. 2023), etc. In the words of Stevenson et al. (2021), “blanket criticism of [participatory science] data quality is no longer appropriate” and instead, researchers must consider the particular strengths and weaknesses of a given project’s structure, participants, and data output.

Integrating datasets with different biases, resolution, and sampling methodologies generates its own questions on best practices (Ball-Damerow et al. 2019; Miller et al. 2019; Van Eupen et al. 2021; Randler 2021; see Supplemental File 1: Appendix A for details of differences in user experience between iNat and eBird). Given the prevalence of using data from multiple projects directly or large-scale observation warehouses such as the Global Biodiversity Information Facility (GBIF), the question of how best to integrate data from multiple participatory science projects remains an important one.

Details and objectives of this study

The primary research objective of this project is to develop a methodology and merge criteria for combining individual observation data from different participatory science platforms (iNaturalist and eBird) using quantitative tools rooted in the cyclical nature of seasonality (e.g., circular statistical methods to reflect seasonality in annual records). Here, merging refers to the process of summing weekly species-level observation counts across distinct datasets to form a single dataset, pooling records together across platforms or years to increase the total sample size for downstream visualization and analysis; see Table 1 for a concrete illustration of an example merge for the acorn woodpecker across the iNat 2022 and eBird 2022 datasets. We first express the general trends in how eBird and iNat differ through exploratory statistics. Then, we explore one example of a biological application of cross-platform data: seasonality of migratory bird populations in Northern California. In this exploratory example, we report if and how the seasonality of merged iNat and eBird observation records match known migratory behavior of a subset of Northern Californian bird species. Finally, in the discussion we emphasize the benefits and importance of working together in a multidisciplinary team of domain specialists, quantitative researchers, and project participants with experience contributing and leveraging participatory science data.

Data

We consider the relative abundance of 254 species of birds in Northern California and Nevada, comparing seasonality from iNat and eBird data over 2019 and 2022. For a practical reference to these birds, see, for example, the Field Guide to Birds of California (Jaramillo and Small 2015). We chose a representative year before and after the COVID-19 pandemic, as shutdowns altered overall observer activity in both iNat and eBird, with some regions showing increased/decreased numbers of observations, as well as consistent shifts in observations towards urban or human-impacted environments (Crimmins et al. 2021; Hochachka et al. 2021; Sánchez-Clavijo et al. 2021).

Our dataset was acquired using eBird and GBIF Application Programming Interfaces (APIs), the latter used for research grade iNat observations. Research grade observations from iNat and eBird from 2019 and 2022 were filtered using a bounding box surrounding Northern California and Nevada, with boundaries at [36.470113°, 42.009518°] latitude and [–124.409591°, –114.131211°] longitude (Figure 1). To ensure that frequency curves were accurately estimable, we removed species with sparse observation counts, defined as absent in more than 46 out of the 52 weeks of the year, from any of the datasets. Our analysis was limited to species split taxonomically to the same degree in both datasets (e.g., ignoring subspecies), and excluded taxonomic revisions during the study period. After filtering, our dataset consisted of 254 unique species of birds, with counts of 3,806,983 (eBird 2019), 5,180,666 (eBird 2022), 68,334 (iNat 2019), and 134,312 bird sightings (iNat 2022).

Figure 1

For visualizing seasonal trends, we construct smooth seasonality curves from raw eBird and iNat observation count data, accessible via the NorCal Bird dashboard, which also provides capability to view original unsmoothed frequency data (Carroll and Waterman 2024). Fourier smoothing is used so that resulting densities respect the periodic nature of seasonal trends over the course of a year; visual inspection finds that truncating the Fourier expansion at B = 8 basis functions preserves fidelity to the raw data. Computation is implemented using the R package “fda” (Ramsay et al. 2024). Code is available at the OSF page provided in the Supplemental File 1: Appendix A.

Merge Criterion

To assess whether distinct databases (or years) should be merged for a species, we develop a resampling-based test to evaluate whether two samples of observations are generated from the same underlying density (Figure 2). For analysis, the eBird 2022 data was taken as the baseline and other datasets were compared relative to this reference. Calculations for merging use raw frequency distributions rather than smoothed curves to ensure the merge criteria is not influenced by the choice of smoothing method.

Figure 2

Differences between a pair of distributions are quantified with the circular optimal transport (COT) distance (Rabin, Delon, and Gousseau 2011). COT distance is suitable since optimal transport distances compare distributions while respecting the geometry of their support (Villani 2021). We modified a two-sample bootstrap version of the circular optimal transport test (Hundreiser, Klatt, and Munk 2021), described briefly in our context as follows.

Suppose for a given species, we seek to test for a difference between iNat 2022 and eBird 2022 observations. If the test finds a difference, we choose not to merge the two datasets; if not, we allow the merge.

To perform inference, the eBird 2022 observations are resampled to create 10,000 bootstrapped null distributions of the original sample size. The COT distance is calculated between the observed eBird 2022 distribution and each bootstrap-generated distribution; these distances are then used to construct an empirical null distribution of the COT distance. The p-value is the fraction of null COT distances that exceed the observed COT distance between the observed eBird 2022 distribution and the observed comparison distribution. A significant difference is detected if the p-value is less than the significance level. Because hypothesis testing requires the detection of a significant difference to prevent merging, species with low observation counts may suffer from low-powered tests and therefore experience higher false negative rates (i.e., Type II errors). This means the risk of mistakenly merging is higher for species with noisy seasonality signals and low observation counts. In such cases, visual inspection should be used as a secondary measure to confirm merging is appropriate.

One may perform this inferential step for multiple data sources and across many species, introducing multiple comparisons. To correct for several comparisons, we modify the merge decisions of the test using the Benjamini-Yekutieli procedure which controls the false discovery rate (i.e., the fraction of true merges missed) at a given level α (Benjamini and Yekutieli 2001). More concretely, for C comparisons indexed by c = 1, …, C, we sort the observed p-values in ascending order, p₍₁₎, …, p_(C), and detect a significant difference for all comparisons such that $p_{(c)}\hspace{0.17em}<\hspace{0.17em}\frac{c\alpha }{k_{C}C}$, where $k_C={\displaystyle \sum _{c=1}^C{c}^{–1}}$. For our analysis we have C = 3 × 254 = 762 comparisons and we bound the false discovery rate at 𝛼 = 0.05.

Data Comparison for iNat and eBird

eBird generally has more sightings, on the scale of 35 to 50 times more records of individual birds in a given year, which is consistent with previous reports (Jacobs and Zipf 2017) and is expected given the ability to report multiple birds at once in an eBird checklist (Table 2). For the 254 sample species, in 2019 there were about ~270 observations per bird in iNat, compared with ~15,000 in eBird. In 2022, these numbers increased: there were about ~525 observations per bird in iNat, compared to ~20,400 in eBird.

For 2022, an overwhelming majority of species (~99.6%) had more observations in eBird than iNat. The average species had about 40 times more observations in eBird than iNat. Only one bird was observed more in iNat than eBird in 2022: the cattle egret (Bubulcus ibis). These patterns are for the most part consistent for 2019 data. In 2019, the average species had about 65 times more observations in eBird than iNat. No bird was observed more on iNat than eBird in 2019.

Merge Criteria Results

The majority of species in each dataset are mergeable with the baseline eBird 2022 dataset (Table 3). The dataset most consistent with eBird 2022 was, unsurprisingly, the eBird 2019 dataset, with 97.6% of species classified as mergeable. iNat 2022 data was comparably consistent, achieving 97.2% mergeability. The iNat 2019 dataset exhibited the most discordance with eBird 2022, with 88.6% mergeability; though this rate suggests data is still fairly consistent for most species, even when varying databases and years.

Figure 3 displays three example species to illustrate the acceptable amount of variation in distributions across sources. Seasonality trends are visualized through relative frequency curves over time in Cartesian coordinates and in circular coordinates, which preserve the cyclic nature of the year and facilitates studying trends of birds whose prevalence is split across the beginning and end of the calendar year (e.g., the bufflehead).

Figure 3

The elegant tern (Thalasseus elegans) is an example bird whose distributions experience variation across platforms and years, but ultimately not enough to flag a significant difference: Each of the compared curves are similar enough to baseline (eBird 2022) to merge across all across datasets. This regularity corresponds to the elegant tern’s consistent annual migration to Northern California in late summer and early fall after breeding. The long-tailed duck (Clangula hyemalis), however, illustrates what more extreme deviations may look like. For this species, the merge criterion does detect a difference for some datasets: neither of eBird or iNat 2019 distributions fell within the acceptable merge envelope, though the iNat 2022 pattern was mergeable. Here, the merge criterion highlights a year-based difference in seasonality patterns. Long-tailed ducks are coastal birds present mainly in winter, but some individuals may linger through the summer. This inter-annual variation in summer occurrence prevents the data from meeting the merge criterion, even though it is not due to bias in detection or reporting.

Spring

The western kingbird represents a typical frequency pattern for the seasonal spring bird archetype with frequency curves that are consistent in shape and timing across platforms and years. The western kingbird is most frequently observed in the spring across both iNat and eBird, then observed at lower rates in summer into the start of fall. Birds then depart in late summer and early fall to return through the American southwest into Mexico.

Summer

The green-tailed towhee typifies a summer-only migrant in this region. Individuals winter in Mexico and parts of the southwestern US, and the bulk of their range during the breeding season spans the Great Basin and bordering mountain ranges. The first northbound individuals reach Northern California breeding sites in late April or early May. After breeding, southbound migration is completed by the end of September. This summer-only pattern is also likely to occur for other summer-breeding species that reach their northernmost extent within our study region.

Fall

The pectoral sandpiper is a model fall bird representative. Its frequency curve is characterized by observations almost only in the early weeks of fall. Pectoral sandpipers are long-distance migrants that breed in the arctic but winter largely in southern South America. During their northbound spring migration, they mostly move through areas east of the Rocky Mountains (Harrington 1984). However, their southbound migration is more longitudinally widespread, with birds occurring throughout the study region. This species demonstrates that data can satisfy inclusion and merging criteria even when sightings are relatively sparse. The iNat 2019 data included only 9 records of pectoral sandpipers, but they align with the predictable movements of this fall migrant.

Winter

The bufflehead typifies a winter prevalent species in Northern California and Nevada. Bufflehead observations start increasing in mid-autumn and peak in the early weeks of winter, then gradually decline through the remainder of winter and early spring. This pattern of winter occurrence primarily arises when species breed north of the study region and arrive in the region only for the winter period.

Year-round

The California scrub-jay (Aphelocoma californica) is non-seasonal, year-round resident of this region, with a frequency curve showing only minor variation across seasons (Figure 5a). California scrub-jays may occasionally undertake regional movements, for example if there is acorn crop failure in an area (Carmen 1988), but are not documented to have a large-scale seasonal migration.

Figure 5

Bimodal Prevalence

The western tanager (Piranga ludoviciana) represents a bimodal seasonal pattern. Its frequency distribution shows two observation peaks, one in late spring and a second in late summer, with fewer observations in early summer (Figure 5b). This pattern arises because most observations are likely to reflect passage migration—the movement of birds through a region to a final destination outside the region. The breeding range of the Western Tanager is primarily north of our study region, and the wintering range is primarily south of the study region.

Caveats in Interpreting Seasonal Patterns

A potential confound when interpreting seasonality patterns in participatory science datasets is seasonality in observer effort or behavior. Intentional observation drives (the City Nature Challenge, Global Big Days, etc.), as well as spatial and temporal variation in observer effort, could produce bias in observation records for a species that appears to reflect seasonality in presence of the species in the study area. The high degree of mergeability across the species presented in this study shows that these potential biases are not severe enough to warrant avoiding merging data entirely; however, they may intersect with other aspects of data bias or species behavior in ways that produce inaccurate outcomes.

The black-backed woodpecker (Picoides arcticus) is an example of a seasonality pattern that appears similar to migration-driven seasonal patterns, but which likely is primarily influenced by observer behavior. Its frequency curve is very consistent across data sources and years, beginning to increase in late spring and peaking in early summer with a slow decline in observation throughout the summer and into early fall (Figure 5c). While this pattern appears to support a summer-migratory behavior, black-backed woodpeckers are not known to have large-scale seasonal migration. The pattern of summer detection of black-backed woodpeckers may instead reflect seasonal variation in observer effort within the species’ range, rather than seasonal movements of birds through the study area. In California, black-backed woodpeckers are found in relatively high elevation and latitude regions (Saracco, Siegel, and Wilkerson 2011), which are less accessible in winter than other seasons. Our methodology of smoothing annual curves controls for overall effort in observing a species, but may still be susceptible to seasonal changes in the type of effort.

Variation in observer behavior can also lead to discrepancies between patterns in eBird and iNat. For example, occurrences of common murre, Uria aalge, peaked around the end of June both years in iNat while occurrence was highest in September in eBird (Figure 5d). Inspection of the spatial distribution of these sightings reveals that eBird has a high number of August, September, and October observations reported from deep water off the California coast. These reports generally come from pelagic birding trips, in which regularly scheduled boats travel to seabird hotspots many miles offshore. On these trips, the group generates a new checklist at least once per hour (eBird pelagic protocol), so a single trip can produce a dozen or more checklists, leading to many separate reports of birds like common murre. iNat, on the other hand, has no such protocol. Even when a birder uses iNat on a pelagic, they are unlikely to be repeatedly adding observations of different common murres (of which there may be many thousands observed). Because pelagic birding trips are conducted most frequently in these months, it produces a strong seasonal pattern that is only reflected on eBird.