Human Supervision is Key to Achieving Accurate AI-assisted Wildlife Identifications in Camera Trap Images

Introduction

The cumulative impacts of human activities are creating rapid changes in the natural world, necessitating frequent measurements and updates to conservation management strategies (Pacifici et al. 2015; Bonebrake et al. 2018; Schlaepfer and Lawler 2023). To understand and mitigate these challenges on effective timescales, ecologists and conservation practitioners require timely access to large amounts of data on wildlife populations. Remote sensors are increasingly deployed to evaluate how habitats and species respond to global or local changes and management or conservation interventions (Anderson et al., 2016; Davis et al., 2023; Kays et al., 2010; Palmer et al., 2022). Among these sensors, motion-activated wildlife cameras or “camera traps” are widely used due to their ability to collect detailed data on animal presence, behavior, health, and population structure with minimal disturbance. Since 2010, the number of ecologists using camera traps has grown exponentially due to the hardware’s relatively low cost, ease of use, and commercial availability, accelerating the pace and volume of ecological data collection and aggregation (Burton et al. 2015; Steenweg et al. 2017). This work has laid the foundations for novel modeling approaches and large-scale inference; however, the power and accuracy of these insights rely on the classification accuracy of raw camera trap (CT) image data (Burton et al. 2015; Caravaggi et al. 2020; Farley et al. 2018; Hofmeester et al. 2020; Wevers et al. 2021; Laporte-Devylder et al. 2023).

Surveys composed of dozens to hundreds of CTs may produce tens of thousands of images over several months, which must subsequently be labeled with the observed species, counts of individual animals, and other variables pertinent to the research or conservation action in question. In the gathered datasets, many “empty” images triggered, for example, by windblown plants or overheated electronics, must be labeled as such and set aside. Even the resulting “animals only” dataset may contain numerous species that are not the focal species relevant to the research or conservation question at hand. This scenario is becoming commonplace as long-term ecological monitoring is adopted to track and combat the current extinction crisis (Ceballos et al. 2017).

Managing such volumes of data can rapidly exceed the capacity of most CT practitioners, leading many to request volunteer help (i.e., citizen or participatory science) via online platforms, such as Zooniverse.org, that facilitate public participation in scientific research (Gadsden et al., 2021; Hsing et al., 2018; Islam and Valles, 2020; Jones et al., 2018). The Zooniverse is the world’s largest citizen science platform, with greater than 2.7 million registered users who assist projects across multiple scientific disciplines, including ecology, astronomy, and biomedical research (Simpson et al., 2014; Trouille et al., 2019). The first wildlife image–based project to launch on Zooniverse was Snapshot Serengeti, a survey of 225 CTs running continuously since 2010 in Serengeti National Park, Tanzania (Swanson et al. 2015; Palmer et al. 2021). Using a suite of tools to narrow the potential species options by body shape, horns, pelage, color, and patterns, and other distinguishing characteristics, citizen scientists were asked to categorize the species observable in each picture along with the number of individuals and basic behaviors and demographics. Separate details were collected about every species in an image when multiple species were visible. Each image was classified by up to 20 volunteers whose responses were aggregated. The species receiving the most votes was accepted as the final label.

When the public-facing webpage launched in 2013, the project had a backlog of 1.2 million observations that were processed by volunteers in three days (Swanson et al., 2015). In the subsequent decade, more than 120 similar CT projects have joined Zooniverse, including 24 additional project pages for the expanded Snapshot Safari network launched in 2018, in which Serengeti is the flagship survey (Pardo et al. 2021). Snapshot Safari is the largest use case on Zooniverse, with 50+ CT surveys in six African countries collecting data using the same protocols. (Surveys with < 10 CTs and data from temporary CT sites are folded into fewer project pages on Zooniverse to facilitate data processing pre- and post-classification.) It is also popular with volunteers, with > 200,000 users from 77 countries logging on regularly to assist with wildlife image classification. Still, increased competition for a finite number of willing volunteers extends the interval necessary to obtain fully and accurately labeled image datasets and return the information they contain to researchers and conservation managers.

Increasingly, improvements in computational power and the lengthening time required to obtain image labels through citizen science have motivated researchers to develop artificial intelligence (AI) models for quicker data classification (Norouzzadeh et al. 2017; Borowiec et al. 2022; Vélez et al. 2023). This presents its own set of challenges, as training a highly accurate species-classifying algorithm (hereafter, “classifier”) has traditionally demanded large volumes of labeled data, on the order of thousands of images per species category (“class”). This process therefore requires millions of labeled images (i.e., a “many shot” approach) for highly diverse ecosystems like the African savanna (Pantazis et al., 2024). A typical CT survey produces data with a long-tailed distribution, in which a few classes contain the vast majority of images while most classes contain relatively few (Miao et al. 2021). Thus, AI models often perform better at identifying highly abundant or visible species than rare and cryptic species that are captured less frequently or threatened species with low population sizes.

Methods

The Snapshot Safari network first deployed AI in 2018, using labeled image data from the Serengeti CT grid and eight other African CT surveys to create a custom classifier for processing the high volumes of data produced by continuous long-term monitoring (Willi et al., 2019). These models are available on GitHub for open access use (See Data Availability Statement 1). Due to Snapshot Serengeti’s early entry into the field, many existing classifiers for African wildlife (including transfer learning techniques and broader extensions of CT data) have been built using the training data labeled by volunteers (Battu and Reddy Lakshmi, 2023; Liu et al., 2024; Norouzzadeh et al., 2021; Villa et al., 2017). All publicly available data is posted to the Labeled Image Library of Alexandria for Biology and Conservation (see Data Availability Statement 2). Although rigorously assessed in an academic context, the specialized Snapshot Safari classifier was largely untested in practice. As such, we opted for a “humans-in-the-loop” (HITL) approach when integrating the AI model into our pipeline to ensure the accuracy of labels prior to use for research and conservation purposes.

Prior to incorporating AI, all image labels were generated by Zooniverse citizen scientists. Capture events consisting of one to three images (three taken in rapid succession during the day and one at night) were evaluated by up to 20 volunteers before retirement (removal from circulation), and a consensus algorithm was run to determine the species with the highest number of votes, which became the ultimate label (Swanson et al. 2015). All images within a capture event were kept together throughout the classification process as they constitute a single observation, and more data may be gleaned from a series of photos than one.

Though the urgent necessity of returning processed data quickly has made it no longer possible to rely on citizen science alone, AI-generated labels are known to be fallible, and many researchers recommend carefully evaluating these predictions to assess accuracy across time, space, and species (Clarfeld et al., 2023; Whytock et al., 2021). AI accuracy varies for many reasons, including low numbers of training images, differing background vegetation, difficult-to-interpret images (Westworth et al., 2022), and/or model drift, that is, the tendency of algorithms to become outdated and lose accuracy over time, which may necessitate additional transfer learning and training (Ackerman et al., 2021). Prior work has documented that volunteers are accurate 96.6% of the time across Serengeti species classes (Swanson et al., 2016), demonstrating that humans effectively identify species even when they have never or rarely viewed them. Therefore, we developed a hybrid approach to classifying images in which human volunteers supervised the classifier’s labels, that is, a HITL strategy. Here, we compare labels produced by AI and HITL against a newly created gold standard (GS) dataset annotated by professional ecologists to evaluate the impact of human guidance on final precision and accuracy.

Snapshot Safari AI models

The Snapshot Safari data pipeline incorporates labels generated by two convolutional neural network (CNN) models trained using the ResNet-18 model architecture (He et al., 2016) and Tensorflow (Abadi et al., 2016). Training data comprised 3.66 million CT images from nine African protected areas participating in Snapshot Safari (five in South Africa, three in Tanzania, and one in Mozambique). CNN models learn the intrinsic features of training data by updating model parameter weights and subsequently outputting inference results or “predictions” on unseen data; therefore, training data distribution has a crucial impact on model performance (Pantazis et al., 2024). Serengeti data constituted 89% of the training images because these models were created shortly after the inception of Snapshot Safari and contained only the first six months of data from the other CT grids. Therefore, we focused on the Serengeti GS dataset for these analyses. The classifier was trained by randomly assigning all images to one of three datasets while preserving class distributions: training (90% of the data), validation (5%), and testing (5%). The models learned from the training set, and were monitored on the validation set to reduce overfitting (Willi et al., 2019).

The Snapshot Safari object detection CNN (“detector”) was trained to flag empty images, which can constitute a significant percentage of images from CTs situated in grasslands or other biomes with rapid vegetation growth. It produces a binary output of “empty” or “not empty” and a confidence (probability) score indicating certainty in its prediction. The detector was evaluated on 71,702 test samples, of which 74.38% were rated as “high confidence,” set at a minimum threshold of 95%. The detector achieved an overall accuracy score of 96.03% rising to 99.5% on high-confidence images.

The species-classifying CNN (“classifier”) generates predictions with associated confidence scores (the probability of a match for every species class based on the classifier’s perception of similarity) for 56 species classes. The classifier was assessed on 36,469 test captures, 69.53% of which were labeled as high confidence, again set at 95% minimum threshold. As differences in regional wildlife morphology, topographic heterogeneity, and background vegetation impact AI accuracy (Beery et al., 2018), we sampled a diverse mixture of training photos across sites, seasons, and species while selecting all available images of rarely seen species. The species assemblages are similar at most of the Snapshot Safari sites in eastern and southern Africa, but regional variations in morphology and colloquial species names exist, so we harmonized the taxonomies prior to training.

The overall model accuracy was 89.4%, rising to 97.2% for high-confidence images. The three most commonly observed species in the Serengeti—wildebeest (Connochaetes taurinus), plains zebra (Equus quagga), and Thomson’s gazelle (Eudorcas thomsonii)—accounted for ~65% of Serengeti training images (see Figure 1). Wildebeest images were 94% accurate across the board, which improved to 98% accuracy on images rated as high confidence. For the wildebeest class, 80% of the images in the testing dataset were marked as high confidence by the classifier. Zebra were correctly identified in 93% of all images and 96% of high-confidence images (86% of test set), and Thomson’s gazelles were correct 96% of the time, improving to 99% on high-confidence images (77% of test set). In contrast, 15 species classes returned fewer than five available training images per class, resulting in 0% accuracy for 11 classes in the testing stage. Two species within this group returned 40% accuracy—bushpigs (Potamochoerus larvatus) and wild dogs (Lycaon pictus)—but zero images were classified as high confidence.

Figure 1

With these species-specific accuracies in mind, we determined that species classes in which many subjects are returned with high confidence scores can be processed more quickly using human supervision as a guardrail to ensure accuracy, whereas low-confidence images should be reviewed by more people when consensus is not achieved early in the classification process. Images of species for which the classifier demonstrated low accuracies were circulated for additional votes to ensure human volunteers converged on the correct label, which also helped to generate additional training data.

Snapshot Safari HITL pipeline

To combine our citizen science and AI processing pipelines, we presented volunteers with images that had also received AI classifications. (See Palmer et al. (2021) for a more detailed description and flow chart of the pipeline.) Volunteers then completed two stages of labeling—detection and classification. The detection stage consisted of a question task, in which volunteers were asked to evaluate whether animals were present in the capture (“Empty or not?”). Where once as many as 20 people needed to vote on whether a capture was empty to ensure an accurate label, adding AI predictions allowed for the reduction of volunteer effort at this stage, provided they agreed with the detector’s label. If two people agreed with the AI prediction that a capture was empty, it was labeled empty. If one of the first two people to view a capture disagreed with the AI label, it remained in circulation to accumulate three more human votes, at which point the majority consensus was accepted.

Captures marked as containing wildlife were moved to the classification phase, in which species identity was annotated. Within the species classification menu, volunteers compared a CT image with several exemplar images of each species from different angles, read detailed descriptions of species traits, and reviewed “confused with” pairs side by side to compare the likeness of two species, as depicted in Figure 2. They could also filter species by features such as body type and horn shape, and turn to detailed field guides that provided additional tips on selecting the correct species.

Figure 2

At this stage, more complex and dynamic rules were used to label captures based on the AI species label, the AI confidence score, the number of volunteer votes, and the agreement of the volunteers with the classifier and one another. The objective was to attain the correct species label as quickly as possible without sacrificing accuracy. For instance, if the classifier returned an 80% confidence score that an image contained a wildebeest (a frequently observed animal on which AI accuracy is typically high), it was assigned that label when the first five volunteers confirmed that a wildebeest was present. This rule was also implemented for zebras and impala (Aepyceros melampus), an antelope species with high abundances across Snapshot Safari field sites. Consensus and classification counts were dynamically checked as classification advanced. For most of the species classes, captures were retired as they reached a consensus of at least 50% at ten votes or 25% at 15 votes. If consensus >50% was not reached after 15 votes, that capture was reviewed by the research team to assign a label. Many images that failed to reach consensus captured an incomplete animal or were the result of the animal being too close to the camera. The three most common species were assigned labels after ten votes maximum. This reduction in the number of votes helped to save time and volunteer effort across the classification process. The ultimate label is the combined effort of AI and humans, described as HITL in the following analyses. The AI values are the top predictions produced by the classifier on unseen CT observations without further training or human assistance.

Analysis and Results

Across all species classes, AI returned an error rate of 34.89%, and HITL reduced the error rate to 8.73%. As shown in Figure 3, the HITL method outperformed AI in 95% of mammal and large bird classes (42 of 44), resulting in a total error reduction of 26.15%. AI achieved its best performance on highly abundant species within the Serengeti, that is, classes for which many training examples were provided. The classifier was trained on 10,079 images of wildebeest and returned an initial error rate of 7.63% on newly presented CT data. HITL reduced the error rate to 5.93% when compared with the GS responses. There were 6,676 training images of zebra, for which AI produced an error rate of 5.63% on unseen images. Adding the volunteers’ input reduced the error rate to 1.25%. Thomson’s gazelles (5,579 training images) were a notable exception to this trend—AI produced an error rate of 7.24%, which rose to 9.5% in our evaluation of the HITL classification method. The largest improvements in error rates came in species classes that are easily recognizable to humans, such as giraffe (Giraffa giraffa), which improved from a 5.59% error rate in AI to 0.62% in HITL; hippopotamus (Hippopotamus amphibius), which improved from 8.24% to 2.35%; and cheetahs (Acinonyx jubatus), which improved from 37.95% to 4.82%.

Figure 3

For species for which fewer than 100 images were available to train the classifier, the HITL classification method yielded an average error rate reduction of 37.94% on novel CT data, except for aardwolves (Proteles cristata), in which the AI error rate of 37.19% was worsened to 44.63% by human classifications. For many of the species in this < 100 category, HITL improved final labels but still returned unacceptable levels of accuracy. For instance, images of steenbok (Raphcerus campestris), a small antelope that is rare in the Serengeti, returned a 100% error rate by AI and 48.57% by HITL. However, there were significant gains in accuracy in other rarely photographed species like leopards (Panthera pardus), which saw an error rate reduction from 35.48% for AI to 4.30% for HITL; ostriches (Struthio camelus) dropped from 25.17% to 2.10%; and waterbuck (Kobus ellipsiprymnus) decreased from 41.67% to 8.33%.

Discussion

A species classifier confronted with a seldom-seen species, camera angle, or location is highly likely to make a classification error (Beery et al., 2018). Snapshot Safari runs a custom classifier trained on data generated by long-standing CT survey grids, yet HITL is still required to ensure accuracy for most species classes when AI is applied to new data from the same sites. The amount of training data is crucial since classifiers cannot identify animals without a plurality of examples to pull from. Humans, on the other hand, can use exemplar images, descriptions (including how to discern the differences between species that are morphologically similar), field guides, and assistance from researchers and moderators to make determinations.

The largest performance improvements achieved by HITL were in classes for which few training images were available, for example, predators and other cryptic species. With just one image of a zorilla (Ictonyx striatus) in the comprehensive training dataset, it is expected that AI will not be sufficiently effective for this class or others with similarly low availability of training images. The effect of small class size may be mitigated for species exhibiting unique morphological traits as in the case of crested porcupines (Hystrix cristata) in this dataset. With only 26 training images, the classifier returned an error rate of 5.15%, the lowest of any class. HITL reduced the error rate to zero.

Within the predator guild, with 29 capture events marked as “caracal” (Caracal caracal) by experts and just four caracal images in the training dataset, AI predicted the wrong species 82.76% of the time, whereas HITL reduced the error rate to 3.45%. Of the 24 errors by AI in caracal images, the most common confusion was lion (Panthera leo). Several classes that were confused by AI and HITL are shown in Figure 4. Caracals and lions are both medium-large cats with tawny coats, but humans readily distinguished between them even at night due to morphological differences in their ears and tails.

Figure 4

It is important to note that improvement from involving humans in the classification effort is context-dependent, and there is also an element of bias in volunteer responses. In two species classes evaluated here, humans performed worse than the classifier. In both cases, species with similar morphologies confused volunteers, particularly in images where only a portion of an animal is visible. HITL accuracy is aided by batch aggregation of classifications to obtain the label with the highest consensus, so correct responses normally outweigh one incorrect vote, but multiple wrong votes can lead to an incorrect label. The volunteers’ level of agreement is tracked in our reporting to each Snapshot Safari site to provide information to the project team about which captures achieve consensus and which may warrant further review. Anything less than 60% consensus among human classifiers likely needs to be checked before use in research studies. As demonstrated here, species with known confusions should also be reviewed by researchers because occasionally humans achieve consensus on the wrong identification.

One instance in which consensus did not reliably produce the correct label was the species category “aardwolf,” in which so many people selected “striped hyena” that the HITL error rate was higher than AI despite only 13 images in the training dataset (see Figure 4). This may be attributed to observer bias from volunteers who are eager to see striped hyenas and hence overestimate their perceived occurrence in the study area. Since many captures of both species occur at night and may not have a view of the tail, these images can be tough for experts to decipher, as well.

In the other instance, Thomson’s gazelle was the third most common species in the AI training dataset and returned an error rate of 7.24% in AI that rose to 9.5% via HITL. Thomson’s gazelles are similar in appearance to Grant’s gazelles (Nanger granti) and impala, which are sympatric throughout the species’ range. Most of the HITL errors for these three species reflected confusion among them (see Figure 4). It is unclear why Thomson’s gazelle, in particular, elicited so many incorrect responses despite extensive field guides on how to tell these three species apart.

AI accuracy can vary widely depending on the ecosystem, species, and number of images available for training models. If researchers are focused only on the most abundant species, AI trained on many images may be sufficient to meet their needs. For projects evaluating community dynamics or a suite of species, quickly removing empty images and the most abundant species classes from circulation allows people to spend more time reviewing images of wildlife that the classifier does not recall well.

With the proliferation of CT studies and field surveys using multitudes of ecological sensors, researchers are searching for ways to quickly and accurately translate raw data into usable data points. AI is an appealing solution because it allows for rapid translation once a model has been trained. It has until recently been a daunting task to build labeled image libraries; however, citizen science provides opportunities for labeling data using crowdsourcing through citizen science platforms. Additionally, recent machine-learning techniques have successfully developed models using fewer training images for select species classes (Schneider et al., 2020; Shahinfar et al., 2020). Generalized models that are agnostic to locations and backgrounds, such as the object detection model MegaDetector (Beery et al., 2019); Beery et al. 2019), and the global species classifier created by Wildlife Insights (Ahumada and Fegraus 2019) represent another encouraging trend.

Yet even these improved AI models require ongoing human training, validation, and supervision to return sufficiently accurate results to inform research studies and conservation management interventions. Ecologists using AI should ensure that model labels are verified and corrected by humans as necessary after initial evaluations of error rates by class. Our results indicate that this validation can be turned over to citizen scientists in most cases, but careful assessment of model and volunteer performance on a case-by-case basis is warranted. It is also important that researchers communicate with volunteers about the introduction of AI into data pipelines using citizen science. Some volunteers may be concerned that they are being replaced and their skills are no longer needed, which could lead to fewer engagements and shorter classification sessions. They should be informed that they are reviewing and correcting AI labels so that they do not become complacent. Further, communicating with volunteers about next-generation projects with which they can engage is crucial to retention of a motivated volunteer workforce. Experienced Snapshot Safari citizen scientists are now asked to take on tasks that help with identifying individual animals within populations or contributing behavioral labels from CT projects that had not previously extracted that information from existing image collections.

Citizen science and AI offer researchers the ability to quickly and efficiently move images through data pipelines to achieve labels and create data points for use in conservation programs and research studies. Citizen science can be a time-consuming process and requires monitoring of talk forums while projects are active. When AI was introduced to the Serengeti pipeline, classification time was halved even using conservative rules for getting enough volunteer votes to ensure accuracy. Times could be reduced further still by lowering confidence thresholds and the number of votes required to achieve a label. In species classes in which AI error rates are low, these images may not need to be presented to humans for help. The involvement of well-trained volunteers can greatly improve the performance of AI alone and is recommended, particularly for CT surveys with large volumes of data.

Conclusions

• HITL produced fewer errors than AI for 95% of species classes evaluated with the Serengeti GS dataset and decreased the overall error rate from 34.89% to 8.73%, with larger gains in species for which fewer than 100 training images were available.

• AI accuracy varied widely and was correlated with the amount of training data available for a particular species. Species classes with high volumes of images and a high proportion of high-confidence scores may be sufficiently accurate to warrant removing HITL for the most common classes in a long-tailed distribution in addition to removing empty images.

• Fewer training images result in worse outcomes from both methods, but humans are more likely to converge on the correct answers due to context clues and consensus.

• Turning over the HITL piece to volunteers can save researchers time while still yielding high-quality labeled image data.

• Researchers incorporating AI into pipelines with citizen science should communicate with volunteers about how their efforts benefit the project and about new tasks suited to humans that will lead to more informative data for biodiversity monitoring.

Data accessibility statement

[1] Snapshot Safari trained CNN models:

The object detector and species classifier used for this work are available on GitHub: https://github.com/marco-willi/camera-trap-classifier.

[2] Labeled camera trap data:

Labeled Image Library of Alexandria – Biology and Conservation (LILA-BC) repository contains camera trap datasets from Snapshot Safari camera traps including data collected in Serengeti National Park (Tanzania), Camdeboo National Park, Karoo National Park, Kgalagadi Transfrontier Park, Kruger National Park, and Mountain Zebra National Park (South Africa) and the Enonkishu Conservancy (Kenya). New data is uploaded as classifications are provided by citizen scientists (maintained by Google AI): http://lila.science/datasets.

[3] Previous gold standard dataset for Snapshot Serengeti:

Swanson, Alexandra B. et al. (2016). Data from: Snapshot Serengeti, high-frequency annotated camera trap images of 40 mammalian species in an African savanna [Dataset]. Data Dryad. https://doi.org/10.5061/dryad.5pt92.

Funding Information

This work was supported by AZA’s Saving Animals from Extinction, Disney’s “Protect the Pride,” Microsoft’s “AI for Earth,” National Geographic Explorer grants, U.S. National Science Foundation grants: PRFB #1810586: IIS #1619177: OAC #1835530, National Aeronautics and Space Administration grant award #80 NSSC20M0057, and the National Research Foundation of South Africa. South African research permits were provided by South African National Parks PACC1403.

Competing Interests

The authors have no competing interests to declare.

Author Contributions

S. Huebner led the preparation of this manuscript. M. Palmer classified the Serengeti gold standard sample and provided support on manuscript writing. C. Packer provided input on analyses and structure. All other contributions shared equally.