Deep learning-based ecological analysis of camera trap images is impacted by training data quality and size

Omiros Pantazis,Peggy Bevan,Holly Pringle,Guilherme Braga Ferreira,Daniel J. Ingram,Emily Madsen,Liam Thomas,Dol Raj Thanet,Thakur Silwal,Santosh Rayamajhi,Gabriel Brostow,Oisin Mac Aodha,Kate E. Jones
2024-08-26
Abstract:Large wildlife image collections from camera traps are crucial for biodiversity monitoring, offering insights into species richness, occupancy, and activity patterns. However, manual processing of these data is time-consuming, hindering analytical processes. To address this, deep neural networks have been widely adopted to automate image analysis. Despite their growing use, the impact of model training decisions on downstream ecological metrics remains unclear. Here, we analyse camera trap data from an African savannah and an Asian sub-tropical dry forest to compare key ecological metrics derived from expert-generated species identifications with those generated from deep neural networks. We assess the impact of model architecture, training data noise, and dataset size on ecological metrics, including species richness, occupancy, and activity patterns. Our results show that while model architecture has minimal impact, large amounts of noise and reduced dataset size significantly affect these metrics. Nonetheless, estimated ecological metrics are resilient to considerable noise, tolerating up to 10% error in species labels and a 50% reduction in training set size without changing significantly. We also highlight that conventional metrics like classification error may not always be representative of a model's ability to accurately measure ecological metrics. We conclude that ecological metrics derived from deep neural network predictions closely match those calculated from expert labels and remain robust to variations in the factors explored. However, training decisions for deep neural networks can impact downstream ecological analysis. Therefore, practitioners should prioritize creating large, clean training sets and evaluate deep neural network solutions based on their ability to measure the ecological metrics of interest.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
### What problems does this paper attempt to solve? This paper aims to explore and evaluate the impact of model training decisions on downstream ecological indicators when deep neural networks process camera trap images. Specifically, the research focuses on the following aspects: 1. **Impact of model architectures**: How different deep neural network architectures (such as ResNet18, ResNet50, ResNet101, and ConvNeXt - T) affect the accuracy of ecological indicators. 2. **Impact of training data noise**: How the proportion of label noise in the training data (from 1% to 50%) affects ecological indicators such as species richness, occupancy, and activity patterns. 3. **Impact of training dataset size**: How changes in the size of the training dataset (from 1% to 100%) affect the above - mentioned ecological indicators. Through these analyses, the author hopes to answer the following key questions: - How accurate are the ecological indicators predicted by deep neural networks compared to data labeled by experts? - Can conventional machine - learning evaluation metrics (such as Top - 1 Accuracy, Precision, Recall, and F1 Score) reliably reflect the performance of models in ecological applications? ### Research background With the increasing global demand for biodiversity monitoring, camera traps, as a passive monitoring tool, can collect data on a large - scale and standardized basis. However, manually processing these image data is very time - consuming, which hinders the speed of data analysis. To improve efficiency, deep - learning methods are widely used in automated image analysis. Although deep neural networks perform well in classification tasks, their impact on downstream ecological analysis is still unclear. ### Research objectives The main objective of this study is to evaluate the impact of different training decisions (such as model architecture selection, training data quality, training data volume) on ecological indicators, thereby providing guidance for practitioners to help them optimize the application of deep - learning models to better support biodiversity monitoring and conservation work. ### Conclusions The research shows that although the impact of model architectures on ecological indicators is relatively small, the noise in the training data and the size of the training dataset significantly affect these indicators. Specifically: - Ecological indicators have a certain tolerance for a certain degree of noise and a reduction in the amount of training data (for example, a label error rate of up to 10% and a 50% reduction in training data will not significantly change the results). - Conventional machine - learning evaluation metrics (such as classification error) do not necessarily represent the performance of models in ecological analysis, so it is recommended to use evaluation criteria that are closer to the actual application scenarios. Through this study, the author hopes to improve the applicability of deep neural networks in the field of ecology and provide valuable references for future research and applications.