Application of a novel deep learning–based 3D videography workflow to bat flight

Jonas Håkansson,Brooke L. Quinn,Abigail L. Shultz,Sharon M. Swartz,Aaron J. Corcoran
DOI: https://doi.org/10.1111/nyas.15143
IF: 6.499
2024-04-25
Annals of the New York Academy of Sciences
Abstract:We introduce a workflow that integrates deep learning‐based tracking of anatomical landmarks with 3D reconstruction to analyze the biomechanics of animal flight. Tested on bats, both in a wind tunnel and a flight arena, the workflow's accuracy closely approximates that resulting from manual digitization. The results demonstrate the workflow's potential for significantly increasing the efficiency and scalability of biomechanical research without sacrificing accuracy. Studying the detailed biomechanics of flying animals requires accurate three‐dimensional coordinates for key anatomical landmarks. Traditionally, this relies on manually digitizing animal videos, a labor‐intensive task that scales poorly with increasing framerates and numbers of cameras. Here, we present a workflow that combines deep learning–powered automatic digitization with filtering and correction of mislabeled points using quality metrics from deep learning and 3D reconstruction. We tested our workflow using a particularly challenging scenario: bat flight. First, we documented four bats flying steadily in a 2 m3 wind tunnel test section. Wing kinematic parameters resulting from manually digitizing bats with markers applied to anatomical landmarks were not significantly different from those resulting from applying our workflow to the same bats without markers for five out of six parameters. Second, we compared coordinates from manual digitization against those yielded via our workflow for bats flying freely in a 344 m3 enclosure. Average distance between coordinates from our workflow and those from manual digitization was less than a millimeter larger than the average human‐to‐human coordinate distance. The improved efficiency of our workflow has the potential to increase the scalability of studies on animal flight biomechanics.
multidisciplinary sciences
What problem does this paper attempt to address?