Abstract:Markerless methods for animal posture tracking have been rapidly developing recently, but frameworks and benchmarks for tracking large animal groups in 3D are still lacking. To overcome this gap in the literature, we present 3D-MuPPET, a framework to estimate and track 3D poses of up to 10 pigeons at interactive speed using multiple camera views. We train a pose estimator to infer 2D keypoints and bounding boxes of multiple pigeons, then triangulate the keypoints to 3D. For identity matching of individuals in all views, we first dynamically match 2D detections to global identities in the first frame, then use a 2D tracker to maintain IDs across views in subsequent frames. We achieve comparable accuracy to a state of the art 3D pose estimator in terms of median error and Percentage of Correct Keypoints. Additionally, we benchmark the inference speed of 3D-MuPPET, with up to 9.45 fps in 2D and 1.89 fps in 3D, and perform quantitative tracking evaluation, which yields encouraging results. Finally, we showcase two novel applications for 3D-MuPPET. First, we train a model with data of single pigeons and achieve comparable results in 2D and 3D posture estimation for up to 5 pigeons. Second, we show that 3D-MuPPET also works in outdoors without additional annotations from natural environments. Both use cases simplify the domain shift to new species and environments, largely reducing annotation effort needed for 3D posture tracking. To the best of our knowledge we are the first to present a framework for 2D/3D animal posture and trajectory tracking that works in both indoor and outdoor environments for up to 10 individuals. We hope that the framework can open up new opportunities in studying animal collective behaviour and encourages further developments in 3D multi-animal posture tracking.

Context-Aware Outlier Rejection for Robust Multi-View 3D Tracking of Similar Small Birds in An Outdoor Aviary

Temporally-consistent 3D Reconstruction of Birds

Three-dimensional Multi-Target Tracking of Point Cloud from Bird's-Eye View

Visual Tracking of Small Animals in Cluttered Natural Environments Using a Freely Moving Camera

3D Bird Reconstruction: a Dataset, Model, and Shape Recovery from a Single View

3D-POP -- An automated annotation approach to facilitate markerless 2D-3D tracking of freely moving birds with marker-based motion capture

BEV-SUSHI: Multi-Target Multi-Camera 3D Detection and Tracking in Bird's-Eye View

Peering into the world of wild passerines with 3D-SOCS: synchronized video capture and posture estimation

Lifting Multi-View Detection and Tracking to the Bird's Eye View

Integration of the 3D Environment for UAV Onboard Visual Object Tracking

Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Multi-Object Tracking?

A Bayesian Filter for Multi-view 3D Multi-object Tracking with Occlusion Handling

3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking

Benchmarking the Complementary-View Multi-human Association and Tracking

Online Learning 3d Context for Robust Visual Tracking

Tracking Small Birds by Detection Candidate Region Filtering and Detection History-aware Association

Multi-camera Realtime 3D Tracking of Multiple Flying Animals

Object Tracking over Multiple Uncalibrated Cameras Using Visual, Spatial and Temporal Similarities.

Real-Time Multi-View Object Tracking In Mediated Environments

BiTrack: Bidirectional Offline 3D Multi-Object Tracking Using Camera-LiDAR Data

Open3DTrack: Towards Open-Vocabulary 3D Multi-Object Tracking