Multi-camera Realtime 3D Tracking of Multiple Flying Animals

Andrew D. Straw,Kristin Branson,Titus R. Neumann,Michael H. Dickinson

DOI: https://doi.org/10.1098/rsif.2010.0230

2010-01-25

Abstract:Automated tracking of animal movement allows analyses that would not otherwise be possible by providing great quantities of data. The additional capability of tracking in realtime - with minimal latency - opens up the experimental possibility of manipulating sensory feedback, thus allowing detailed explorations of the neural basis for control of behavior. Here we describe a new system capable of tracking the position and body orientation of animals such as flies and birds. The system operates with less than 40 msec latency and can track multiple animals simultaneously. To achieve these results, a multi target tracking algorithm was developed based on the Extended Kalman Filter and the Nearest Neighbor Standard Filter data association algorithm. In one implementation, an eleven camera system is capable of tracking three flies simultaneously at 60 frames per second using a gigabit network of nine standard Intel Pentium 4 and Core 2 Duo computers. This manuscript presents the rationale and details of the algorithms employed and shows three implementations of the system. An experiment was performed using the tracking system to measure the effect of visual contrast on the flight speed of Drosophila melanogaster. At low contrasts, speed is more variable and faster on average than at high contrasts. Thus, the system is already a useful tool to study the neurobiology and behavior of freely flying animals. If combined with other techniques, such as `virtual reality'-type computer graphics or genetic manipulation, the tracking system would offer a powerful new way to investigate the biology of flying animals.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to develop a system that can track the positions and body orientations of multiple flying animals in real - time and with high precision. Specifically, the researchers designed a multi - camera - based system that can simultaneously track the positions and postures of multiple insects or birds with low latency (less than 40 milliseconds). To achieve this goal, the researchers developed a multi - target tracking algorithm, which combines the Extended Kalman Filter (EKF) and the Nearest Neighbor Standard Filter (NNSF). Through these techniques, the system can track targets with high temporal and spatial resolution in a large - scale space while reducing the impact of occlusion.

The paper mentions that this real - time tracking system can not only provide a large amount of data to support in - depth analysis of animal behavior, but also be used in manipulating sensory - feedback experiments, thereby exploring the basis of neural - control behavior. In addition, this system can also be used in combination with other technologies, such as virtual - reality computer graphics or gene manipulation, providing a powerful new tool for studying the biology of flying animals.

In summary, the main objective of the paper is to develop an efficient, real - time multi - target tracking system to support the study of the neurobiology and behavior of freely flying animals.