Versatile multiple object tracking in sparse 2D/3D videos via deformable image registration
James Ryu,Amin Nejatbakhsh,Mahdi Torkashvand,Sahana Gangadharan,Maedeh Seyedolmohadesin,Jinmahn Kim,Liam Paninski,Vivek Venkatachalam
DOI: https://doi.org/10.1371/journal.pcbi.1012075
2024-05-22
PLoS Computational Biology
Abstract:Tracking body parts in behaving animals, extracting fluorescence signals from cells embedded in deforming tissue, and analyzing cell migration patterns during development all require tracking objects with partially correlated motion. As dataset sizes increase, manual tracking of objects becomes prohibitively inefficient and slow, necessitating automated and semi-automated computational tools. Unfortunately, existing methods for multiple object tracking (MOT) are either developed for specific datasets and hence do not generalize well to other datasets, or require large amounts of training data that are not readily available. This is further exacerbated when tracking fluorescent sources in moving and deforming tissues, where the lack of unique features and sparsely populated images create a challenging environment, especially for modern deep learning techniques. By leveraging technology recently developed for spatial transformer networks, we propose ZephIR, an image registration framework for semi-supervised MOT in 2D and 3D videos. ZephIR can generalize to a wide range of biological systems by incorporating adjustable parameters that encode spatial (sparsity, texture, rigidity) and temporal priors of a given data class. We demonstrate the accuracy and versatility of our approach in a variety of applications, including tracking the body parts of a behaving mouse and neurons in the brain of a freely moving C. elegans . We provide an open-source package along with a web-based graphical user interface that allows users to provide small numbers of annotations to interactively improve tracking results. While deep learning with convolutional neural networks has been successfully applied to many multiple object tracking problems, these advances do not immediately generalize to videos of fluorescence reported dynamics in living tissue, where the combination of sparse global distributions and locally dense, homogeneous peaks present a challenging instance of a multiple object tracking problem. Imaging such sparse fluorescent signals is a standard tool for observing neuronal activity in genetically engineered animals, and performing imaging in naturally behaving animals to place that activity in the context of behavior only increases the difficulty of the problem. Thus, this step is typically a significant bottleneck in efforts to understand the relationship between neuronal activity and naturalistic behavior. We build upon recent advances in spatial transformers and differentiable grid sampling to develop a new registration-based approach: ZephIR, a semi-supervised multiple object tracking algorithm with a novel cost function that can incorporate a diverse set of spatio-temporal constraints that can change dynamically during optimization. Local registration of image features enables tracking of keypoints even in sparse imaging conditions, such as fluorescent cellular data, while a spring network incorporates a flexible motion model of the neighboring keypoints without the need for a highly specialized skeletal model. Feature detection can help fine-tune tracking results to match a nearby detected feature in the image or even recover good tracking accuracy in cases where registration clearly fails to produce good gradients. In addition, any amount of new manual labor, whether simply verifying correct results or fixing incorrect ones, can dramatically improve ZephIR's accuracy. Through this workflow, ZephIR achieves state-of-the-art accuracy with minimal manual labor, even on a freely behaving C. elegans , where large deformations present a particularly challenging tracking problem.
biochemical research methods,mathematical & computational biology