Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling and cloud-native open-source tools
Dan Biderman,Matthew R. Whiteway,Cole Hurwitz,Nicholas Greenspan,Robert S. Lee,Ankit Vishnubhotla,Richard Warren,Federico Pedraja,Dillon Noone,Michael M. Schartner,Julia M. Huntenburg,Anup Khanal,Guido T. Meijer,Jean-Paul Noel,Alejandro Pan-Vazquez,Karolina Z. Socha,Anne E. Urai,Larry Abbot,Luigi Acerbi,Valeria Aguillon-Rodriguez,Mandana Ahmadi,Jaweria Amjad,Dora Angelaki,Jaime Arlandis,Zoe C. Ashwood,Kush Banga,Hailey Barrell,Hannah M. Bayer,Brandon Benson,Julius Benson,Jai Bhagat,Dan Birman,Niccolò Bonacchi,Kcenia Bougrova,Julien Boussard,Sebastian A. Bruijns,E. Kelly Buchanan,Robert Campbell,Matteo Carandini,Joana A. Catarino,Fanny Cazettes,Gaelle A. Chapuis,Anne K. Churchland,Yang Dan,Felicia Davatolhagh,Peter Dayan,Sophie Denève,Eric E. J. DeWitt,Ling Liang Dong,Tatiana Engel,Michele Fabbri,Mayo Faulkner,Robert Fetcho,Ila Fiete,Charles Findling,Laura Freitas-Silva,Surya Ganguli,Berk Gercek,Naureen Ghani,Ivan Gordeliy,Laura M. Haetzel,Kenneth D. Harris,Michael Hausser,Naoki Hiratani,Sonja Hofer,Fei Hu,Felix Huber,Cole Hurwitz,Anup Khanal,Christopher S. Krasniak,Sanjukta Krishnagopal,Michael Krumin,Debottam Kundu,Agnès Landemard,Christopher Langdon,Christopher Langfield,Inês Laranjeira,Peter Latham,Petrina Lau,Hyun Dong Lee,Ari Liu,Zachary F. Mainen,Amalia Makri-Cottington,Hernando Martinez-Vergara,Brenna McMannon,Isaiah McRoberts,Guido T. Meijer,Maxwell Melin,Leenoy Meshulam,Kim Miller,Nathaniel J. Miska,Catalin Mitelut,Zeinab Mohammadi,Thomas Mrsic-Flogel,Masayoshi Murakami,Jean-Paul Noel,Kai Nylund,Farideh Oloomi,Alejandro Pan-Vazquez,Liam Paninski,Alberto Pezzotta,Samuel Picard,Jonathan W. Pillow,Alexandre Pouget,Florian Rau,Cyrille Rossant,Noam Roth,Nicholas A. Roy,Kamron Saniee,Rylan Schaeffer,Michael M. Schartner,Yanliang Shi,Carolina Soares,Karolina Z. Socha,Cristian Soitu,Nicholas A. Steinmetz,Karel Svoboda,Marsa Taheri,Charline Tessereau,Anne E. Urai,Erdem Varol,Miles J. Wells,Steven J. West,Matthew R. Whiteway,Charles Windolf,Olivier Winter,Ilana Witten,Lauren E. Wool,Zekai Xu,Han Yu,Anthony M. Zador,Yizi Zhang,John P. Cunningham,Nathaniel B. Sawtell,Liam Paninski,The International Brain Laboratory
DOI: https://doi.org/10.1038/s41592-024-02319-1
IF: 48
2024-06-26
Nature Methods
Abstract:Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce 'Lightning Pose', an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We released a cloud application that allows users to label data, train networks and process new videos directly from the browser.
biochemical research methods