BigNeuron: a resource to benchmark and predict performance of algorithms for automated tracing of neurons in light microscopy datasets
Linus Manubens-Gil,Zhi Zhou,Hanbo Chen,Arvind Ramanathan,Xiaoxiao Liu,Yufeng Liu,Alessandro Bria,Todd Gillette,Zongcai Ruan,Jian Yang,Miroslav Radojević,Ting Zhao,Li Cheng,Lei Qu,Siqi Liu,Kristofer E Bouchard,Lin Gu,Weidong Cai,Shuiwang Ji,Badrinath Roysam,Ching-Wei Wang,Hongchuan Yu,Amos Sironi,Daniel Maxim Iascone,Jie Zhou,Erhan Bas,Eduardo Conde-Sousa,Paulo Aguiar,Xiang Li,Yujie Li,Sumit Nanda,Yuan Wang,Leila Muresan,Pascal Fua,Bing Ye,Hai-Yan He,Jochen F Staiger,Manuel Peter,Daniel N Cox,Michel Simonneau,Marcel Oberlaender,Gregory Jefferis,Kei Ito,Paloma Gonzalez-Bellido,Jinhyun Kim,Edwin Rubel,Hollis T Cline,Hongkui Zeng,Aljoscha Nern,Ann-Shyn Chiang,Jianhua Yao,Jane Roskams,Rick Livesey,Janine Stevens,Tianming Liu,Chinh Dang,Yike Guo,Ning Zhong,Georgia Tourassi,Sean Hill,Michael Hawrylycz,Christof Koch,Erik Meijering,Giorgio A Ascoli,Hanchuan Peng
DOI: https://doi.org/10.1038/s41592-023-01848-5
Abstract:BigNeuron is an open community bench-testing platform with the goal of setting open standards for accurate and fast automatic neuron tracing. We gathered a diverse set of image volumes across several species that is representative of the data obtained in many neuroscience laboratories interested in neuron tracing. Here, we report generated gold standard manual annotations for a subset of the available imaging datasets and quantified tracing quality for 35 automatic tracing algorithms. The goal of generating such a hand-curated diverse dataset is to advance the development of tracing algorithms and enable generalizable benchmarking. Together with image quality features, we pooled the data in an interactive web application that enables users and developers to perform principal component analysis, t-distributed stochastic neighbor embedding, correlation and clustering, visualization of imaging and tracing data, and benchmarking of automatic tracing algorithms in user-defined data subsets. The image quality metrics explain most of the variance in the data, followed by neuromorphological features related to neuron size. We observed that diverse algorithms can provide complementary information to obtain accurate results and developed a method to iteratively combine methods and generate consensus reconstructions. The consensus trees obtained provide estimates of the neuron structure ground truth that typically outperform single algorithms in noisy datasets. However, specific algorithms may outperform the consensus tree strategy in specific imaging conditions. Finally, to aid users in predicting the most accurate automatic tracing results without manual annotations for comparison, we used support vector machine regression to predict reconstruction quality given an image volume and a set of automatic tracings.