NRTR: Neuron Reconstruction with Transformer from 3D Optical Microscopy Images
Yijun Wang,Rui Lang,Rui Li,Junsong Zhang
DOI: https://doi.org/10.1109/tmi.2023.3323466
IF: 10.6
2023-01-01
IEEE Transactions on Medical Imaging
Abstract:The neuron reconstruction from raw Optical Microscopy (OM) image stacks is the basis of neuroscience. Manual annotation and semi-automatic neuron tracing algorithms are time-consuming and inefficient. Existing deep learning neuron reconstruction methods, although demonstrating exemplary performance, greatly demand complex rule-based components. Therefore, a crucial challenge is designing an end-to-end neuron reconstruction method that makes the overall framework simpler and model training easier. We propose a Neuron Reconstruction Transformer (NRTR) that, discarding the complex rule-based components, views neuron reconstruction as a direct set-prediction problem. To the best of our knowledge, NRTR is the first image-to-set deep learning model for end-to-end neuron reconstruction. The overall pipeline consists of the CNN backbone, Transformer encoder-decoder, and connectivity construction module. NRTR generates a point set representing neuron morphological characteristics for raw neuron images. The relationships among the points are established through connectivity construction. The point set is saved as a standard SWC file. In experiments using the BigNeuron and VISoR-40 datasets, NRTR achieves excellent neuron reconstruction results for comprehensive benchmarks and outperforms competitive baselines. Results of extensive experiments indicate that NRTR is effective at showing that neuron reconstruction is viewed as a set-prediction problem, which makes end-to-end model training available.
engineering, biomedical,imaging science & photographic technology, electrical & electronic,computer science, interdisciplinary applications,radiology, nuclear medicine & medical imaging