Giovanni Volpe,Carolina Wählby,Lei Tian,Michael Hecht,Artur Yakimovich,Kristina Monakhova,Laura Waller,Ivo F. Sbalzarini,Christopher A. Metzler,Mingyang Xie,Kevin Zhang,Isaac C.D. Lenton,Halina Rubinsztein-Dunlop,Daniel Brunner,Bijie Bai,Aydogan Ozcan,Daniel Midtvedt,Hao Wang,Nataša Sladoje,Joakim Lindblad,Jason T. Smith,Marien Ochoa,Margarida Barroso,Xavier Intes,Tong Qiu,Li-Yu Yu,Sixian You,Yongtao Liu,Maxim A. Ziatdinov,Sergei V. Kalinin,Arlo Sheridan,Uri Manor,Elias Nehme,Ofri Goldenberg,Yoav Shechtman,Henrik K. Moberg,Christoph Langhammer,Barbora Špačková,Saga Helgadottir,Benjamin Midtvedt,Aykut Argun,Tobias Thalheim,Frank Cichos,Stefano Bo,Lars Hubatsch,Jesus Pineda,Carlo Manzo,Harshith Bachimanchi,Erik Selander,Antoni Homs-Corbera,Martin Fränzl,Kevin de Haan,Yair Rivenson,Zofia Korczak,Caroline Beck Adiels,Mite Mijalkov,Dániel Veréb,Yu-Wei Chang,Joana B. Pereira,Damian Matuszewski,Gustaf Kylberg,Ida-Maria Sintorn,Juan C. Caicedo,Beth A Cimini,Muyinatu A. Lediju Bell,Bruno M. Saraiva,Guillaume Jacquemet,Ricardo Henriques,Wei Ouyang,Trang Le,Estibaliz Gómez-de-Mariscal,Daniel Sage,Arrate Muñoz-Barrutia,Ebba Josefson Lindqvist,Johanna Bergman

Abstract:Through digital imaging, microscopy has evolved from primarily being a means for visual observation of life at the micro- and nano-scale, to a quantitative tool with ever-increasing resolution and throughput. Artificial intelligence, deep neural networks, and machine learning are all niche terms describing computational methods that have gained a pivotal role in microscopy-based research over the past decade. This Roadmap is written collectively by prominent researchers and encompasses selected aspects of how machine learning is applied to microscopy image data, with the aim of gaining scientific knowledge by improved image quality, automated detection, segmentation, classification and tracking of objects, and efficient merging of information from multiple imaging modalities. We aim to give the reader an overview of the key developments and an understanding of possibilities and limitations of machine learning for microscopy. It will be of interest to a wide cross-disciplinary audience in the physical sciences and life sciences.

What problem does this paper attempt to address?

The main problems that this paper attempts to solve are to use deep - learning techniques to improve multiple challenges in microscope image processing, including but not limited to improving image quality, achieving automated detection, segmentation, classification and tracking of objects, and efficiently integrating information from multiple imaging modalities. Specifically: 1. **Data Scarcity and Domain Specificity**: Due to the very strong domain specificity of microscope data sets, machine - learning and deep - learning algorithms face significant challenges when applied across domains. For example, the data of electron microscopes (EM) and confocal fluorescence microscopes are very different, making it difficult for traditional image enhancement methods to be directly applied. In addition, the data formats produced by hardware provided by different vendors are not unified, further limiting the universality of data and the generalization ability of models. 2. **Stability of Inverse Problems**: Many microscope image - processing tasks can be classified as nonlinear inverse problems, which are inherently unstable. This means that even a tiny change in the input data may lead to a huge difference in the output results. The "Universal Instability Theorem" mentioned in the paper proves this characteristic, which places higher requirements on deep - learning - based solutions and requires the development of new strategies to resist this instability. 3. **Acquisition of High - Quality Labeled Data**: Although microscope image data are gradually becoming available, high - quality labeled data with a high degree of consensus are still difficult to obtain, which limits the application and development of supervised - learning methods. 4. **Reliability and Robustness of Algorithms**: For scientific and clinical applications, the reliability of microscopes is crucial. Therefore, how to ensure the stability and reliability of machine - learning algorithms under various conditions and avoid introducing indistinguishable artifacts has become an important research direction. To address these challenges, the paper explores methods that combine physical knowledge and deep learning, such as using differentiable physical models to optimize optical designs and reducing the dependence on large - scale labeled data through self - supervised learning methods. At the same time, the paper also emphasizes that the research on uncertainty quantification and robustness is of great significance for promoting the application of deep learning in the field of microscope image processing.

Roadmap on Deep Learning for Microscopy

A Survey on Applications of Deep Learning in Microscopy Image Analysis

Deep Learning Microscopy

The Rise of Data-Driven Microscopy powered by Machine Learning

Deep-Learning-Based Image Reconstruction and Enhancement in Optical Microscopy

The rise of data‐driven microscopy powered by machine learning

Deep Learning for Imaging and Detection of Microorganisms

Deep Learning for Automated Experimentation in Scanning Transmission Electron Microscopy

Deep Learning-Enabled Resolution-Enhancement in Mini- and Regular Microscopy for Biomedical Imaging

Advances in Deep Learning for Super-Resolution Microscopy(Invited)

Deep Learning Analysis on Microscopic Imaging in Materials Science

Deep learning-based image processing in optical microscopy

Rapid Image Deconvolution and Multiview Fusion for Optical Microscopy.

Advancing electron microscopy using deep learning

Machine Learning for Microscopy Data Analysis: Toward Real-time Optical and Electrical Characterization of Sub-micron Materials

Deep Learning Based Fluorescence Microscopy Imaging Technologies and Applications

Toward a Thinking Microscope: Deep Learning in Optical Microscopy and Image Reconstruction

Review: Deep Learning in Electron Microscopy

Machine learning in microscopy – insights, opportunities and challenges

Human-in-the-loop: The future of Machine Learning in Automated Electron Microscopy

AIMIC: Deep Learning for Microscopic Image Classification