Deep-learning-enhanced light-field microscopy

Kai Wang

DOI: https://doi.org/10.1038/s41592-021-01151-1

IF: 48

2021-05-01

Nature Methods

Abstract:Light-field microscopes can image three-dimensional dynamics of biological samples at unprecedented speed, but the computational reconstruction necessary for image formation is artifact-prone and time-consuming. Deep learning closes this gap between imaging and reconstruction speed.

biochemical research methods

What problem does this paper attempt to address?

Deep-learning Light-Field Microscopy with Improved Resolution and Reconstruction Speed

Guo Li,Zhaoqiang Wang,Hao Zhang,Lanxin Zhu,Peng Fei

DOI: https://doi.org/10.1364/translational.2020.jw5a.1

2020-01-01

Abstract:We demonstrate a deep-learning based light-field microscopy method, which achieves greatly improved spatial resolution and video-rate reconstruction throughput, thereby allowing sustained volumetric recording of fast biological dynamics at high spatiotemporal resolution.
Single-shot Autofocusing in Light Sheet Fluorescence Microscopy with Multiplexed Structured Illumination and Deep Learning

Yanhong Gan,Zitong Ye,Yubing Han,Ye Ma,Chuankang Li,Qiulan Liu,Wenjie Liu,Cuifang Kuang,Xu Liu

DOI: https://doi.org/10.1016/j.optlaseng.2023.107663

IF: 5.666

2023-01-01

Optics and Lasers in Engineering

Abstract:Light sheet fluorescence microscopy (LSFM) has been proposed as a promising tool for biological research due to its ability to observe the dynamics of living cells for hours and days. Nonetheless, this raises the strict requirement for the overlapping between the light sheet and the detection focal plane to get optimal image quality. Here, we describe a fast and accurate deep learning-based autofocusing method to overcome the challenge of unstable focusing in LSFM with a single shot. This method is compatible with any light sheet imaging setup with a spatial light modulator for light sheet generation. A predicted root-mean-square error of 0.0942 & mu;m across a range of & PLUSMN; 0.7 & mu;m in a light sheet microscope using a 1.1 numerical aperture detection objective can be obtained. The neural network architecture we proposed shows advantages in small memory size, few training data set requirements, and good generalization to untrained sample types.
DeepLFM: Deep Learning-based 3D Reconstruction for Light Field Microscopy

Xiaoxu Li,Hui Qiao,Jiamin Wu,Zhi Lu,Tao Yan,Ruxin Zhang,Xu Zhang,Qionghai Dai

DOI: https://doi.org/10.1364/ntm.2019.nm3c.2

2019-01-01

Abstract:We propose a high-resolution 3D reconstruction method for light field microscopy via deep learning. Experimental results on K562 cells verify its superior performance, which exhibit less artifacts especially near the native object plane.
Deep Learning Enhanced Electrochemiluminescence Microscopy.

Pinlong Zhao,Wenxin Zhu,Min Zheng,Jiandong Feng

DOI: https://doi.org/10.1021/acs.analchem.3c00274

IF: 7.4

2023-01-01

Analytical Chemistry

Abstract:Limited by the efficiency of electrochemiluminescence, tens of seconds of exposure time are typically required to get a high-quality image. Image enhancement of short exposure time images to obtain a well-defined electrochemiluminescence image can meet the needs of high-throughput or dynamic imaging. Here, we propose deep enhanced ECL microscopy (DEECL), a general strategy that utilizes artificial neural networks to reconstruct electrochemiluminescence images with millisecond exposure times to have similar quality as high-quality electrochemiluminescence images with second-long exposure time. Electrochemiluminescence imaging of fixed cells demonstrates that DEECL allows improvement of the imaging efficiency by 1 to 2 orders than usual. This approach is further used for a data-intensive analysis application, cell classification, achieving an accuracy of 85% with ECL data at an exposure time of 50 ms. We anticipate that the computationally enhanced electrochemiluminescence microscopy will enable fast and information-rich imaging and prove useful for understanding dynamic chemical and biological processes.
Deep-learning Based Dual-View Light-Field Microscopy Enabling High-Resolution 3D Imaging of Dense Signals

Lanxin Zhu,Chengqiang Yi,Guo Li,Yuxuan Zhao,Peng Fei

DOI: https://doi.org/10.1364/boda.2021.dth2a.3

2021-01-01

Abstract:Poor performance on reconstructing dense signals is the major weakness of light-field microscopy. We report a deep-learning-based dual-view light-field microscopy which can reconstruct highly dense signals with high resolution.
Deep Learning Light Field Microscopy for Video-Rate Volumetric Functional Imaging of Behaving Animal

Zhaoqiang Wang,Hao Zhang,Yucheng Yang,Li Guo,Lanxin Zhu,Yang Li,Mingyang He,Tingting Zhu,Tzung K. Hsiai,Shangbang Gao,Fei Peng

2019-01-01

Abstract:We demonstrate a deep learning-enabled light field microscopy (LFM) to achieve 1 μm -resolution volumetric functional imaging of acting samples at high speed up to 100 Hz. For the raw light field measurement, the use of field detection optics usually limits the recoverable spatial resolution. We develop a deep learning approach to mitigate this limitation and effectively recover a time sequence of higher-resolution 3-D images based on the raw 2-D light field video. Unlike slow iterative LF reconstruction methods, our network approach permits direct inference at video-rate processing throughput (16 volumes per second), which is well suited for sustained volumetric recording of biological dynamics. To obtain optimal network recovery for reconstructing thick specimens, we further use a light-sheet illumination geometry to record high-contrast light field images with less signal overlapping. We demonstrate the success of our approach by intoto imaging the neuronal activities of behaving C. elegans and visualizing the cardiovascular hemodynamics inside zebrafish embryo.
Enhancing fluorescence microscopy performance with Richardson-Lucy based deep learning

Yue Li,Yijun Su,Huafeng Liu,Yicong Wu,Hari Shroff

DOI: https://doi.org/10.1364/3d.2022.jw5b.3

2022-01-01

Abstract:We present a deep learning method which incorporates the Richardson-Lucy formula into a fully convolutional network and demonstrate its performance on cells, tissues, and embryos imaged with widefield-, confocal- light sheet-, and super-resolution microscopy.
Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning

Zhaoqiang Wang,Lanxin Zhu,Hao Zhang,Guo Li,Chengqiang Yi,Yi Li,Yicong Yang,Yichen Ding,Mei Zhen,Shangbang Gao,Tzung K. Hsiai,Peng Fei

DOI: https://doi.org/10.1038/s41592-021-01058-x

IF: 48

2021-02-11

Nature Methods

Abstract:Light-field microscopy has emerged as a technique of choice for high-speed volumetric imaging of fast biological processes. However, artifacts, nonuniform resolution and a slow reconstruction speed have limited its full capabilities for in toto extraction of dynamic spatiotemporal patterns in samples. Here, we combined a view-channel-depth (VCD) neural network with light-field microscopy to mitigate these limitations, yielding artifact-free three-dimensional image sequences with uniform spatial resolution and high-video-rate reconstruction throughput. We imaged neuronal activities across moving <i>Caenorhabditis elegans</i> and blood flow in a beating zebrafish heart at single-cell resolution with volumetric imaging rates up to 200 Hz.

biochemical research methods
Deep Learning Microscopy

Yair Rivenson,Zoltan Gorocs,Harun Gunaydin,Yibo Zhang,Hongda Wang,Aydogan Ozcan

DOI: https://doi.org/10.1364/OPTICA.4.001437

2017-05-13

Abstract:We demonstrate that a deep neural network can significantly improve optical microscopy, enhancing its spatial resolution over a large field-of-view and depth-of-field. After its training, the only input to this network is an image acquired using a regular optical microscope, without any changes to its design. We blindly tested this deep learning approach using various tissue samples that are imaged with low-resolution and wide-field systems, where the network rapidly outputs an image with remarkably better resolution, matching the performance of higher numerical aperture lenses, also significantly surpassing their limited field-of-view and depth-of-field. These results are transformative for various fields that use microscopy tools, including e.g., life sciences, where optical microscopy is considered as one of the most widely used and deployed techniques. Beyond such applications, our presented approach is broadly applicable to other imaging modalities, also spanning different parts of the electromagnetic spectrum, and can be used to design computational imagers that get better and better as they continue to image specimen and establish new transformations among different modes of imaging.

Machine Learning,Computer Vision and Pattern Recognition,Optics
Deep-Learning-Based Image Reconstruction and Enhancement in Optical Microscopy

Kevin de Haan,Y. Rivenson,Yichen Wu,A. Ozcan

DOI: https://doi.org/10.1109/JPROC.2019.2949575

IF: 20.6

2020-01-01

Proceedings of the IEEE

Abstract:In recent years, deep learning has been shown to be one of the leading machine learning techniques for a wide variety of inference tasks. In addition to its mainstream applications, such as classification, it has created transformative opportunities for image reconstruction and enhancement in optical microscopy. Some of these emerging applications of deep learning range from image transformations between microscopic imaging systems to adding new capabilities to existing imaging techniques, as well as solving various inverse problems based on microscopy image data. Deep learning is helping us move toward data-driven instrument designs that blend microscopy and computing to achieve what neither can do alone. This article provides an overview of some of the recent work using deep neural networks to advance computational microscopy and sensing systems, also covering their current and future biomedical applications.

Physics,Engineering,Computer Science
Efficient high-resolution fluorescence projection imaging over an extended depth of field through optical hardware and deep learning optimizations

Xin Luo,Zhi Lu,Manchang Jin,Shuai Chen,Jingyu Yang

DOI: https://doi.org/10.1364/BOE.523312

2024-05-20

Abstract:Optical microscopy has witnessed notable advancements but has also become more costly and complex. Conventional wide field microscopy (WFM) has low resolution and shallow depth-of-field (DOF), which limits its applications in practical biological experiments. Recently, confocal and light sheet microscopy become major workhorses for biology that incorporate high-precision scanning to perform imaging within an extended DOF but at the sacrifice of expense, complexity, and imaging speed. Here, we propose deep focus microscopy, an efficient framework optimized both in hardware and algorithm to address the tradeoff between resolution and DOF. Our deep focus microscopy achieves large-DOF and high-resolution projection imaging by integrating a deep focus network (DFnet) into light field microscopy (LFM) setups. Based on our constructed dataset, deep focus microscopy features a significantly enhanced spatial resolution of ∼260 nm, an extended DOF of over 30 µm, and broad generalization across diverse sample structures. It also reduces the computational costs by four orders of magnitude compared to conventional LFM technologies. We demonstrate the excellent performance of deep focus microscopy in vivo, including long-term observations of cell division and migrasome formation in zebrafish embryos and mouse livers at high resolution without background contamination.
Light Field Imaging: Models, Calibrations, Reconstructions, and Applications

Hao Zhu,Qing Wang,Jingyi Yu

DOI: https://doi.org/10.1631/fitee.1601727

IF: 2.526

2017-01-01

Frontiers of Information Technology & Electronic Engineering

Abstract:Light field imaging is an emerging technology in computational photography areas. Based on innovative designs of the imaging model and the optical path, light field cameras not only record the spatial intensity of threedimensional (3D) objects, but also capture the angular information of the physical world, which provides new ways to address various problems in computer vision, such as 3D reconstruction, saliency detection, and object recognition. In this paper, three key aspects of light field cameras, i.e., model, calibration, and reconstruction, are reviewed extensively. Furthermore, light field based applications on informatics, physics, medicine, and biology are exhibited. Finally, open issues in light field imaging and long-term application prospects in other natural sciences are discussed.
Deep Learning Based Fluorescence Microscopy Imaging Technologies and Applications

Li Haoyu,Qu Liying,Hua Zijie,Wang Xinwei,Zhao Weisong,Liu Jian

DOI: https://doi.org/10.3788/lop202158.1811007

2021-01-01

Laser & Optoelectronics Progress

Abstract:In recent years, fluorescence microscopy has been commonly applied in various fields of scientific research, such as biophysics, neuroscience, cell biology, and molecular biology, owing to its specificity, high contrast, and high signal-to-noise ratio. However, traditional fluorescence microscopes have limitations regarding spatial resolution, imaging speed, field of view, phototoxicity, and photobleaching; these limitations compromise their applications in subcellular observation, in vivo imaging, and molecular structure profiling. To moderate such limitations, researchers have adopted data-driven deep learning methods, which can enrich the existing fluorescence microscopy technologies and boost the performance boundary of traditional fluorescence microscopy. This article focuses on the technologies and applications of deep learning based fluorescence microscopy. First, we briefly summarize the basic principle and development path of deep learning technologies; then, we introduce the latest domestic and global progress of deep learning based fluorescence microscopy. Compared with the traditional microscopic imaging system, we show the superiority of deep learning in solving fluorescence microscopy problems. Finally, the future potential of developing deep learning based microscopy is highlighted.
High-resolution 3D light-field imaging

Qiang Geng,Zhiqiang Fu,Shih-Chi Chen

DOI: https://doi.org/10.1117/1.jbo.25.10.106502

IF: 3.758

2020-10-12

Journal of Biomedical Optics

Abstract:SIGNIFICANCE: High-speed 3D imaging methods have been playing crucial roles in many biological discoveries.AIM: We present a hybrid light-field imaging system and image processing algorithm that can visualize high-speed biological events.APPROACH: The hybrid light-field imaging system uses the selective plane optical illumination, which simultaneously records a high-resolution 2D image and a low-resolution 4D light-field image. The high-resolution 4D light-field image is obtained by applying the hybrid algorithm derived from the deconvolution and phase retrieval methods.RESULTS: High-resolution 3D imaging at a speed of 100-s volumes per second over an imaging field of 250 × 250 × 80 μm3 in the x, y, and z axis, respectively, is achieved with a 2.5 times enhancement in lateral resolution over the entire imaging field compared with standard light-field systems. In comparison to the deconvolution algorithm, the hybrid algorithm addresses the artifact issue at the focal plane and reduces the computation time by a factor of 4.CONCLUSIONS: The new hybrid light-field imaging method realizes high-resolution and ultrafast 3D imaging with a compact setup and simple algorithm, which may help discover important applications in biophotonics to visualize high-speed biological events.

radiology, nuclear medicine & medical imaging,biochemical research methods,optics
Design of a light field microscope with uniform spatial resolution across an extended 3D volume

Qi Zheng,Xin Jin,Yanqin Chen

DOI: https://doi.org/10.1117/12.2601433

2021-01-01

Abstract:Light field microscopy (LFM) can capture the 4D light field for high-speed volumetric imaging with single-shot image by introducing a microlens array in traditional microscopy and combing with Richardson-Lucy deconvolution algorithm, which leads to wide usage in neuroimaging and dynamics observing, etc. However, the spatial resolution of the reconstructed results shows non-uniformity along the axial optical axis due to its varying spatial-angular sampling ratio along axial axis, which limits its application in observing thick samples. In this paper, we propose a new method including light field microscope scanning strategy and a multi-image deconvolution algorithm to address this problem. First, by designing several groups of different distances between the native object plane and microlens array, we can ensure appropriate spatial-angular sampling ratio among a wide range of depth. Secondly, according to the characteristics of camera noise in the process of fluorescence microscopy imaging, we get the loss function by calculating the K-L divergence of the Poisson noise likelihood, then provide an iterative reconstruction deconvolution algorithm. Applying the multi-image deconvolution algorithm to the captured light field images, we can finally get an extended 3D volume with uniform spatial resolution. We validate our approach by using both simulation and real experiments, which contain resolution measurements of a USAF 1951 resolution target and fluorescent beads. Under the same configuration, the proposed method can reduce the variance of the resolution across 200 mu m by 20%, compared with existing method, which demonstrates a more uniform resolution distribution.
Noise-robust phase-space deconvolution for light-field microscopy

Tianyi Zhu,Yuduo Guo,Yi Zhang,Zhi Lu,Xing Lin,Lu Fang,Jiamin Wu,Qionghai Dai

DOI: https://doi.org/10.1117/1.JBO.27.7.076501

IF: 3.758

2022-01-01

Journal of Biomedical Optics

Abstract:Significance: Light-field microscopy has achieved success in various applications of life sciences that require high-speed volumetric imaging. However, existing light-field reconstruction algorithms degrade severely in low-light conditions, and the deconvolution process is time-consuming. Aim: This study aims to develop a noise robustness phase-space deconvolution method with low computational costs. Approach: We reformulate the light-field phase-space deconvolution model into the Fourier domain with random-subset ordering and total-variation (TV) regularization. Additionally, we build a time-division-based multicolor light-field microscopy and conduct the three-dimensional (3D) imaging of the heart beating in zebrafish larva at over 95 Hz with a low light dose. Results: We demonstrate that this approach reduces computational resources, brings a tenfold speedup, and achieves a tenfold improvement for the noise robustness in terms of SSIM over the state-of-the-art approach. Conclusions: We proposed a phase-space deconvolution algorithm for 3D reconstructions in fluorescence imaging. Compared with the state-of-the-art method, we show significant improvement in both computational effectiveness and noise robustness; we further demonstrated practical application on zebrafish larva with low exposure and low light dose. (C) The Authors. Published by SPIE under a Creative Commons Attribution 4.0 International License.
Deep-3D Microscope: 3D volumetric microscopy of thick scattering samples using a wide-field microscope and machine learning

Bowen Li,Shiyu Tan,Jiuyang Dong,Xiaocong Lian,Yongbing Zhang,Xiangyang Ji,Ashok Veeraraghavan

DOI: https://doi.org/10.48550/arXiv.2110.07218

2021-10-14

Abstract:Confocal microscopy is the standard approach for obtaining volumetric images of a sample with high axial and lateral resolution, especially when dealing with scattering samples. Unfortunately, a confocal microscope is quite expensive compared to traditional microscopes. In addition, the point scanning in a confocal leads to slow imaging speed and photobleaching due to the high dose of laser energy. In this paper, we demonstrate how the advances in machine learning can be exploited to "teach" a traditional wide-field microscope, one that's available in every lab, into producing 3D volumetric images like a confocal. The key idea is to obtain multiple images with different focus settings using a wide-field microscope and use a 3D Generative Adversarial Network (GAN) based neural network to learn the mapping between the blurry low-contrast image stack obtained using wide-field and the sharp, high-contrast images obtained using a confocal. After training the network with widefield-confocal image pairs, the network can reliably and accurately reconstruct 3D volumetric images that rival confocal in terms of its lateral resolution, z-sectioning and image contrast. Our experimental results demonstrate generalization ability to handle unseen data, stability in the reconstruction results, high spatial resolution even when imaging thick ($\sim40$ microns) highly-scattering samples. We believe that such learning-based-microscopes have the potential to bring confocal quality imaging to every lab that has a wide-field microscope.

Optics,Biological Physics
High-Quality Light Field Microscope Imaging Based on Microlens Arrays

Tongkai Gu,Sitong Yan,Lanlan Wang,Yasheng Chang,Hongzhong Liu

DOI: https://doi.org/10.1109/jmems.2023.3349299

IF: 2.829

2024-01-01

Journal of Microelectromechanical Systems

Abstract:High-quality optical observation through traditional microscopes faces significant challenges due to their low spatial sampling and the limited ability to respond only to the light intensity characteristics of optoelectronic devices. This limitation results in an inability to measure other critical optical information during imaging, such as phase, angle, polarization, and coherence. In response to these challenges, light field microscope (LFM) as a powerful imaging technique is capable of measuring samples with unprecedented depth and detail. LFM overcomes the limitations of conventional microscope methods by capturing both spatial and angular information of light rays. To further demonstrate these capabilities, the LFM based on microlens arrays is constructed here. These arrays are fabricated using advanced techniques such as laser lithography, microimprinting, and self-assembly technology. Using light field imaging, image segmentation methods, and deep learning fusion, the imaging quality is nearly doubled, significantly enhancing the quality of observations. LFM based on microlens arrays offers great promise for improving the quality of imaging observations in the field of microsope. [2023-0167]

engineering, electrical & electronic,nanoscience & nanotechnology,instruments & instrumentation,physics, applied
Deep learning extended depth-of-field microscope for fast and slide-free histology

Lingbo Jin,Yubo Tang,Yicheng Wu,Jackson B Coole,Melody T Tan,Xuan Zhao,Hawraa Badaoui,Jacob T Robinson,Michelle D Williams,Ann M Gillenwater,Rebecca R Richards-Kortum,Ashok Veeraraghavan

DOI: https://doi.org/10.1073/pnas.2013571117

2020-12-29

Abstract:Microscopic evaluation of resected tissue plays a central role in the surgical management of cancer. Because optical microscopes have a limited depth-of-field (DOF), resected tissue is either frozen or preserved with chemical fixatives, sliced into thin sections placed on microscope slides, stained, and imaged to determine whether surgical margins are free of tumor cells-a costly and time- and labor-intensive procedure. Here, we introduce a deep-learning extended DOF (DeepDOF) microscope to quickly image large areas of freshly resected tissue to provide histologic-quality images of surgical margins without physical sectioning. The DeepDOF microscope consists of a conventional fluorescence microscope with the simple addition of an inexpensive (less than $10) phase mask inserted in the pupil plane to encode the light field and enhance the depth-invariance of the point-spread function. When used with a jointly optimized image-reconstruction algorithm, diffraction-limited optical performance to resolve subcellular features can be maintained while significantly extending the DOF (200 µm). Data from resected oral surgical specimens show that the DeepDOF microscope can consistently visualize nuclear morphology and other important diagnostic features across highly irregular resected tissue surfaces without serial refocusing. With the capability to quickly scan intact samples with subcellular detail, the DeepDOF microscope can improve tissue sampling during intraoperative tumor-margin assessment, while offering an affordable tool to provide histological information from resected tissue specimens in resource-limited settings.
Field-dependent deep learning enables high-throughput whole-cell 3D super-resolution imaging

Shuang Fu,Wei Shi,Tingdan Luo,Yingchuan He,Lulu Zhou,Jie Yang,Zhichao Yang,Jiadong Liu,Xiaotian Liu,Zhiyong Guo,Chengyu Yang,Chao Liu,Zhen-li Huang,Jonas Ries,Mingjie Zhang,Peng Xi,Dayong Jin,Yiming Li

DOI: https://doi.org/10.1038/s41592-023-01775-5

IF: 48

2023-02-25

Nature Methods

Abstract:Single-molecule localization microscopy in a typical wide-field setup has been widely used for investigating subcellular structures with super resolution; however, field-dependent aberrations restrict the field of view (FOV) to only tens of micrometers. Here, we present a deep-learning method for precise localization of spatially variant point emitters (FD-DeepLoc) over a large FOV covering the full chip of a modern sCMOS camera. Using a graphic processing unit-based vectorial point spread function (PSF) fitter, we can fast and accurately model the spatially variant PSF of a high numerical aperture objective in the entire FOV. Combined with deformable mirror-based optimal PSF engineering, we demonstrate high-accuracy three-dimensional single-molecule localization microscopy over a volume of ~180 × 180 × 5 μm 3 , allowing us to image mitochondria and nuclear pore complexes in entire cells in a single imaging cycle without hardware scanning; a 100-fold increase in throughput compared to the state of the art.

biochemical research methods

Deep-learning-enhanced light-field microscopy

Deep-learning Light-Field Microscopy with Improved Resolution and Reconstruction Speed

Single-shot Autofocusing in Light Sheet Fluorescence Microscopy with Multiplexed Structured Illumination and Deep Learning

DeepLFM: Deep Learning-based 3D Reconstruction for Light Field Microscopy

Deep Learning Enhanced Electrochemiluminescence Microscopy.

Deep-learning Based Dual-View Light-Field Microscopy Enabling High-Resolution 3D Imaging of Dense Signals

Deep Learning Light Field Microscopy for Video-Rate Volumetric Functional Imaging of Behaving Animal

Enhancing fluorescence microscopy performance with Richardson-Lucy based deep learning

Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning

Deep Learning Microscopy

Deep-Learning-Based Image Reconstruction and Enhancement in Optical Microscopy

Efficient high-resolution fluorescence projection imaging over an extended depth of field through optical hardware and deep learning optimizations

Light Field Imaging: Models, Calibrations, Reconstructions, and Applications

Deep Learning Based Fluorescence Microscopy Imaging Technologies and Applications

High-resolution 3D light-field imaging

Design of a light field microscope with uniform spatial resolution across an extended 3D volume

Noise-robust phase-space deconvolution for light-field microscopy

Deep-3D Microscope: 3D volumetric microscopy of thick scattering samples using a wide-field microscope and machine learning

High-Quality Light Field Microscope Imaging Based on Microlens Arrays

Deep learning extended depth-of-field microscope for fast and slide-free histology

Field-dependent deep learning enables high-throughput whole-cell 3D super-resolution imaging