Deep-learning Based Dual-View Light-Field Microscopy Enabling High-Resolution 3D Imaging of Dense Signals

Lanxin Zhu,Chengqiang Yi,Guo Li,Yuxuan Zhao,Peng Fei

DOI: https://doi.org/10.1364/boda.2021.dth2a.3

2021-01-01

Abstract:Poor performance on reconstructing dense signals is the major weakness of light-field microscopy. We report a deep-learning-based dual-view light-field microscopy which can reconstruct highly dense signals with high resolution.

What problem does this paper attempt to address?

Deep Learning Enhanced Fluorescence Emission Difference Microscopy

Yueshu Xu,Yuran Huang,Yuxuan Qiu,Zhimin Zhang,Yubing Han,Yudong Cui,Xiang Hao,Cuifang Kuang,Xu Liu

DOI: https://doi.org/10.1016/j.optlastec.2023.110009

2024-01-01

Abstract:Fluorescence emission difference microscopy (FED) obtains super-resolution microscopic images by extracting the information of the intensity differences between the solid and doughnut confocal images. Due to the mismatch of the outer contour of the solid and doughnut point spread function, FED suffers the problem of information loss. Here, we present a framework for deep-learning-enhanced FED (DL-FED) microscopy through cycle-consistent generative adversarial network (CycleGAN) based image reconstruction. Using this framework, we effectively avoid the information loss and enhance the spatial resolution of the fluorescence images acquired by the standard FED. We also demonstrate that the standard FED images can be transformed to match the results of Airyscan-based FED and saturated-FED, which can effectively enhance the signal-to-noise ratio and avoid the photo-bleaching effect. The validity of DL-FED is demonstrated by simulations and experiments based on fluorescent nanoparticles and biological cells. Featuring the potential to realize a high imaging speed, this approach may be widely applied in live cells investigations.
Deep-learning Light-Field Microscopy with Improved Resolution and Reconstruction Speed

Guo Li,Zhaoqiang Wang,Hao Zhang,Lanxin Zhu,Peng Fei

DOI: https://doi.org/10.1364/translational.2020.jw5a.1

2020-01-01

Abstract:We demonstrate a deep-learning based light-field microscopy method, which achieves greatly improved spatial resolution and video-rate reconstruction throughput, thereby allowing sustained volumetric recording of fast biological dynamics at high spatiotemporal resolution.
Extended Depth of Field Reconstruction with Complex Field Estimation

Jing Hu,Sunzheng Li,Yibing Shen,JinLei Zhang,Zhenrong Zheng

DOI: https://doi.org/10.1016/j.optlastec.2022.108118

IF: 4.939

2022-01-01

Optics & Laser Technology

Abstract:The contradiction between high resolution and large depth of field (DOF) of microscopic imaging limits its application in imaging 3D distributed samples. Here, we propose a new technique to achieve extended depth of field reconstruction in bright-field microscopy through complex field estimation. The complex field is estimated by a phase retrieval algorithm with the four illumination modulation patterns and the corresponding captured Fourier intensities as dual constraints. Then, the digital refocusing strategy based on Fresnel propagator is integrated into reconstruction framework to achieve the reconstruction of high-resolution complex-valued information for each part of the sample, which ensures an effective extension of the DOF. The feasibility of the proposed method to maintain high resolution over extended DOF was verified in actual experiments with a tilted USAF target, biological sample and microlens array (MLA). The extended DOF reconstruction provides a robust method for imaging in both industrial inspection and biomedical research.
DeepLFM: Deep Learning-based 3D Reconstruction for Light Field Microscopy

Xiaoxu Li,Hui Qiao,Jiamin Wu,Zhi Lu,Tao Yan,Ruxin Zhang,Xu Zhang,Qionghai Dai

DOI: https://doi.org/10.1364/ntm.2019.nm3c.2

2019-01-01

Abstract:We propose a high-resolution 3D reconstruction method for light field microscopy via deep learning. Experimental results on K562 cells verify its superior performance, which exhibit less artifacts especially near the native object plane.
Deep-learning-enhanced light-field microscopy

Kai Wang

DOI: https://doi.org/10.1038/s41592-021-01151-1

IF: 48

2021-05-01

Nature Methods

Abstract:Light-field microscopes can image three-dimensional dynamics of biological samples at unprecedented speed, but the computational reconstruction necessary for image formation is artifact-prone and time-consuming. Deep learning closes this gap between imaging and reconstruction speed.

biochemical research methods
Deep Learning Light Field Microscopy for Video-Rate Volumetric Functional Imaging of Behaving Animal

Zhaoqiang Wang,Hao Zhang,Yucheng Yang,Li Guo,Lanxin Zhu,Yang Li,Mingyang He,Tingting Zhu,Tzung K. Hsiai,Shangbang Gao,Fei Peng

2019-01-01

Abstract:We demonstrate a deep learning-enabled light field microscopy (LFM) to achieve 1 μm -resolution volumetric functional imaging of acting samples at high speed up to 100 Hz. For the raw light field measurement, the use of field detection optics usually limits the recoverable spatial resolution. We develop a deep learning approach to mitigate this limitation and effectively recover a time sequence of higher-resolution 3-D images based on the raw 2-D light field video. Unlike slow iterative LF reconstruction methods, our network approach permits direct inference at video-rate processing throughput (16 volumes per second), which is well suited for sustained volumetric recording of biological dynamics. To obtain optimal network recovery for reconstructing thick specimens, we further use a light-sheet illumination geometry to record high-contrast light field images with less signal overlapping. We demonstrate the success of our approach by intoto imaging the neuronal activities of behaving C. elegans and visualizing the cardiovascular hemodynamics inside zebrafish embryo.
Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning

Zhaoqiang Wang,Lanxin Zhu,Hao Zhang,Guo Li,Chengqiang Yi,Yi Li,Yicong Yang,Yichen Ding,Mei Zhen,Shangbang Gao,Tzung K. Hsiai,Peng Fei

DOI: https://doi.org/10.1038/s41592-021-01058-x

IF: 48

2021-02-11

Nature Methods

Abstract:Light-field microscopy has emerged as a technique of choice for high-speed volumetric imaging of fast biological processes. However, artifacts, nonuniform resolution and a slow reconstruction speed have limited its full capabilities for in toto extraction of dynamic spatiotemporal patterns in samples. Here, we combined a view-channel-depth (VCD) neural network with light-field microscopy to mitigate these limitations, yielding artifact-free three-dimensional image sequences with uniform spatial resolution and high-video-rate reconstruction throughput. We imaged neuronal activities across moving <i>Caenorhabditis elegans</i> and blood flow in a beating zebrafish heart at single-cell resolution with volumetric imaging rates up to 200 Hz.

biochemical research methods
Enhancing fluorescence microscopy performance with Richardson-Lucy based deep learning

Yue Li,Yijun Su,Huafeng Liu,Yicong Wu,Hari Shroff

DOI: https://doi.org/10.1364/3d.2022.jw5b.3

2022-01-01

Abstract:We present a deep learning method which incorporates the Richardson-Lucy formula into a fully convolutional network and demonstrate its performance on cells, tissues, and embryos imaged with widefield-, confocal- light sheet-, and super-resolution microscopy.
Light field angular super-resolution based on intrinsic and geometric information.

Lingyu Wang,Lifei Ren,Xiaoyao Wei,Jiangxin Yang,Yanlong Cao,Yanpeng Cao

DOI: https://doi.org/10.1016/j.knosys.2023.110553

2023-01-01

Abstract:Light field imaging can encode abundant scene information, including the intensity and direction of light rays, into 4D light field images. However, the limited number of sensors in commercial light field cameras leads to a trade-off between spatial and angular resolutions. In this paper, an angular super-resolution framework is proposed to synthesize new views and overcome hardware restrictions. First, light field intrinsic feature convolution is proposed to extract intrinsic information, i.e., scene content, complete view correlations, and epipolar structures. Consequently, spatial, angular, and cross-domain information can be preserved in the extracted features. Second, the spatial-angular and depth streams are proposed based on the light field intrinsic feature convolution to synthesize high angular resolution light fields. The spatial-angular stream utilizes the light field intrinsic information to improve the angular resolution, whereas the depth stream disentangles the geometric information from the extracted light field intrinsic features, which is used to warp the given sub-aperture images to the new view positions. Both streams can synthesize high-quality intermediate results, where the intrinsic and geometric information are utilized separately. Finally, a confidence-based stream fusion module is proposed to fuse the outputs from the two streams, achieving the joint employment of light field intrinsic and geometric information and solving the problem of insufficient information exploration in the current methods. We conduct a series of experiments to validate the effectiveness of each component in the framework and demonstrate that our method can achieve state-of-the-art performance in various scenes.(c) 2023 Elsevier B.V. All rights reserved.
Noise-robust phase-space deconvolution for light-field microscopy

Tianyi Zhu,Yuduo Guo,Yi Zhang,Zhi Lu,Xing Lin,Lu Fang,Jiamin Wu,Qionghai Dai

DOI: https://doi.org/10.1117/1.JBO.27.7.076501

IF: 3.758

2022-01-01

Journal of Biomedical Optics

Abstract:Significance: Light-field microscopy has achieved success in various applications of life sciences that require high-speed volumetric imaging. However, existing light-field reconstruction algorithms degrade severely in low-light conditions, and the deconvolution process is time-consuming. Aim: This study aims to develop a noise robustness phase-space deconvolution method with low computational costs. Approach: We reformulate the light-field phase-space deconvolution model into the Fourier domain with random-subset ordering and total-variation (TV) regularization. Additionally, we build a time-division-based multicolor light-field microscopy and conduct the three-dimensional (3D) imaging of the heart beating in zebrafish larva at over 95 Hz with a low light dose. Results: We demonstrate that this approach reduces computational resources, brings a tenfold speedup, and achieves a tenfold improvement for the noise robustness in terms of SSIM over the state-of-the-art approach. Conclusions: We proposed a phase-space deconvolution algorithm for 3D reconstructions in fluorescence imaging. Compared with the state-of-the-art method, we show significant improvement in both computational effectiveness and noise robustness; we further demonstrated practical application on zebrafish larva with low exposure and low light dose. (C) The Authors. Published by SPIE under a Creative Commons Attribution 4.0 International License.
Light Field Reconstruction with Arbitrary Angular Resolution Using a Deep Coarse-To-Fine Framework

Ran Li,Li Fang,Long Ye,Wei Zhong,Qin Zhang

DOI: https://doi.org/10.1007/978-981-16-1194-0_34

2021-01-01

Abstract:Densely-sampled light fields (LFs) are favorable for numerous applications like 3D scene reconstruction, virtual reality, et al., while the acquisition is costly. Most of the current view synthesis approaches need to sample the input LFs in a special or regular pattern, which makes the actual acquisition difficult. In this article, a new coarse-to-fine deep learning framework is presented to reconstruct densely-sampled LFs with arbitrary angular resolution. Concretely, a rough reconstruction based on meta-learning is performed on each epipolar plane image (EPI) to achieve arbitrary proportion of upsampling, followed by a refinement with 3D convolutional neural networks (CNNs) on stacked EPIs. Both modules are differentiable so that the network is end-to-end trainable. In addition, these two steps are performed on 3D volumes extracted from LF data first horizontally, and then vertically, forming a pseudo-4DCNN which can synthesize 4D LFs from a group of sparse input views effectively. The key advantage is to efficiently synthesize LFs with arbitrary angular resolution using a single model. The presented approach compares superiorly against various state-of-the-art methods on various challenging scenes.
Design of a light field microscope with uniform spatial resolution across an extended 3D volume

Qi Zheng,Xin Jin,Yanqin Chen

DOI: https://doi.org/10.1117/12.2601433

2021-01-01

Abstract:Light field microscopy (LFM) can capture the 4D light field for high-speed volumetric imaging with single-shot image by introducing a microlens array in traditional microscopy and combing with Richardson-Lucy deconvolution algorithm, which leads to wide usage in neuroimaging and dynamics observing, etc. However, the spatial resolution of the reconstructed results shows non-uniformity along the axial optical axis due to its varying spatial-angular sampling ratio along axial axis, which limits its application in observing thick samples. In this paper, we propose a new method including light field microscope scanning strategy and a multi-image deconvolution algorithm to address this problem. First, by designing several groups of different distances between the native object plane and microlens array, we can ensure appropriate spatial-angular sampling ratio among a wide range of depth. Secondly, according to the characteristics of camera noise in the process of fluorescence microscopy imaging, we get the loss function by calculating the K-L divergence of the Poisson noise likelihood, then provide an iterative reconstruction deconvolution algorithm. Applying the multi-image deconvolution algorithm to the captured light field images, we can finally get an extended 3D volume with uniform spatial resolution. We validate our approach by using both simulation and real experiments, which contain resolution measurements of a USAF 1951 resolution target and fluorescent beads. Under the same configuration, the proposed method can reduce the variance of the resolution across 200 mu m by 20%, compared with existing method, which demonstrates a more uniform resolution distribution.
Deep Learning Microscopy

Yair Rivenson,Zoltan Gorocs,Harun Gunaydin,Yibo Zhang,Hongda Wang,Aydogan Ozcan

DOI: https://doi.org/10.1364/OPTICA.4.001437

2017-05-13

Abstract:We demonstrate that a deep neural network can significantly improve optical microscopy, enhancing its spatial resolution over a large field-of-view and depth-of-field. After its training, the only input to this network is an image acquired using a regular optical microscope, without any changes to its design. We blindly tested this deep learning approach using various tissue samples that are imaged with low-resolution and wide-field systems, where the network rapidly outputs an image with remarkably better resolution, matching the performance of higher numerical aperture lenses, also significantly surpassing their limited field-of-view and depth-of-field. These results are transformative for various fields that use microscopy tools, including e.g., life sciences, where optical microscopy is considered as one of the most widely used and deployed techniques. Beyond such applications, our presented approach is broadly applicable to other imaging modalities, also spanning different parts of the electromagnetic spectrum, and can be used to design computational imagers that get better and better as they continue to image specimen and establish new transformations among different modes of imaging.

Machine Learning,Computer Vision and Pattern Recognition,Optics
Dense reconstruction for sparsely-sampled light field

Kun LI,Yali MAO,Jian MA,Jingyu YANG,Xinchen YE

DOI: https://doi.org/10.3969/j.issn.2095-2783.2018.20.006

2018-01-01

Abstract:Light field cameras usually sample sparsely in angular domain due to limited sensor resolution.To mitigate the problem, we proposed a method based on deep learning which can reconstruct 8×8 or even denser light fields from a 2×2 sparse light field.The method first uses CNN to estimate the depth information of the novel view from input views, and then obtain a synthesized image for the novel view by warping the input images using the estimated depth map.Finally, a high-quality synthesized image is generated by CNN with Resnet blocks.Experimental results show that our approach greatly improves the performance compared with the state-of-the-art methods, especially for the edges and the regions with rich textures.
Deep Learning Enables Large Depth-of-Field Images for Sub-Diffraction-Limit Scanning Superlens Microscopy

Hui Sun,Hao Luo,Feifei Wang,Qingjiu Chen,Meng Chen,Xiaoduo Wang,Haibo Yu,Guanglie Zhang,Lianqing Liu,Jianping Wang,Dapeng Wu,Wen Jung Li

2023-10-27

Abstract:Scanning electron microscopy (SEM) is indispensable in diverse applications ranging from microelectronics to food processing because it provides large depth-of-field images with a resolution beyond the optical diffraction limit. However, the technology requires coating conductive films on insulator samples and a vacuum environment. We use deep learning to obtain the mapping relationship between optical super-resolution (OSR) images and SEM domain images, which enables the transformation of OSR images into SEM-like large depth-of-field images. Our custom-built scanning superlens microscopy (SSUM) system, which requires neither coating samples by conductive films nor a vacuum environment, is used to acquire the OSR images with features down to ~80 nm. The peak signal-to-noise ratio (PSNR) and structural similarity index measure values indicate that the deep learning method performs excellently in image-to-image translation, with a PSNR improvement of about 0.74 dB over the optical super-resolution images. The proposed method provides a high level of detail in the reconstructed results, indicating that it has broad applicability to chip-level defect detection, biological sample analysis, forensics, and various other fields.

Optics,Artificial Intelligence,Image and Video Processing
Light Field Reconstruction With A Hybrid Sparse Regularization-Pseudo 4dcnn Framework

Li Fang,Wei Zhong,Long Ye,Ran Li,Qin Zhang

DOI: https://doi.org/10.1109/ACCESS.2020.3023505

IF: 3.9

2020-01-01

IEEE Access

Abstract:The densely-sampled light field (LF) is highly desirable in many applications, such as 3D reconstruction, post-capture refocusing and virtual/augmented reality. However, it is costly and challenging to capture them because of the high dimensional nature. Existing view synthesis methods employ depth information for the densely-sampled LF reconstruction from the undersampled LF with a large disparity range, but fail to achieve non-Lambertian performance. In this article, a novel coarse-to-fine LF reconstruction method is proposed. We will develop a hybrid reconstruction framework that fuses model-based sparse regularization with deep learning. Specifically, sparse regularization with directional filter bank is utilized to solve the large baseline problem and gives a coarse densely-sampled LF. In addition, for those that cannot be recovered by classical model-based methods due to limited angular resolution, is estimated by learning a pseudo 4D convolutional neural network, and thereby offering a refinement on the intermediate LF. Extensive experiments demonstrate the superiority of our method on both real-world and synthetic LF images when compared with state-of-the-art methods.
Field-dependent deep learning enables high-throughput whole-cell 3D super-resolution imaging

Shuang Fu,Wei Shi,Tingdan Luo,Yingchuan He,Lulu Zhou,Jie Yang,Zhichao Yang,Jiadong Liu,Xiaotian Liu,Zhiyong Guo,Chengyu Yang,Chao Liu,Zhen-li Huang,Jonas Ries,Mingjie Zhang,Peng Xi,Dayong Jin,Yiming Li

DOI: https://doi.org/10.1038/s41592-023-01775-5

IF: 48

2023-02-25

Nature Methods

Abstract:Single-molecule localization microscopy in a typical wide-field setup has been widely used for investigating subcellular structures with super resolution; however, field-dependent aberrations restrict the field of view (FOV) to only tens of micrometers. Here, we present a deep-learning method for precise localization of spatially variant point emitters (FD-DeepLoc) over a large FOV covering the full chip of a modern sCMOS camera. Using a graphic processing unit-based vectorial point spread function (PSF) fitter, we can fast and accurately model the spatially variant PSF of a high numerical aperture objective in the entire FOV. Combined with deformable mirror-based optimal PSF engineering, we demonstrate high-accuracy three-dimensional single-molecule localization microscopy over a volume of ~180 × 180 × 5 μm 3 , allowing us to image mitochondria and nuclear pore complexes in entire cells in a single imaging cycle without hardware scanning; a 100-fold increase in throughput compared to the state of the art.

biochemical research methods
Deep Learning Assisted Plasmonic Dark-Field Microscopy for Super-Resolution Label-Free Imaging

Ming Lei,Junxiang Zhao,Ayse Z. Sahan,Jie Hu,Junxiao Zhou,Hongki Lee,Qianyi Wu,Jin Zhang,Zhaowei Liu

DOI: https://doi.org/10.1021/acs.nanolett.4c04399

IF: 10.8

2024-11-28

Nano Letters

Abstract:Dark-field microscopy (DFM) is a widely used imaging tool, due to its high-contrast capability in imaging label-free specimens. Traditional DFM requires optical alignment to block the oblique illumination, and the resolution is diffraction-limited to the wavelength scale. In this work, we present deep-learning assisted plasmonic dark-field microscopy (DAPD), which is a single-frame super-resolution method using plasmonic dark-field (PDF) microscopy and deep-learning assisted image...

materials science, multidisciplinary,chemistry, physical,physics, applied, condensed matter,nanoscience & nanotechnology
High-resolution 3D light-field imaging

Qiang Geng,Zhiqiang Fu,Shih-Chi Chen

DOI: https://doi.org/10.1117/1.jbo.25.10.106502

IF: 3.758

2020-10-12

Journal of Biomedical Optics

Abstract:SIGNIFICANCE: High-speed 3D imaging methods have been playing crucial roles in many biological discoveries.AIM: We present a hybrid light-field imaging system and image processing algorithm that can visualize high-speed biological events.APPROACH: The hybrid light-field imaging system uses the selective plane optical illumination, which simultaneously records a high-resolution 2D image and a low-resolution 4D light-field image. The high-resolution 4D light-field image is obtained by applying the hybrid algorithm derived from the deconvolution and phase retrieval methods.RESULTS: High-resolution 3D imaging at a speed of 100-s volumes per second over an imaging field of 250 × 250 × 80 μm3 in the x, y, and z axis, respectively, is achieved with a 2.5 times enhancement in lateral resolution over the entire imaging field compared with standard light-field systems. In comparison to the deconvolution algorithm, the hybrid algorithm addresses the artifact issue at the focal plane and reduces the computation time by a factor of 4.CONCLUSIONS: The new hybrid light-field imaging method realizes high-resolution and ultrafast 3D imaging with a compact setup and simple algorithm, which may help discover important applications in biophotonics to visualize high-speed biological events.

radiology, nuclear medicine & medical imaging,biochemical research methods,optics
Video-rate 3D imaging of living cells using Fourier view-channel-depth light field microscopy

Chengqiang Yi,Lanxin Zhu,Jiahao Sun,Zhaofei Wang,Meng Zhang,Fenghe Zhong,Luxin Yan,Jiang Tang,Liang Huang,Yu-Hui Zhang,Dongyu Li,Peng Fei

DOI: https://doi.org/10.1038/s42003-023-05636-x

IF: 6.548

2023-12-13

Communications Biology

Abstract:Interrogation of subcellular biological dynamics occurring in a living cell often requires noninvasive imaging of the fragile cell with high spatiotemporal resolution across all three dimensions. It thereby poses big challenges to modern fluorescence microscopy implementations because the limited photon budget in a live-cell imaging task makes the achievable performance of conventional microscopy approaches compromise between their spatial resolution, volumetric imaging speed, and phototoxicity. Here, we incorporate a two-stage view-channel-depth (VCD) deep-learning reconstruction strategy with a Fourier light-field microscope based on diffractive optical element to realize fast 3D super-resolution reconstructions of intracellular dynamics from single diffraction-limited 2D light-filed measurements. This VCD-enabled Fourier light-filed imaging approach (F-VCD), achieves video-rate (50 volumes per second) 3D imaging of intracellular dynamics at a high spatiotemporal resolution of ~180 nm × 180 nm × 400 nm and strong noise-resistant capability, with which light field images with a signal-to-noise ratio (SNR) down to -1.62 dB could be well reconstructed. With this approach, we successfully demonstrate the 4D imaging of intracellular organelle dynamics, e.g., mitochondria fission and fusion, with ~5000 times of observation.

biology

Deep-learning Based Dual-View Light-Field Microscopy Enabling High-Resolution 3D Imaging of Dense Signals

Deep Learning Enhanced Fluorescence Emission Difference Microscopy

Deep-learning Light-Field Microscopy with Improved Resolution and Reconstruction Speed

Extended Depth of Field Reconstruction with Complex Field Estimation

DeepLFM: Deep Learning-based 3D Reconstruction for Light Field Microscopy

Deep-learning-enhanced light-field microscopy

Deep Learning Light Field Microscopy for Video-Rate Volumetric Functional Imaging of Behaving Animal

Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning

Enhancing fluorescence microscopy performance with Richardson-Lucy based deep learning

Light field angular super-resolution based on intrinsic and geometric information.

Noise-robust phase-space deconvolution for light-field microscopy

Light Field Reconstruction with Arbitrary Angular Resolution Using a Deep Coarse-To-Fine Framework

Design of a light field microscope with uniform spatial resolution across an extended 3D volume

Deep Learning Microscopy

Dense reconstruction for sparsely-sampled light field

Deep Learning Enables Large Depth-of-Field Images for Sub-Diffraction-Limit Scanning Superlens Microscopy

Light Field Reconstruction With A Hybrid Sparse Regularization-Pseudo 4dcnn Framework

Field-dependent deep learning enables high-throughput whole-cell 3D super-resolution imaging

Deep Learning Assisted Plasmonic Dark-Field Microscopy for Super-Resolution Label-Free Imaging

High-resolution 3D light-field imaging

Video-rate 3D imaging of living cells using Fourier view-channel-depth light field microscopy