DeepLFM: Deep Learning-based 3D Reconstruction for Light Field Microscopy

Xiaoxu Li,Hui Qiao,Jiamin Wu,Zhi Lu,Tao Yan,Ruxin Zhang,Xu Zhang,Qionghai Dai

DOI: https://doi.org/10.1364/ntm.2019.nm3c.2

2019-01-01

Abstract:We propose a high-resolution 3D reconstruction method for light field microscopy via deep learning. Experimental results on K562 cells verify its superior performance, which exhibit less artifacts especially near the native object plane.

What problem does this paper attempt to address?

Three Dimensional Surface Reconstruction Based on Light Field Imaging

Sun Shiyi,Kuang Cuifang,Liu Xu

DOI: https://doi.org/10.5768/jao201738.0202001

2017-01-01

Journal of Applied Optics

Abstract:Traditional imaging methods can not capture axial information through one shot,which lead to information deficiency and incapability of three dimentional surface reconstruction.Unlike traditional camera,lytro's first generation pocket-sized camera using light field sensor can capture multi-dimensional light field information through one single exposure,and at the same time it has advantages such as portable.By using light field camera,with digital refocusing method,defocusing and correspondence cues,acquirement of high quality depth map can be realized.By using matlab platform to process acquired depth map matrix,reconstruction of three dimensional surface can be realized.Normalized depth data of object space is obtained.In this experiment,three dimensional surface reconstruction is realized with average error of 5.47％ in depth range of 100mm to 1 500 mm,and maximum reconstruction error is 8.30&.
Deep Learning Enhanced Fluorescence Emission Difference Microscopy

Yueshu Xu,Yuran Huang,Yuxuan Qiu,Zhimin Zhang,Yubing Han,Yudong Cui,Xiang Hao,Cuifang Kuang,Xu Liu

DOI: https://doi.org/10.1016/j.optlastec.2023.110009

2024-01-01

Abstract:Fluorescence emission difference microscopy (FED) obtains super-resolution microscopic images by extracting the information of the intensity differences between the solid and doughnut confocal images. Due to the mismatch of the outer contour of the solid and doughnut point spread function, FED suffers the problem of information loss. Here, we present a framework for deep-learning-enhanced FED (DL-FED) microscopy through cycle-consistent generative adversarial network (CycleGAN) based image reconstruction. Using this framework, we effectively avoid the information loss and enhance the spatial resolution of the fluorescence images acquired by the standard FED. We also demonstrate that the standard FED images can be transformed to match the results of Airyscan-based FED and saturated-FED, which can effectively enhance the signal-to-noise ratio and avoid the photo-bleaching effect. The validity of DL-FED is demonstrated by simulations and experiments based on fluorescent nanoparticles and biological cells. Featuring the potential to realize a high imaging speed, this approach may be widely applied in live cells investigations.
Deep-learning Light-Field Microscopy with Improved Resolution and Reconstruction Speed

Guo Li,Zhaoqiang Wang,Hao Zhang,Lanxin Zhu,Peng Fei

DOI: https://doi.org/10.1364/translational.2020.jw5a.1

2020-01-01

Abstract:We demonstrate a deep-learning based light-field microscopy method, which achieves greatly improved spatial resolution and video-rate reconstruction throughput, thereby allowing sustained volumetric recording of fast biological dynamics at high spatiotemporal resolution.
Three Dimensional Reconstruction Using a Lenslet Light Field Camera

Wentao Zhang,Shengqian Chang,Xiao Tao,Chang Wang,Chenning Tao,Peizheng Huang,Zhenrong Zheng

DOI: https://doi.org/10.1117/12.2319552

2018-01-01

Abstract:A novel method is proposed in this paper to accurately reconstruct the three-dimensional scenes by using a passive single-shot exposure with a lenslet light field camera. This method has better performance of 3D scenes reconstruction with both defocus and disparity depth cues captured by light field camera. First, the light field data is used to refocus and shift viewpoints to get a focal stack and multi-view images. In refocusing procedure, the phase shift theorem in the Fourier domain is first introduced to substitute shift in spatial domain, and sharper focal stacks can be obtained with less blurriness. Thus, 3D scenes can be reconstructed more accurately. Next, through multi-view images, disparity depth cues are obtained by performing correspondence measure. Then, the focal stack is used to compute defocus depth cues by focus measure based on gray variance. Finally, the focus cost is built to integrate both defocus and disparity depth cues, and the accurate depth map is estimated by using Graph Cuts based on the focus cost. Using this accurate depth map and all-in-focus image, the 3D structure in real world are accurately reconstructed. Our method is verified by a number of synthetic and real-world examples captured with a dense camera array and a Lytro light field camera.
Deep-learning-enhanced light-field microscopy

Kai Wang

DOI: https://doi.org/10.1038/s41592-021-01151-1

IF: 48

2021-05-01

Nature Methods

Abstract:Light-field microscopes can image three-dimensional dynamics of biological samples at unprecedented speed, but the computational reconstruction necessary for image formation is artifact-prone and time-consuming. Deep learning closes this gap between imaging and reconstruction speed.

biochemical research methods
Deep-learning Based Dual-View Light-Field Microscopy Enabling High-Resolution 3D Imaging of Dense Signals

Lanxin Zhu,Chengqiang Yi,Guo Li,Yuxuan Zhao,Peng Fei

DOI: https://doi.org/10.1364/boda.2021.dth2a.3

2021-01-01

Abstract:Poor performance on reconstructing dense signals is the major weakness of light-field microscopy. We report a deep-learning-based dual-view light-field microscopy which can reconstruct highly dense signals with high resolution.
Light Field Reconstruction with Arbitrary Angular Resolution Using a Deep Coarse-To-Fine Framework

Ran Li,Li Fang,Long Ye,Wei Zhong,Qin Zhang

DOI: https://doi.org/10.1007/978-981-16-1194-0_34

2021-01-01

Abstract:Densely-sampled light fields (LFs) are favorable for numerous applications like 3D scene reconstruction, virtual reality, et al., while the acquisition is costly. Most of the current view synthesis approaches need to sample the input LFs in a special or regular pattern, which makes the actual acquisition difficult. In this article, a new coarse-to-fine deep learning framework is presented to reconstruct densely-sampled LFs with arbitrary angular resolution. Concretely, a rough reconstruction based on meta-learning is performed on each epipolar plane image (EPI) to achieve arbitrary proportion of upsampling, followed by a refinement with 3D convolutional neural networks (CNNs) on stacked EPIs. Both modules are differentiable so that the network is end-to-end trainable. In addition, these two steps are performed on 3D volumes extracted from LF data first horizontally, and then vertically, forming a pseudo-4DCNN which can synthesize 4D LFs from a group of sparse input views effectively. The key advantage is to efficiently synthesize LFs with arbitrary angular resolution using a single model. The presented approach compares superiorly against various state-of-the-art methods on various challenging scenes.
Enhancing fluorescence microscopy performance with Richardson-Lucy based deep learning

Yue Li,Yijun Su,Huafeng Liu,Yicong Wu,Hari Shroff

DOI: https://doi.org/10.1364/3d.2022.jw5b.3

2022-01-01

Abstract:We present a deep learning method which incorporates the Richardson-Lucy formula into a fully convolutional network and demonstrate its performance on cells, tissues, and embryos imaged with widefield-, confocal- light sheet-, and super-resolution microscopy.
Light Field Reconstruction With A Hybrid Sparse Regularization-Pseudo 4dcnn Framework

Li Fang,Wei Zhong,Long Ye,Ran Li,Qin Zhang

DOI: https://doi.org/10.1109/ACCESS.2020.3023505

IF: 3.9

2020-01-01

IEEE Access

Abstract:The densely-sampled light field (LF) is highly desirable in many applications, such as 3D reconstruction, post-capture refocusing and virtual/augmented reality. However, it is costly and challenging to capture them because of the high dimensional nature. Existing view synthesis methods employ depth information for the densely-sampled LF reconstruction from the undersampled LF with a large disparity range, but fail to achieve non-Lambertian performance. In this article, a novel coarse-to-fine LF reconstruction method is proposed. We will develop a hybrid reconstruction framework that fuses model-based sparse regularization with deep learning. Specifically, sparse regularization with directional filter bank is utilized to solve the large baseline problem and gives a coarse densely-sampled LF. In addition, for those that cannot be recovered by classical model-based methods due to limited angular resolution, is estimated by learning a pseudo 4D convolutional neural network, and thereby offering a refinement on the intermediate LF. Extensive experiments demonstrate the superiority of our method on both real-world and synthetic LF images when compared with state-of-the-art methods.
DeepLeMiN: Deep-learning-empowered Physics-aware Lensless Miniscope

Feng Tian,Ben Mattison,Weijian Yang

DOI: https://doi.org/10.1101/2024.05.03.592471

2024-05-06

Abstract:Mask-based lensless fluorescence microscopy is a compact, portable imaging technique promising for biomedical research. It forms images through a thin optical mask near the camera without bulky optics, enabling snapshot three-dimensional imaging and a scalable field of view (FOV) without increasing device thickness. Lensless microscopy relies on computational algorithms to solve the inverse problem of object reconstruction. However, there has been a lack of efficient reconstruction algorithms for large-scale data. Furthermore, the entire FOV is typically reconstructed as a whole, which demands substantial computational resources and limits the scalability of the FOV. Here, we developed DeepLeMiN, a lensless microscope with a custom designed optical mask and a multi-stage physics-informed deep learning model. This not only enables the reconstruction of localized FOVs, but also significantly reduces the computational resource demands and facilitates real-time reconstruction. Our deep learning algorithm can reconstruct object volumes over 4×6×0.6 mm , achieving lateral and axial resolution of ∼10 µm and ∼50 µm respectively. We demonstrated significant improvement in both reconstruction quality and speed compared to traditional methods, across various fluorescent samples with dense structures. Notably, we achieved high-quality reconstruction of 3D motion of hydra and the neuronal activity with cellular resolution in awake mouse cortex. DeepLeMiN holds great promise for scalable, large FOV, real-time, 3D imaging applications with compact device footprint.

Neuroscience
Design of a light field microscope with uniform spatial resolution across an extended 3D volume

Qi Zheng,Xin Jin,Yanqin Chen

DOI: https://doi.org/10.1117/12.2601433

2021-01-01

Abstract:Light field microscopy (LFM) can capture the 4D light field for high-speed volumetric imaging with single-shot image by introducing a microlens array in traditional microscopy and combing with Richardson-Lucy deconvolution algorithm, which leads to wide usage in neuroimaging and dynamics observing, etc. However, the spatial resolution of the reconstructed results shows non-uniformity along the axial optical axis due to its varying spatial-angular sampling ratio along axial axis, which limits its application in observing thick samples. In this paper, we propose a new method including light field microscope scanning strategy and a multi-image deconvolution algorithm to address this problem. First, by designing several groups of different distances between the native object plane and microlens array, we can ensure appropriate spatial-angular sampling ratio among a wide range of depth. Secondly, according to the characteristics of camera noise in the process of fluorescence microscopy imaging, we get the loss function by calculating the K-L divergence of the Poisson noise likelihood, then provide an iterative reconstruction deconvolution algorithm. Applying the multi-image deconvolution algorithm to the captured light field images, we can finally get an extended 3D volume with uniform spatial resolution. We validate our approach by using both simulation and real experiments, which contain resolution measurements of a USAF 1951 resolution target and fluorescent beads. Under the same configuration, the proposed method can reduce the variance of the resolution across 200 mu m by 20%, compared with existing method, which demonstrates a more uniform resolution distribution.
DiLFM: an artifact-suppressed and noise-robust light-field microscopy through dictionary learning

Yuanlong Zhang,Bo Xiong,Yi Zhang,Zhi Lu,Jiamin Wu,Qionghai Dai

DOI: https://doi.org/10.1038/s41377-021-00587-6

2021-01-01

Abstract:Light field microscopy (LFM) has been widely used for recording 3D biological dynamics at camera frame rate. However, LFM suffers from artifact contaminations due to the illness of the reconstruction problem via naïve Richardson–Lucy (RL) deconvolution. Moreover, the performance of LFM significantly dropped in low-light conditions due to the absence of sample priors. In this paper, we thoroughly analyze different kinds of artifacts and present a new LFM technique termed dictionary LFM (DiLFM) that substantially suppresses various kinds of reconstruction artifacts and improves the noise robustness with an over-complete dictionary. We demonstrate artifact-suppressed reconstructions in scattering samples such as Drosophila embryos and brains. Furthermore, we show our DiLFM can achieve robust blood cell counting in noisy conditions by imaging blood cell dynamic at 100 Hz and unveil more neurons in whole-brain calcium recording of zebrafish with low illumination power in vivo.
Deep learning-based scattering removal of light field imaging

Weihao Wang,Xing Zhao,Zhixiang Jiang,Ya Wen

DOI: https://doi.org/10.3788/COL202220.041101

IF: 2.56

2022-01-01

Chinese Optics Letters

Abstract:Light field imaging has shown significance in research fields for its high-temporal-resolution 3D imaging ability. However, in scenes of light field imaging through scattering, such as biological imaging in vivo and imaging in fog, the quality of 3D reconstruction will be severely reduced due to the scattering of the light field information. In this paper, we propose a deep learning-based method of scattering removal of light field imaging. In this method, a neural network, trained by simulation samples that are generated by light field imaging forward models with and without scattering, is utilized to remove the effect of scattering on light fields captured experimentally. With the deblurred light field and the scattering-free forward model, 3D reconstruction with high resolution and high contrast can be realized. We demonstrate the proposed method by using it to realize high-quality 3D reconstruction through a single scattering layer experimentally.
Deep Learning Light Field Microscopy for Video-Rate Volumetric Functional Imaging of Behaving Animal

Zhaoqiang Wang,Hao Zhang,Yucheng Yang,Li Guo,Lanxin Zhu,Yang Li,Mingyang He,Tingting Zhu,Tzung K. Hsiai,Shangbang Gao,Fei Peng

2019-01-01

Abstract:We demonstrate a deep learning-enabled light field microscopy (LFM) to achieve 1 μm -resolution volumetric functional imaging of acting samples at high speed up to 100 Hz. For the raw light field measurement, the use of field detection optics usually limits the recoverable spatial resolution. We develop a deep learning approach to mitigate this limitation and effectively recover a time sequence of higher-resolution 3-D images based on the raw 2-D light field video. Unlike slow iterative LF reconstruction methods, our network approach permits direct inference at video-rate processing throughput (16 volumes per second), which is well suited for sustained volumetric recording of biological dynamics. To obtain optimal network recovery for reconstructing thick specimens, we further use a light-sheet illumination geometry to record high-contrast light field images with less signal overlapping. We demonstrate the success of our approach by intoto imaging the neuronal activities of behaving C. elegans and visualizing the cardiovascular hemodynamics inside zebrafish embryo.
LF-fusion: Dense and Accurate 3D Reconstruction from Light Field Images

Jiayong Peng,Zhiwei Xiong,Yueyi Zhang,Dong Liu,Feng Wu

DOI: https://doi.org/10.1109/vcip.2017.8305046

2017-01-01

Abstract:Light field (LF) cameras offer the capability of depth estimation in a single shot, which facilitates real-time 3D reconstruction of dynamic scenes. However, the accuracy of depth estimated from LF is still limited. Different from previous methods that generally focus on improving the fidelity of the central view depth, we argue that depth maps obtained at different views contain complementary information. Inspired by the principle of Kinect-fusion, we then propose a novel method for dense and accurate 3D reconstruction from LF images, namely, LF-fusion. Specifically, we use the iterative closest point (ICP) algorithm to register the point clouds generated from different views, and then employ a volumetric integration algorithm based on the truncated signed distance function (TSDF) to reconstruct the final 3D surface. Experiments demonstrate that the proposed method produces superior 3D reconstruction results on two representative LF datasets.
Deep Light Field Super-Resolution Using Frequency Domain Analysis and Semantic Prior

Yeyao Chen,Gangyi Jiang,Zhidi Jiang,Mei Yu,Yo-Sung Ho

DOI: https://doi.org/10.1109/tmm.2021.3106775

IF: 7.3

2021-01-01

IEEE Transactions on Multimedia

Abstract:Light field (LF) camera can simultaneously capture the intensity and direction information of light rays, which has been widely concerned. However, limited by the size of the imaging sensor, the captured LF image (LFI) has a trade-off between spatial and angular resolutions. To this end, this paper proposes a new LF super-resolution method using frequency domain analysis and semantic prior, which designs a two-stage learning framework to enhance the spatial and angular resolutions of LFI. Specifically, the proposed method first decomposes the spatial and angular information to explore the 4D structure of LFI by using frequency domain transformation, and formulates the LF super-resolution as a frequency restoration process. Then, the decomposed frequency components are recovered in a progressive restoration manner, with new cascaded 2D and 3D convolutional neural networks. To further improve the quality of the reconstructed LFI, especially at the object boundary, the semantic prior is incorporated into the designed network to enhance its representation ability. Finally, the super-resolved LFI is reconstructed by inverse frequency domain transformation. Experimental results show that the proposed method can effectively generate high-resolution LFI, and outperforms other state-of-the-art methods in terms of both subjective visual perception and objective quality evaluation. Moreover, the proposed method can enhance the performance of LF applications such as depth estimation.

computer science, information systems,telecommunications, software engineering
The Light Field 3D Scanner.

Yingliang Zhang,Zhong Li,Wei Yang,Peihong Yu,Haiting Lin,Jingyi Yu

DOI: https://doi.org/10.1109/iccphot.2017.7951484

2017-01-01

Abstract:We present a novel light field structure-from-motion (SfM) framework for reliable 3D object reconstruction. Specifically, we use the light field (LF) camera such as Lytro and Raytrix as a virtual 3D scanner. We move an LF camera around the object and register between multiple LF shots. We show that applying conventional SfM on sub-aperture images is not only expensive but also unreliable due to ultra-small baseline and low image resolution. Instead, our LF-SfM scheme maps ray manifolds across LFs. Specifically, we show how rays passing through a common 3D point transform between two LFs and we develop reliable technique for extracting extrinsic parameters from this ray transform. Next, we apply a new edge-preserving stereo matching technique on individual LFs and conduct LF bundle adjustment to jointly optimize pose and geometry. Comprehensive experiments show our solution outperforms many state-of-the-art passive and even active techniques especially on topologically complex objects.
Light Field Reconstruction via Deep Adaptive Fusion of Hybrid Lenses

Jing Jin,Mantang Guo,Junhui Hou,Hui Liu,Hongkai Xiong

2023-06-18

Abstract:This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses, including a high-resolution camera surrounded by multiple low-resolution cameras. The performance of existing methods is still limited, as they produce either blurry results on plain textured areas or distortions around depth discontinuous boundaries. To tackle this challenge, we propose a novel end-to-end learning-based approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation, while the other module warps another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations adaptively via the learned attention maps, leading to the final high-resolution LF image with satisfactory results on both plain textured areas and depth discontinuous boundaries. Besides, to promote the effectiveness of our method trained with simulated hybrid data on real hybrid data captured by a hybrid LF imaging system, we carefully design the network architecture and the training strategy. Extensive experiments on both real and simulated hybrid data demonstrate the significant superiority of our approach over state-of-the-art ones. To the best of our knowledge, this is the first end-to-end deep learning method for LF reconstruction from a real hybrid input. We believe our framework could potentially decrease the cost of high-resolution LF data acquisition and benefit LF data storage and transmission.

Image and Video Processing,Computer Vision and Pattern Recognition
Deep Coarse-to-Fine Dense Light Field Reconstruction With Flexible Sampling and Geometry-Aware Fusion

Jing Jin,Junhui Hou,Jie Chen,Huanqiang Zeng,Sam Kwong,Jingyi Yu

DOI: https://doi.org/10.1109/TPAMI.2020.3026039

Abstract:A densely-sampled light field (LF) is highly desirable in various applications, such as 3-D reconstruction, post-capture refocusing and virtual reality. However, it is costly to acquire such data. Although many computational methods have been proposed to reconstruct a densely-sampled LF from a sparsely-sampled one, they still suffer from either low reconstruction quality, low computational efficiency, or the restriction on the regularity of the sampling pattern. To this end, we propose a novel learning-based method, which accepts sparsely-sampled LFs with irregular structures, and produces densely-sampled LFs with arbitrary angular resolution accurately and efficiently. We also propose a simple yet effective method for optimizing the sampling pattern. Our proposed method, an end-to-end trainable network, reconstructs a densely-sampled LF in a coarse-to-fine manner. Specifically, the coarse sub-aperture image (SAI) synthesis module first explores the scene geometry from an unstructured sparsely-sampled LF and leverages it to independently synthesize novel SAIs, in which a confidence-based blending strategy is proposed to fuse the information from different input SAIs, giving an intermediate densely-sampled LF. Then, the efficient LF refinement module learns the angular relationship within the intermediate result to recover the LF parallax structure. Comprehensive experimental evaluations demonstrate the superiority of our method on both real-world and synthetic LF images when compared with state-of-the-art methods. In addition, we illustrate the benefits and advantages of the proposed approach when applied in various LF-based applications, including image-based rendering and depth estimation enhancement. The code is available at https://github.com/jingjin25/LFASR-FS-GAF.
Light Field Reconstruction Via Attention-Guided Deep Fusion of Hybrid Lenses

Jing Jin,Hui Liu,Junhui Hou,Hongkai Xiong

2021-01-01

Abstract:This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses, including a high-resolution camera surrounded by multiple lowresolution cameras. The performance of existing methods is still limited, as they produce either blurry results on plain textured areas or distortions around depth discontinuous boundaries. To tackle this challenge, we propose a novel end-to-end learningbased approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation, while the other module warps another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations adaptively via the learned attention maps, leading to the final high-resolution LF image with satisfactory results on both plain textured areas and depth discontinuous boundaries. Besides, to promote the effectiveness of our method trained with simulated hybrid data on real hybrid data captured by a hybrid LF imaging system, we carefully design the network architecture and the training strategy. Extensive experiments on both real and simulated hybrid data demonstrate the significant superiority of our approach over state-of-the-art ones. To the best of our knowledge, this is the first end-to-end deep learning method for LF reconstruction from a real hybrid input. We believe our framework could potentially decrease the cost of high-resolution LF data acquisition and benefit LF data storage and transmission. The code will be publicly available at https://github.com/jingjin25/LFhybridSR-Fusion.

DeepLFM: Deep Learning-based 3D Reconstruction for Light Field Microscopy

Three Dimensional Surface Reconstruction Based on Light Field Imaging

Deep Learning Enhanced Fluorescence Emission Difference Microscopy

Deep-learning Light-Field Microscopy with Improved Resolution and Reconstruction Speed

Three Dimensional Reconstruction Using a Lenslet Light Field Camera

Deep-learning-enhanced light-field microscopy

Deep-learning Based Dual-View Light-Field Microscopy Enabling High-Resolution 3D Imaging of Dense Signals

Light Field Reconstruction with Arbitrary Angular Resolution Using a Deep Coarse-To-Fine Framework

Enhancing fluorescence microscopy performance with Richardson-Lucy based deep learning

Light Field Reconstruction With A Hybrid Sparse Regularization-Pseudo 4dcnn Framework

DeepLeMiN: Deep-learning-empowered Physics-aware Lensless Miniscope

Design of a light field microscope with uniform spatial resolution across an extended 3D volume

DiLFM: an artifact-suppressed and noise-robust light-field microscopy through dictionary learning

Deep learning-based scattering removal of light field imaging

Deep Learning Light Field Microscopy for Video-Rate Volumetric Functional Imaging of Behaving Animal

LF-fusion: Dense and Accurate 3D Reconstruction from Light Field Images

Deep Light Field Super-Resolution Using Frequency Domain Analysis and Semantic Prior

The Light Field 3D Scanner.

Light Field Reconstruction via Deep Adaptive Fusion of Hybrid Lenses

Deep Coarse-to-Fine Dense Light Field Reconstruction With Flexible Sampling and Geometry-Aware Fusion

Light Field Reconstruction Via Attention-Guided Deep Fusion of Hybrid Lenses