Dynamic light field reconstruction via densely connected deep equilibrium model

Ruixue Wang,Xue Wang,Zhaolin Xiao,Qing Wang

DOI: https://doi.org/10.1364/oe.542159

IF: 3.8

2024-12-12

Optics Express

Abstract:Ruixue Wang, Xue Wang, Zhaolin Xiao, Qing Wang High-resolution consumer plenoptic cameras usually feature low frame rates, making them not well-suited for capturing high-speed motion ... [Opt. Express 32, 46829-46848 (2024)]

optics

What problem does this paper attempt to address?

Dynamic Light Field Reconstruction Based on Gradient Descent Deep Equilibrium Model(Invited)

Ruixue Wang,Xue Wang,Guoqing Zhou,Zhaolin Xiao,Qing Wang

DOI: https://doi.org/10.3788/lop241400

2024-01-01

Laser & Optoelectronics Progress

Abstract:In response to snapshot compressive imaging (SCI) in dynamic light field temporal domain, a method called dynamic light field deep equilibrium (DLFDEQ) was proposed to reconstruct high- quality dynamic light field image frames (5D data) from known encoding patterns and acquired snapshot compressed measurements (4D data). First, based on compressive sensing, the same compression encoding for each viewpoint of dynamic light field image frames in the temporal domain was adopted. Second, the reconstruction process of compressive measurements was modeled as an inverse problem with an implicit regularization term. Finally, the inverse problem was solved through a deep equilibrium model based on gradient descent (DEQ-GD). The DEQ-GD model allows for the stable reconstruction of the required dynamic light field image frames from snapshot compressive measurements. Experimental results demonstrate that proposed method can recover a 5x5 viewpoints dynamic light field composed of 4 frames of images from a single snapshot light field measurement of a 5x5 viewpoints. Compared with the current state-of-the-art- of- the- art methods, proposed method demonstrates stronger robustness and preserves more accurate details in the reconstructed dynamic light field image frames. By repeatedly capturing and recovering these compressive measurement values, the temporal frame rate of reconstructed image is 4 times of the original camera frame rate.
Three Dimensional Reconstruction Using a Lenslet Light Field Camera

Wentao Zhang,Shengqian Chang,Xiao Tao,Chang Wang,Chenning Tao,Peizheng Huang,Zhenrong Zheng

DOI: https://doi.org/10.1117/12.2319552

2018-01-01

Abstract:A novel method is proposed in this paper to accurately reconstruct the three-dimensional scenes by using a passive single-shot exposure with a lenslet light field camera. This method has better performance of 3D scenes reconstruction with both defocus and disparity depth cues captured by light field camera. First, the light field data is used to refocus and shift viewpoints to get a focal stack and multi-view images. In refocusing procedure, the phase shift theorem in the Fourier domain is first introduced to substitute shift in spatial domain, and sharper focal stacks can be obtained with less blurriness. Thus, 3D scenes can be reconstructed more accurately. Next, through multi-view images, disparity depth cues are obtained by performing correspondence measure. Then, the focal stack is used to compute defocus depth cues by focus measure based on gray variance. Finally, the focus cost is built to integrate both defocus and disparity depth cues, and the accurate depth map is estimated by using Graph Cuts based on the focus cost. Using this accurate depth map and all-in-focus image, the 3D structure in real world are accurately reconstructed. Our method is verified by a number of synthetic and real-world examples captured with a dense camera array and a Lytro light field camera.
Light Field Imaging: Models, Calibrations, Reconstructions, and Applications

Hao Zhu,Qing Wang,Jingyi Yu

DOI: https://doi.org/10.1631/fitee.1601727

IF: 2.526

2017-01-01

Frontiers of Information Technology & Electronic Engineering

Abstract:Light field imaging is an emerging technology in computational photography areas. Based on innovative designs of the imaging model and the optical path, light field cameras not only record the spatial intensity of threedimensional (3D) objects, but also capture the angular information of the physical world, which provides new ways to address various problems in computer vision, such as 3D reconstruction, saliency detection, and object recognition. In this paper, three key aspects of light field cameras, i.e., model, calibration, and reconstruction, are reviewed extensively. Furthermore, light field based applications on informatics, physics, medicine, and biology are exhibited. Finally, open issues in light field imaging and long-term application prospects in other natural sciences are discussed.
Unsupervised Dense Light Field Reconstruction with Occlusion Awareness

Lixia Ni,Haiyong Jiang,Jianfei Cai,Jianmin Zheng,Haifeng Li,Xu Liu

DOI: https://doi.org/10.1111/cgf.13849

IF: 2.5

2019-01-01

Computer Graphics Forum

Abstract:Light field (LF) reconstruction is a fundamental technique in light field imaging and has applications in both software and hardware aspects. This paper presents an unsupervised learning method for LF-oriented view synthesis, which provides a simple solution for generating quality light fields from a sparse set of views. The method is built on disparity estimation and image warping. Specifically, we first use per-view disparity as a geometry proxy to warp input views to novel views. Then we compensate the occlusion with a network by a forward-backward warping process. Cycle-consistency between different views are explored to enable unsupervised learning and accurate synthesis. The method overcomes the drawbacks of fully supervised learning methods that require large labeled training dataset and epipolar plane image based interpolation methods that do not make full use of geometry consistency in LFs. Experimental results demonstrate that the proposed method can generate high quality views for LF, which outperforms unsupervised approaches and is comparable to fully-supervised approaches.
An Efficient Anti-Occlusion Depth Estimation Using Generalized EPI Representation in Light Field

Hao Zhu,Qing Wang

DOI: https://doi.org/10.1117/12.2246086

2016-01-01

Abstract:Light field cameras have been rapidly developed and are likely to appear in mobile devices in near future. It is essential to develop efficient and robust depth estimation algorithm for mobile applications. However, existing methods are either slow or lack of adaptability to occlusion such that they are not suitable to mobile computing platform. In this paper, we present the generalized EPI representation in light field and formulate it using two linear functions. By combining it with the light field occlusion theory, a highly efficient and anti-occlusion depth estimation algorithm is proposed. Our algorithm outperforms the previous local method, especially in occlusion areas. Experimental results on public light field datasets have demonstrated the effectiveness and efficiency of the proposed algorithm.
3D Scene Reconstruction with an Un-calibrated Light Field Camera

Qi Zhang,Hongdong Li,Xue Wang,Qing Wang

DOI: https://doi.org/10.1007/s11263-021-01516-1

IF: 13.369

2021-01-01

International Journal of Computer Vision

Abstract:This paper is concerned with the problem of multi-view 3D reconstruction with an un-calibrated micro-lens array based light field camera. To acquire 3D Euclidean reconstruction, existing approaches commonly apply the calibration with a checkerboard and motion estimation from static scenes in two steps. Self-calibration is the process of simultaneously estimating intrinsic and extrinsic parameters directly from un-calibrated light fields without the help of a checkerboard. While the self-calibration technique for conventional (pinhole) camera is well understood, how to extend it to light field camera remains a challenging task. This is primarily due to the ultra-small baseline of the light field camera. We propose an effective self-calibration method for a light field camera for automatic metric reconstruction without a laborious pre-calibration process. In contrast to conventional self-calibration, we show how such a self-calibration method can be made numerically stable, by exploiting the regularity and measurement redundancies unique for the light field camera. The proposed method is built upon the derivation of a novel ray-space homography constraint (RSHC) using Plücker parameterization as well as a ray-space infinity homography (RSIH). We also propose a new concept of “rays of the absolute conic (RAC)” defined as a special quadric in 5D projective space ℙ^5 . A set of new equations are established and solved for self-calibration and 3D metric reconstruction specifically designed for a light field camera . We validate the efficacy of the proposed method on both synthetic and real light fields, and have obtained superior results in both accuracy and robustness.
Anti-Occlusion Light-Field Optical Flow Estimation Using Light-Field Super-Pixels

Hao Zhu,Xiaoming Sun,Qi Zhang,Qing Wang,Antonio Robles-Kelly,Hongdong Li

DOI: https://doi.org/10.1007/978-3-030-21074-8_1

2018-01-01

Abstract:Optical flow estimation is one of the most important problem in community. However, current methods still can not provide reliable results in occlusion boundary areas. Light field cameras provide hundred of views in a single shot, so the ambiguity can be better analysed using other views. In this paper, we present a novel method for anti-occlusion optical flow estimation in a dynamic light field. We first model the light field superpixel (LFSP) as a slanted plane in 3D. Then the motion of the occluded pixels in central view slice can be optimized by the un-occluded pixels in other views. Thus the optical flow in occlusion boundary areas can be well computed. Experimental results on both synthetic and real light fields demonstrate the advantages over state-of-the-arts and the performance on 4D optical flow computation.
Epipolar Focus Spectrum: A Novel Light Field Representation and Application in Dense-view Reconstruction

Yaning Li,Xue Wang,Hao Zhu,Guoqing Zhou,Qing Wang

DOI: https://doi.org/10.48550/arxiv.2204.00193

2022-01-01

Abstract:Existing light field representations, such as epipolar plane image (EPI) and sub-aperture images, do not consider the structural characteristics across the views, so they usually require additional disparity and spatial structure cues for follow-up tasks. Besides, they have difficulties dealing with occlusions or larger disparity scenes. To this end, this paper proposes a novel Epipolar Focus Spectrum (EFS) representation by rearranging the EPI spectrum. Different from the classical EPI representation where an EPI line corresponds to a specific depth, there is a one-to-one mapping from the EFS line to the view. Accordingly, compared to a sparsely-sampled light field, a densely-sampled one with the same field of view (FoV) leads to a more compact distribution of such linear structures in the double-cone-shaped region with the identical opening angle in its corresponding EFS. Hence the EFS representation is invariant to the scene depth. To demonstrate its effectiveness, we develop a trainable EFS-based pipeline for light field reconstruction, where a dense light field can be reconstructed by compensating the"missing EFS lines"given a sparse light field, yielding promising results with cross-view consistency, especially in the presence of severe occlusion and large disparity. Experimental results on both synthetic and real-world datasets demonstrate the validity and superiority of the proposed method over SOTA methods.
Sheared Epipolar Focus Spectrum for Dense Light Field Reconstruction.

Yaning Li,Xue Wang,Guoqing Zhou,Hao Zhu,Qing Wang

DOI: https://doi.org/10.1109/tpami.2023.3337516

IF: 23.6

2024-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Existing light field (LF) representations, such as epipolar plane image (EPI) and sub-aperture images, do not consider the structural characteristics across the views, so they usually require additional disparity and spatial structure cues for follow-up tasks. Besides, they have difficulties dealing with occlusions or large disparity scenes. To this end, this paper proposes a novel Epipolar Focus Spectrum (EFS) representation by rearranging the EPI spectrum. Different from the classical EPI representation where an EPI line corresponds to a specific depth, there is a one-to-one mapping from the EFS line to the view. By exploring the EFS sampling task, the analytical function is derived for constructing a non-aliasing EFS. To demonstrate its effectiveness, we develop a trainable EFS-based pipeline for light field reconstruction, where a dense light field can be reconstructed by compensating the missing EFS lines given a sparse light field, yielding promising results with cross-view consistency, especially in the presence of severe occlusion and large disparity. Experimental results on both synthetic and real-world datasets demonstrate the validity and superiority of the proposed method over SOTA methods.
Deep-learning Based Dual-View Light-Field Microscopy Enabling High-Resolution 3D Imaging of Dense Signals

Lanxin Zhu,Chengqiang Yi,Guo Li,Yuxuan Zhao,Peng Fei

DOI: https://doi.org/10.1364/boda.2021.dth2a.3

2021-01-01

Abstract:Poor performance on reconstructing dense signals is the major weakness of light-field microscopy. We report a deep-learning-based dual-view light-field microscopy which can reconstruct highly dense signals with high resolution.
Light-field-depth-estimation Network Based on Epipolar Geometry and Image Segmentation.

Xucheng Wang,Chenning Tao,Rengmao Wu,Xiao Tao,Peng Sun,Yong Li,Zhenrong Zheng

DOI: https://doi.org/10.1364/josaa.388555

2020-01-01

Abstract:In this paper, we propose a convolutional neural network based on epipolar geometry and image segmentation for light-field depth estimation. Epipolar geometry is utilized to estimate the initial disparity map. Multi-orientation epipolar images are selected as input data, and the convolutional blocks are adopted based on the disparity of different-direction epipolar images. Image segmentation is used to obtain the edge information of the central sub-aperture image. By concatenating the output of the two parts, an accurate depth map could be generated with fast speed. Our method achieves a high rank on most quality assessment metrics in the HCI 4D Light Field Benchmark and also shows effectiveness in estimating accurate depth on real-world light-field images.
Dense reconstruction for sparsely-sampled light field

Kun LI,Yali MAO,Jian MA,Jingyu YANG,Xinchen YE

DOI: https://doi.org/10.3969/j.issn.2095-2783.2018.20.006

2018-01-01

Abstract:Light field cameras usually sample sparsely in angular domain due to limited sensor resolution.To mitigate the problem, we proposed a method based on deep learning which can reconstruct 8×8 or even denser light fields from a 2×2 sparse light field.The method first uses CNN to estimate the depth information of the novel view from input views, and then obtain a synthesized image for the novel view by warping the input images using the estimated depth map.Finally, a high-quality synthesized image is generated by CNN with Resnet blocks.Experimental results show that our approach greatly improves the performance compared with the state-of-the-art methods, especially for the edges and the regions with rich textures.
72‐2: A High Resolution Near Eye Light Field Display with Fast Reconstruction Speed

Mali Liu,Chihao Lu,Haifeng Li,Xu Liu

DOI: https://doi.org/10.1002/sdtp.11820

2017-01-01

SID Symposium Digest of Technical Papers

Abstract:We proposed an acceleration of reconstructing a high resolution near eye light field display based on human visual features. The reconstructed light field supports high resolution, low‐latency and focus cues. The resolution distribution of the light field is identical to human visual features that visual acuity is different over field of visual.
Light field angular super-resolution based on intrinsic and geometric information.

Lingyu Wang,Lifei Ren,Xiaoyao Wei,Jiangxin Yang,Yanlong Cao,Yanpeng Cao

DOI: https://doi.org/10.1016/j.knosys.2023.110553

2023-01-01

Abstract:Light field imaging can encode abundant scene information, including the intensity and direction of light rays, into 4D light field images. However, the limited number of sensors in commercial light field cameras leads to a trade-off between spatial and angular resolutions. In this paper, an angular super-resolution framework is proposed to synthesize new views and overcome hardware restrictions. First, light field intrinsic feature convolution is proposed to extract intrinsic information, i.e., scene content, complete view correlations, and epipolar structures. Consequently, spatial, angular, and cross-domain information can be preserved in the extracted features. Second, the spatial-angular and depth streams are proposed based on the light field intrinsic feature convolution to synthesize high angular resolution light fields. The spatial-angular stream utilizes the light field intrinsic information to improve the angular resolution, whereas the depth stream disentangles the geometric information from the extracted light field intrinsic features, which is used to warp the given sub-aperture images to the new view positions. Both streams can synthesize high-quality intermediate results, where the intrinsic and geometric information are utilized separately. Finally, a confidence-based stream fusion module is proposed to fuse the outputs from the two streams, achieving the joint employment of light field intrinsic and geometric information and solving the problem of insufficient information exploration in the current methods. We conduct a series of experiments to validate the effectiveness of each component in the framework and demonstrate that our method can achieve state-of-the-art performance in various scenes.(c) 2023 Elsevier B.V. All rights reserved.
High-resolution Light Field Reconstruction Using a Hybrid Imaging System

Xiang Wang,Lin Li,Guangqi Hou

DOI: https://doi.org/10.1364/ao.55.002580

IF: 1.9

2016-01-01

Applied Optics

Abstract:Recently, light field cameras have drawn much attraction for their innovative performance in photographic and scientific applications. However, narrow baselines and constrained spatial resolution of current light field cameras impose restrictions on their usability. Therefore, we design a hybrid imaging system containing a light field camera and a high-resolution digital single lens reflex camera, and these two kinds of cameras share the same optical path with a beam splitter so as to achieve the reconstruction of high-resolution light fields. The high-resolution 4D light fields are reconstructed with a phase-based perspective variation strategy. First, we apply complex steerable pyramid decomposition on the high-resolution image from the digital single lens reflex camera. Then, we perform phase-based perspective-shift processing with the disparity value, which is extracted from the upsampled light field depth map, to create high-resolution synthetic light field images. High-resolution digital refocused images and high-resolution depth maps can be generated in this way. Furthermore, controlling the magnitude of the perspective shift enables us to change the depth of field rendering in the digital refocused images. We show several experimental results to demonstrate the effectiveness of our approach.
Light Field Reconstruction Using Deep Convolutional Network On Epi

Gaochang Wu,Mandan Zhao,Liangyong Wang,Qionghai Dai,Tianyou Chai,Yebin Liu

DOI: https://doi.org/10.1109/CVPR.2017.178

2017-01-01

Abstract:In this paper, we take advantage of the clear texture structure of the epipolar plane image (EPI) in the light field data and model the problem of light field reconstruction from a sparse set of views as a CNN-based angular detail restoration on EPI. We indicate that one of the main challenges in sparsely sampled light field reconstruction is the information asymmetry between the spatial and angular domain, where the detail portion in the angular domain is damaged by undersampling. To balance the spatial and angular information, the spatial high frequency components of an EPI is removed using EPI blur, before feeding to the network. Finally, a non-blind deblur operation is used to recover the spatial detail suppressed by the EPI blur. We evaluate our approach on several datasets including synthetic scenes, real-world scenes and challenging microscope light field data. We demonstrate the high performance and robustness of the proposed framework compared with the state-of-the-arts algorithms. We also show a further application for depth enhancement by using the reconstructed light field.
Dense Light Field Reconstruction from Sparse Sampling Using Residual Network

Mantang Guo,Hao Zhu,Guoqing Zhou,Qing Wang

DOI: https://doi.org/10.1007/978-3-030-20876-9_4

2019-01-01

Abstract:A light field records numerous light rays from a real-world scene. However, capturing a dense light field by existing devices is a time-consuming process. Besides, reconstructing a large amount of light rays equivalent to multiple light fields using sparse sampling arises a severe challenge for existing methods. In this paper, we present a learning-based method to reconstruct multiple novel light fields between two mutually independent light fields. We indicate that light rays distributed in different light fields have the same consistent constraints under a certain condition. The most significant constraint is a depth related correlation between angular and spatial dimensions. Our method avoids working out the error-sensitive constraint by employing a deep neural network. We predict residual values of pixels on epipolar plane image (EPI) to reconstruct novel light fields. Our method is able to reconstruct 2 to 4 novel light fields between two mutually independent input light fields. We also compare our results with those yielded by a number of alternatives elsewhere in the literature, which shows our reconstructed light fields have better structure similarity and occlusion.
Light Field Reconstruction Using Convolutional Network on EPI and Extended Applications

Gaochang Wu,Yebin Liu,Lu Fang,Qionghai Dai,Tianyou Chai

DOI: https://doi.org/10.1109/TPAMI.2018.2845393

2021-03-24

Abstract:In this paper, a novel convolutional neural network (CNN)-based framework is developed for light field reconstruction from a sparse set of views. We indicate that the reconstruction can be efficiently modeled as angular restoration on an epipolar plane image (EPI). The main problem in direct reconstruction on the EPI involves an information asymmetry between the spatial and angular dimensions, where the detailed portion in the angular dimensions is damaged by undersampling. Directly upsampling or super-resolving the light field in the angular dimensions causes ghosting effects. To suppress these ghosting effects, we contribute a novel "blur-restoration-deblur" framework. First, the "blur" step is applied to extract the low-frequency components of the light field in the spatial dimensions by convolving each EPI slice with a selected blur kernel. Then, the "restoration" step is implemented by a CNN, which is trained to restore the angular details of the EPI. Finally, we use a non-blind "deblur" operation to recover the spatial high frequencies suppressed by the EPI blur. We evaluate our approach on several datasets, including synthetic scenes, real-world scenes and challenging microscope light field data. We demonstrate the high performance and robustness of the proposed framework compared with state-of-the-art algorithms. We further show extended applications, including depth enhancement and interpolation for unstructured input. More importantly, a novel rendering approach is presented by combining the proposed framework and depth information to handle large disparities.

Image and Video Processing,Computer Vision and Pattern Recognition
Plenoptic Reconstruction

Lu Fang

DOI: https://doi.org/10.1007/978-981-97-6915-5_4

2024-01-01

Abstract:AbstractEmpowered by advanced plenoptic sensing systems, light-field imaging becomes one of the most extensively used methods for capturing 3D views of a scene. In contrast to the traditional input to a 3D graphics system, namely, scenes consisting of pre-defined geometric primitives with different materials and sets of lights, the input to a light field is only a set of 2D images which are informative and cost effective. Unfortunately, due to the limited sensor resolution, existing systems must balance the spatial and angular resolution, i.e., one can obtain dense sampling images in the spatial dimension but only sparse sampling images in the angular (viewing angle) dimension or vice versa.
Deep-learning-enhanced light-field microscopy

Kai Wang

DOI: https://doi.org/10.1038/s41592-021-01151-1

IF: 48

2021-05-01

Nature Methods

Abstract:Light-field microscopes can image three-dimensional dynamics of biological samples at unprecedented speed, but the computational reconstruction necessary for image formation is artifact-prone and time-consuming. Deep learning closes this gap between imaging and reconstruction speed.

biochemical research methods

Dynamic light field reconstruction via densely connected deep equilibrium model

Dynamic Light Field Reconstruction Based on Gradient Descent Deep Equilibrium Model(Invited)

Three Dimensional Reconstruction Using a Lenslet Light Field Camera

Light Field Imaging: Models, Calibrations, Reconstructions, and Applications

Unsupervised Dense Light Field Reconstruction with Occlusion Awareness

An Efficient Anti-Occlusion Depth Estimation Using Generalized EPI Representation in Light Field

3D Scene Reconstruction with an Un-calibrated Light Field Camera

Anti-Occlusion Light-Field Optical Flow Estimation Using Light-Field Super-Pixels

Epipolar Focus Spectrum: A Novel Light Field Representation and Application in Dense-view Reconstruction

Sheared Epipolar Focus Spectrum for Dense Light Field Reconstruction.

Deep-learning Based Dual-View Light-Field Microscopy Enabling High-Resolution 3D Imaging of Dense Signals

Light-field-depth-estimation Network Based on Epipolar Geometry and Image Segmentation.

Dense reconstruction for sparsely-sampled light field

72‐2: A High Resolution Near Eye Light Field Display with Fast Reconstruction Speed

Light field angular super-resolution based on intrinsic and geometric information.

High-resolution Light Field Reconstruction Using a Hybrid Imaging System

Light Field Reconstruction Using Deep Convolutional Network On Epi

Dense Light Field Reconstruction from Sparse Sampling Using Residual Network

Light Field Reconstruction Using Convolutional Network on EPI and Extended Applications

Plenoptic Reconstruction

Deep-learning-enhanced light-field microscopy