Defocused Image Formation Model for Plenoptic Imaging

Yanqin Chen,Xin Jin,Qionghai Dai

DOI: https://doi.org/10.1364/3d.2018.jtu4a.40

2018-01-01

Abstract:This paper proposes to extend the PSF matrix of plenoptic imaging systems at focused depth to desirable defocused depth by multiplying a defocus approximation matrix. This extension is beneficial for saving time and hardware memory.

What problem does this paper attempt to address?

Investigating deep optics model representation in affecting resolved all-in-focus image quality and depth estimation fidelity

Xin Liu,Linpei LI,Xu Liu,Xiang Hao,Yifan Peng

DOI: https://doi.org/10.1364/OE.473084

IF: 3.8

2022-01-01

Optics Express

Abstract:The end-to-end (E2E) optimization of optics and image processing, dubbed deep optics, has renewed the state-of-the-art in various computer vision tasks. However, specifying the proper model representation or parameterization of the optical elements remains elusive. This article comprehensibly investigates three modeling hypotheses of the phase coded-aperture imaging under a representative context of deep optics, joint all-in-focus (AiF) imaging and monocular depth estimation (MDE). Specifically, we analyze the respective trade-off of these models and provide insights into relevant domain-specific requirements, explore the connection between the spatial feature of the point spread function (PSF) and the performance trade-off between the AiF and MDE tasks, and discuss the model sensitivity to possible fabrication errors. This study provides new prospects for future deep optics designs, particularly those aiming for AiF and/or MDE.(c) 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
Multi-image-distance Imaging System for Extending Depth-of-field

Jixiang Tang,Xuanyin Wang,Huan Zhou,Jiayu Ji,Zhengxiao Li,Zijian Ye

DOI: https://doi.org/10.1016/j.ijleo.2023.170965

IF: 3.1

2023-01-01

Optik

Abstract:Depth of field (DoF) defines the longitudinal range of objects in sharp focus. Imaging systems for machine vision are generally limited to a small DoF by various conditions, constraining the visual inspection of objects with large depth. In this paper, we develop a multi-image-distance imaging system, which can extend the DoF by capturing multi-focus images for fusion. A three-way beam-splitter prism is designed to support independent imaging of three image sensors. Through adjusting the sensors to different back focal lengths, multiple images focusing at different depths are captured in a single exposure. Finally, the focused regions of each image are fused for extending the DoF. Analyses of the proposed DoF extension strategy illustrate excellent enhancement more than three times. Moreover, we introduce a DoF measurement method based on the estimation of the point spread function, and measurement experiments are conducted using the fabricated prototype. Imaging experiments for various scenes demonstrate great imaging quality and DoF extension performance. The proposed multi-image-distance imaging system provides an effective and practical DoF extension solution for close-range visual inspection.
Extended Depth of Field Reconstruction with Complex Field Estimation

Jing Hu,Sunzheng Li,Yibing Shen,JinLei Zhang,Zhenrong Zheng

DOI: https://doi.org/10.1016/j.optlastec.2022.108118

IF: 4.939

2022-01-01

Optics & Laser Technology

Abstract:The contradiction between high resolution and large depth of field (DOF) of microscopic imaging limits its application in imaging 3D distributed samples. Here, we propose a new technique to achieve extended depth of field reconstruction in bright-field microscopy through complex field estimation. The complex field is estimated by a phase retrieval algorithm with the four illumination modulation patterns and the corresponding captured Fourier intensities as dual constraints. Then, the digital refocusing strategy based on Fresnel propagator is integrated into reconstruction framework to achieve the reconstruction of high-resolution complex-valued information for each part of the sample, which ensures an effective extension of the DOF. The feasibility of the proposed method to maintain high resolution over extended DOF was verified in actual experiments with a tilted USAF target, biological sample and microlens array (MLA). The extended DOF reconstruction provides a robust method for imaging in both industrial inspection and biomedical research.
An Efficient Projection Defocus Algorithm Based on Multi-Scale Convolution Kernel Templates

Bo Zhu,Li-jun Xie,Guang-hua Song,Yao Zheng

DOI: https://doi.org/10.1631/jzus.c1300080

2013-01-01

Journal of Zhejiang University SCIENCE C

Abstract:The focal problems of projection include out-of-focus projection images from the projector caused by incomplete mechanical focus and screen-door effects produced by projection pixilation. To eliminate these defects and enhance the imaging quality and clarity of projectors, a novel adaptive projection defocus algorithm is proposed based on multi-scale convolution kernel templates. This algorithm applies the improved Sobel-Tenengrad focus evaluation function to calculate the sharpness degree of intensity equalization and then constructs multi-scale defocus convolution kernels to remap and render the defocus projection image. The resulting projection defocus corrected images can eliminate out-of-focus effects and improve the sharpness of uncorrected images. Experiments show that the algorithm works quickly and robustly and that it not only effectively eliminates visual artifacts and can run on a self-designed smart projection system in real time but also significantly improves the resolution and clarity of the observer’s visual perception.
High-Speed Light Field Image Formation Analysis Using Wavefield Modeling with Flexible Sampling

Yanqin Chen,Xin Jin,Li Liu,Qionghai Dai

DOI: https://doi.org/10.1109/icassp.2018.8461335

2018-01-01

Abstract:Understanding the image formation inside plenoptic cameras is significant for the investigations of improving the low spatial resolution. Most researches explore the image formation from the perspective of geometric optics. However, as the hardware components in combination with low-aperture optical systems become smaller and smaller, geometric analysis will no longer be valid due to diffraction effects. In this paper, a wave-optic-based model is proposed that uses the Fresnel diffraction equation to propagate the whole object field into the plenoptic systems. The proposed model employs averaging of intensities on the sensor from uncorrelated coherent wave to avoid interference during propagations among the optical component planes. Besides, by utilizing the method of multiple partial propagations, the proposed model is much flexible at sampling on propagation planes. In order to verify the effectiveness of the proposed model, numerical simulations are conducted by comparing with existing wave optic model under different optical configurations of plenoptic cameras. Results demonstrate that the proposed model can describe the light field image formation properly. In addition, the time for image formation has been reduced by a factor of 19.22 using the proposed model.
Depth-of-Field-Extended Plenoptic Camera Based on Tunable Multi-Focus Liquid-Crystal Microlens Array

Mingce Chen,Wenda He,Dong Wei,Chai Hu,Jiashuo Shi,Xinyu Zhang,Haiwei Wang,Changsheng Xie

DOI: https://doi.org/10.3390/s20154142

IF: 3.9

2020-07-25

Sensors

Abstract:Plenoptic cameras have received a wide range of research interest because it can record the 4D plenoptic function or radiance including the radiation power and ray direction. One of its important applications is digital refocusing, which can obtain 2D images focused at different depths. To achieve digital refocusing in a wide range, a large depth of field (DOF) is needed, but there are fundamental optical limitations to this. In this paper, we proposed a plenoptic camera with an extended DOF by integrating a main lens, a tunable multi-focus liquid-crystal microlens array (TMF-LCMLA), and a complementary metal oxide semiconductor (CMOS) sensor together. The TMF-LCMLA was fabricated by traditional photolithography and standard microelectronic techniques, and its optical characteristics including interference patterns, focal lengths, and point spread functions (PSFs) were experimentally analyzed. Experiments demonstrated that the proposed plenoptic camera has a wider range of digital refocusing compared to the plenoptic camera based on a conventional liquid-crystal microlens array (LCMLA) with only one corresponding focal length at a certain voltage, which is equivalent to the extension of DOF. In addition, it also has a 2D/3D switchable function, which is not available with conventional plenoptic cameras.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
Point Spread Function for the Wide-Field-of-view Plenoptic Cameras

Xin Jin,Kunyi Li,Chuanpu Li,Xufu Sun

DOI: https://doi.org/10.1364/oe.431006

IF: 3.8

2021-01-01

Optics Express

Abstract:Recently, single or multi-layer spherical lens (monocentric lens) coupled with a microlens array (MLA) and an imaging sensor are under investigation to expand the field of view (FOV) for handheld plenoptic cameras. However, there lacks modeling the point spread functions (PSFs) for them to improve the imaging quality and to reconstruct the light field in the object space. In this paper, a generic image formation model is proposed for wide-FOV plenoptic cameras that use a monocentric lens and an MLA. By analyzing the optical characteristics of the monocentric lens, we propose to approximate it by a superposition of a series of concentric lenses with variable apertures. Based on geometry simplification and wave propagation, the equivalent imaging process of each portion of a wide-FOV plenoptic camera is modeled, based on which the PSF is derived. By comparing PSFs captured by real wide-FOV plenoptic camera and those generated by the proposed model, the validity of this model is verified. Further, reconstruction process is applied by deconvolving captured images with the PSFs generated by the proposed model. Experimental results show that the quality of reconstructed images is better than that of subaperture images, which demonstrates that our proposed PSF model is beneficial for imaging quality improvement and light field reconstruction.
Defocusing Response Function Optimization in Depth Estimation of Boundary Complex Light Field Image

Wu Yingchun,Cheng Xing,Xie Yingxian,Wang Anhong

DOI: https://doi.org/10.3788/lop57.181027

2020-01-01

Laser & Optoelectronics Progress

Abstract:The defocus and correspondence (DCDC) algorithm uses the spatial information and angular information of four-dimensional light field to complete depth estimation. The defocus response function established by the DCDC algorithm has the problems of limited second-order derivative direction and positive and negative energy offset, which limit its depth reconstruction accuracy in complex scenes. Aiming at the problems, the defocusing response function is improved, and a light field depth estimation algorithm based on energy enhanced defocusing response is proposed. The algorithm fully considers the influences of the surrounding pixels on the defocusing degree of the current position, and realizes the energy enhancement of the defocusing response by increasing the number and direction of second-order derivatives with different weights. Experiments verify the effectiveness of the proposed method. For some light field images with complex boundaries, the depth map obtained by the proposed algorithm has better visual effects. Compared with the DCDC algorithm, the root mean square error of the depth image obtained by the proposed method decreases by 3.95% on average.
Depth Extension For Multiple Focused Images By Adaptive Point Spread Function

Yuanqing Wang,Yijun Mei

DOI: https://doi.org/10.1007/978-3-642-05253-8_60

2009-01-01

Abstract:A novel depth fusion algorithm is proposed for multi-focused images based on point spread functions (PSF). In this paper, firstly, a discrete PSF model is present and functions with different parameters are prepared for the proposed algorithm. Based on the analysis of the effect of PSF, the detail process to fuse the depth of multi-focused images is described. By employing PSF to convolve each original images and comparing with its adjacent one in the image sequence, the focused and defocused region in the original images may be located. Combining the the focused region in the images according to Choose Max rules, an all-in-focus image may be got. The complexity of the algorithm for an image series with more than two original images is discussed at the end of this paper. Experimental results show that the image is distinctly segmented into multi-regions and the image edge is legible as well. The proposed algorithm based on PSF convolvetion as a focus measure has been shown to be experimentally valid. The fusion results are satisfactory with smooth transitions across region boundaries.
Super-resolved Refocusing with a Plenoptic Camera

Zhiliang Zhou,Yan Yuan,Bin Xiangli,Lulu Qian

DOI: https://doi.org/10.1117/12.872447

2011-01-01

Abstract:This paper presents an approach to enhance the resolution of refocused images by super resolution methods. In plenoptic imaging, we demonstrate that the raw sensor image can be divided to a number of low-resolution angular images with sub-pixel shifts between each other. The sub-pixel shift, which defines the super-resolving ability, is mathematically derived by considering the plenoptic camera as equivalent camera arrays. We implement simulation to demonstrate the imaging process of a plenoptic camera. A high-resolution image is then reconstructed using maximum a posteriori (MAP) super resolution algorithms. Without other degradation effects in simulation, the super resolved image achieves a resolution as high as predicted by the proposed model. We also build an experimental setup to acquire light fields. With traditional refocusing methods, the image is rendered at a rather low resolution. In contrast, we implement the super-resolved refocusing methods and recover an image with more spatial details. To evaluate the performance of the proposed method, we finally compare the reconstructed images using image quality metrics like peak signal to noise ratio (PSNR).
Multi-channel Residual Network Model for Accurate Estimation of Spatially-Varying and Depth-Dependent Defocus Kernels.

Yanpeng Cao,Zhangyu Ye,Zewei He,Jiangxin Yang,Yanlong Cao,Christel-Loic Tisse,Michael Ying Yang

DOI: https://doi.org/10.1364/oe.383127

IF: 3.8

2020-01-01

Optics Express

Abstract:Digital projectors have been increasingly utilized in various commercial and scientific applications. However, they are prone to the out-of-focus blurring problem since their depth-of-fields are typically limited. In this paper, we explore the feasibility of utilizing a deep learning-based approach to analyze the spatially-varying and depth-dependent defocus properties of digital projectors. A multimodal displaying/imaging system is built for capturing images projected at various depths. Based on the constructed dataset containing well-aligned in-focus, out-of-focus, and depth images, we propose a novel multi-channel residual deep network model to learn the end-to-end mapping function between the in-focus and out-of-focus image patches captured at different spatial locations and depths. To the best of our knowledge, it is the first research work revealing that the complex spatially-varying and depth-dependent blurring effects can be accurately learned from a number of real-captured image pairs instead of being hand-crafted as before. Experimental results demonstrate that our proposed deep learning-based method significantly outperforms the state-of-the-art defocus kernel estimation techniques and thus leads to better out-of-focus compensation for extending the dynamic ranges of digital projectors.
Theory and implementation of depth photography

Ziwei Liu,Tingfa Xu,Hongqing Wang,Ziyi Shen,Zhitao Rao

DOI: https://doi.org/10.3788/IRLA201645.0726001

2016-01-01

Infrared and Laser Engineering

Abstract:Traditional imaging methods are subject to limitations on information acquisition, which brings about the deficiency of image quality. Therefore, a depth photography model was proposed here. This model contains a depth matrix, a decomposition function, a defocus operator, and an adaptive regularization term. The depth matrix could be estimated using the binocular stereo vision method, the structured light approach, or the time-of-flight algorithm. The decomposition function was used to segment the image into pieces according to the distinct depth values. The defocus operator was calculated through a defocus-from-depth method. The adaptive regularization term reduces the staircase effect and enhances image smoothness. Local standard deviations and local average gradients were used to evaluate the effectiveness of depth photography model. Experimental results demonstrate that the proposed model is effective.
Point Spread Function and Depth-Invariant Focal Sweep Point Spread Function for Plenoptic Camera 20

Xin Jin,Li Liu,Yanqin Chen,Qionghai Dai

DOI: https://doi.org/10.1364/oe.25.009947

IF: 3.8

2017-01-01

Optics Express

Abstract:This paper derives a mathematical point spread function (PSF) and a depth-invariant focal sweep point spread function (FSPSF) for plenoptic camera 2.0. Derivation of PSF is based on the Fresnel diffraction equation and image formation analysis of a self-built imaging system which is divided into two sub-systems to reflect the relay imaging properties of plenoptic camera 2.0. The variations in PSF, which are caused by changes of object's depth and sensor position variation, are analyzed. A mathematical model of FSPSF is further derived, which is verified to be depth-invariant. Experiments on the real imaging systems demonstrate the consistency between the proposed PSF and the actual imaging results.
Large Depth-of-field Fringe Projection Profilometry Applied to Defocused Scenes

Peng Chen,Yunjie Wu,Shixiang Wang,Lingbao Kong

DOI: https://doi.org/10.1364/oe.519459

IF: 3.8

2024-01-01

Optics Express

Abstract:Fringe projection profilometry (FPP) is a widely adopted technique for threedimensional (3D) reconstruction. However, its depth-of-field (DOF) is constrained when reconstructing defocused scenes, mainly due to limitations in the camera model and image blur. This study introduces a camera model based on the ideal optical system, which effectively reduces the systematic errors associated with the conventional pinhole camera model. A calibration method to determine the optical system parameters of the improved camera model is proposed. Additionally, the point spread function (PSF) of the camera is calibrated and the image is deblur red through non-blind deconvolution, thereby minimizing the phase aliasing resulting from defocus. Experimental results validate the potential of the proposed method for accurate 3D reconstruction in scenes with a wide depth range.
Extended depth of field for Fresnel zone aperture camera via fast passive depth estimation

Chen Yang,Cong Ni,Zhang Xinye,Yusen Li,Yusheng Zhai,Weiji He,Wenwen Zhang,Qian Chen

DOI: https://doi.org/10.1364/oe.519871

IF: 3.8

2024-03-14

Optics Express

Abstract:Chen Yang, Cong Ni, Xinye Zhang, Yusen Li, Yusheng Zhai, Weiji He, Wenwen Zhang, Qian Chen The lensless camera with incoherent illumination has gained significant research interest for its thin and flexible structure. However, it ... [Opt. Express 32, 11323-11336 (2024)]

optics
Depth-of-field extended Fourier ptychographic microscopy without defocus distance priori

Yanqi Chen,Jinghao Xu,An Pan

DOI: https://doi.org/10.1364/ol.524267

IF: 3.6

2024-06-01

Optics Letters

Abstract:Yanqi Chen, Jinghao Xu, An Pan Fourier ptychographic microscopy (FPM) provides a solution of high-throughput phase imaging. Thanks to its coherent imaging model, FPM has ... [Opt. Lett. 49, 3222-3225 (2024)]

optics
Image formation analysis and light field information reconstruction for plenoptic camera 2.0

Li Liu,Xin Jin,Qionghai Dai

DOI: https://doi.org/10.1007/978-3-319-77380-3_58

2017-01-01

Abstract:This paper builds an image formation model of plenoptic camera 2.0 and reconstructs the light field via solving a regularized optimization problem. The image formation model is built based on the cognition that the plenoptic camera 2.0 is a linear forward imaging system, which is also verified via the derivation of the impulse response of plenoptic camera 2.0. Based on the proposed image formation model, the method to reconstruct the light field is analyzed by solving an inverse optimization problem. Simulation results for plenoptic camera 2.0 with a single microlens and a microlens array all indicate that the reconstructed the light field has more details and conveys more objects' information.
Study on the Method of Depth Measurement Based on Depth from Defocus

FENG Xiaoyun,FU Jianzhong

DOI: https://doi.org/10.3969/j.issn.1001-3881.2006.10.016

2006-01-01

Abstract:Depth from defocus is the method to get an image's depth in Z direction from 2-D image plane,there are many methods to calculate the defocus feature σ.a optimized method which is less sensitive to noise to extract the depth was presented,which improves the precision of the depth measurement.
Plenoptic 2.0 Intra Coding Using Imaging Principle

Xin Jin,Fan Jiang,Lingjun Li,Tingting Zhong

DOI: https://doi.org/10.1109/tbc.2021.3108058

IF: 4.5

2022-01-01

IEEE Transactions on Broadcasting

Abstract:Plenoptic 2.0 videos that are captured by focused plenoptic cameras outperform the traditional plenoptic videos with higher spatial resolution of rendered subaperture images and more effective light field sampling. However, its distinct imaging principle causes different pixel intensity distribution and complex macropixel structures, which results in ineffectiveness in coding them using the existing plenoptic video compression methods. Thus, a novel compression method is proposed in this paper to dramatically improve the intra prediction efficiency for multi-focus plenoptic 2.0 video coding. It consists of two new intra prediction modes: imaging-principle-guided static prediction (ISP) and imaging-principle-guided zoomed prediction (IZP). In ISP, the prediction candidates spatially positioned by plenoptic 2.0 imaging principle are weighted by minimizing boundary error to generate the predicted samples. In IZP, the spatial positions and scales of the prediction candidates vary according to the focusing status and the focal length of the microlens that generates the pixels in the prediction candidates. The combination of two modes generalizes plenoptic 2.0 video coding for multi-focus plenoptic cameras. The proposed approach outperforms high efficiency video coding (HEVC) standard by an average of 15.37% bitrate reduction under All Intra configuration. It also outperforms other state-of-the-art methods with obvious compression efficiency improvements.
Depth of Field Extended Scattering Imaging by Light Field Estimation

Xin Jin,Zhouping Wang,Xiaoyu Wang,Qionghai Dai

DOI: https://doi.org/10.1364/ol.43.004871

IF: 3.6

2018-01-01

Optics Letters

Abstract:Imaging through scattering media is challenging especially for three-dimensional scenarios. A lot of efforts have been made, but they suffer from tedious point spread function acquisition process or limited depth of field (DOF). In this Letter, a DOF extended scattering imaging method is proposed based on light field estimation. Using speckle intensities captured at two depths as constraints, a fast wavefront recovery model is proposed to estimate the phases with low computational complexity. Secondary propagation of the recovered wavefront is performed to generate the speckle intensity patterns at the target depths, and they are utilized to deconvolute integrated intensity matrices for extending imaging DOF. The effectiveness of the proposed method is demonstrated by both simulated and real experiments.

Defocused Image Formation Model for Plenoptic Imaging

Investigating deep optics model representation in affecting resolved all-in-focus image quality and depth estimation fidelity

Multi-image-distance Imaging System for Extending Depth-of-field

Extended Depth of Field Reconstruction with Complex Field Estimation

An Efficient Projection Defocus Algorithm Based on Multi-Scale Convolution Kernel Templates

High-Speed Light Field Image Formation Analysis Using Wavefield Modeling with Flexible Sampling

Depth-of-Field-Extended Plenoptic Camera Based on Tunable Multi-Focus Liquid-Crystal Microlens Array

Point Spread Function for the Wide-Field-of-view Plenoptic Cameras

Defocusing Response Function Optimization in Depth Estimation of Boundary Complex Light Field Image

Depth Extension For Multiple Focused Images By Adaptive Point Spread Function

Super-resolved Refocusing with a Plenoptic Camera

Multi-channel Residual Network Model for Accurate Estimation of Spatially-Varying and Depth-Dependent Defocus Kernels.

Theory and implementation of depth photography

Point Spread Function and Depth-Invariant Focal Sweep Point Spread Function for Plenoptic Camera 20

Large Depth-of-field Fringe Projection Profilometry Applied to Defocused Scenes

Extended depth of field for Fresnel zone aperture camera via fast passive depth estimation

Depth-of-field extended Fourier ptychographic microscopy without defocus distance priori

Image formation analysis and light field information reconstruction for plenoptic camera 2.0

Study on the Method of Depth Measurement Based on Depth from Defocus

Plenoptic 2.0 Intra Coding Using Imaging Principle

Depth of Field Extended Scattering Imaging by Light Field Estimation