Light field camera calibration and point spread function calculation based on differentiable ray tracing

Di Mei,Yinsen Luan,Xiangchao Li,XiJiang Wu

DOI: https://doi.org/10.1364/ol.507898

IF: 3.6

2024-02-14

Optics Letters

Abstract:Di Mei, Yinsen Luan, Xiangchao Li, XiJiang Wu The imaging process of the light field (LF) camera with a micro-lens array (MLA) may suffer from multiple aberrations. It is thus difficult ... [Opt. Lett. 49, 965-968 (2024)]

optics

What problem does this paper attempt to address?

Three Dimensional Surface Reconstruction Based on Light Field Imaging

Sun Shiyi,Kuang Cuifang,Liu Xu

DOI: https://doi.org/10.5768/jao201738.0202001

2017-01-01

Journal of Applied Optics

Abstract:Traditional imaging methods can not capture axial information through one shot,which lead to information deficiency and incapability of three dimentional surface reconstruction.Unlike traditional camera,lytro's first generation pocket-sized camera using light field sensor can capture multi-dimensional light field information through one single exposure,and at the same time it has advantages such as portable.By using light field camera,with digital refocusing method,defocusing and correspondence cues,acquirement of high quality depth map can be realized.By using matlab platform to process acquired depth map matrix,reconstruction of three dimensional surface can be realized.Normalized depth data of object space is obtained.In this experiment,three dimensional surface reconstruction is realized with average error of 5.47％ in depth range of 100mm to 1 500 mm,and maximum reconstruction error is 8.30&.
Simulation Of Imaging And Refocusing For Light Field Photography

Han Huang,Xiaotong Li,Zhaofeng Cen

DOI: https://doi.org/10.1117/12.2037128

2013-01-01

Abstract:The entire process of image formation and refocusing of light field photography with microlens array is simulated using the in-house software OTS based on real ray tracing. The optical system consists of a photographic lens and the microlens array which are specifically designed. Objects are placed in different distance before the lens, and images of them on the detector plane are got by imaging simulation algorithm. Utilizing the color, intensity and the corresponding direction information of light rays on the image plane, pictures focusing at different depth in the image space are calculated, so that we can refocus at different objects in different distance. The paper also details the basic requirement of optical system and this simulation process is a fundamental work of research on design method and performance of lens using in light field photography.
Calibration of a Camera-Array-based Microscopic System with Spatiotemporal Structured Light Encoding

Jing Hu,Jingtao Fan,Yibing Shen

DOI: https://doi.org/10.1016/j.optcom.2021.127503

IF: 2.4

2021-01-01

Optics Communications

Abstract:The strong need in life science and medicine research has prompted the development of wide-field, high resolution and real-time microscopic imaging based on camera array. For large-scale sub-image stitching on the order of billion pixels, it is highly challenging to calibrate the camera array with low-overlapping fields of view (FOVs). To address the challenge, we propose a camera array calibration method to obtain the complete camera model and global mapping coordinates. The method utilizes spatiotemporal structured light encoding provided by a spatial light modulator to establish the correspondence between the physical coordinates of the keypoints in the object plane and their pixel coordinates in the sub-image plane. This achieves fast and accurate calibration of a camera-array-based microscopic system without the requirement of sub-FOV overlap. A series of experiments, including reprojection evaluation and global image stitching on the resolution chart and biological specimen slice, verified the effectiveness of the proposed approach. The method supports calibration of global geometric position with 0.2-pixel accuracy and has a broad potential for application to general camera array-based imaging systems.
Three Dimensional Reconstruction Using a Lenslet Light Field Camera

Wentao Zhang,Shengqian Chang,Xiao Tao,Chang Wang,Chenning Tao,Peizheng Huang,Zhenrong Zheng

DOI: https://doi.org/10.1117/12.2319552

2018-01-01

Abstract:A novel method is proposed in this paper to accurately reconstruct the three-dimensional scenes by using a passive single-shot exposure with a lenslet light field camera. This method has better performance of 3D scenes reconstruction with both defocus and disparity depth cues captured by light field camera. First, the light field data is used to refocus and shift viewpoints to get a focal stack and multi-view images. In refocusing procedure, the phase shift theorem in the Fourier domain is first introduced to substitute shift in spatial domain, and sharper focal stacks can be obtained with less blurriness. Thus, 3D scenes can be reconstructed more accurately. Next, through multi-view images, disparity depth cues are obtained by performing correspondence measure. Then, the focal stack is used to compute defocus depth cues by focus measure based on gray variance. Finally, the focus cost is built to integrate both defocus and disparity depth cues, and the accurate depth map is estimated by using Graph Cuts based on the focus cost. Using this accurate depth map and all-in-focus image, the 3D structure in real world are accurately reconstructed. Our method is verified by a number of synthetic and real-world examples captured with a dense camera array and a Lytro light field camera.
Simulation of Light-Field Camera Imaging Based on Ray Splitting Monte Carlo Method

Bin Liu,Yuan,Sai Li,Yong Shuai,He-Ping Tan

DOI: https://doi.org/10.1016/j.optcom.2015.06.020

IF: 2.4

2015-01-01

Optics Communications

Abstract:As microlens technology matures, studies of structural design and reconstruction algorithm optimization for light-field cameras are increasing. However, few of these studies address numerical physical simulation of the camera, and it is difficult to track lighting technology for forward simulations because of its low efficiency. In this paper, we develop a Monte Carlo method (MCM) based on ray splitting and build a physical model of a light-field camera with a microlens array to simulate its imaging and refocusing processes. The model enables simulation of different imaging modalities, and will be useful for camera structural design and error analysis system construction.
Development of a Light Field Measuring System with Corrected Distortion Based on Imaging Analysis.

Panyu Zhou,Lingbao Kong,Junhua Wang,Min Xu

DOI: https://doi.org/10.1364/ao.393395

IF: 1.9

2020-01-01

Applied Optics

Abstract:A light field microscope can realize three-dimensional measurement through only one exposure by the addition of a critical optical component called a microlens array (MLA). MLA errors will introduce aberrations in the captured image and then affect measurement results. This paper proposes a light field measuring system with aberration correction based on imaging analysis. The effect of MLA errors is investigated and quantitatively analyzed through a series of simulation studies. Aberration correction is realized based on computational imaging, in which an image segmentation and fusion distortion model is employed to correct the distortion, while a gradient-based linear recognition algorithm is used to address MLA rotation errors. Experimental results show that the developed light field measuring system can achieve improved measurement accuracy by correcting aberrations with the proposed algorithms.
Light field angular super-resolution based on intrinsic and geometric information.

Lingyu Wang,Lifei Ren,Xiaoyao Wei,Jiangxin Yang,Yanlong Cao,Yanpeng Cao

DOI: https://doi.org/10.1016/j.knosys.2023.110553

2023-01-01

Abstract:Light field imaging can encode abundant scene information, including the intensity and direction of light rays, into 4D light field images. However, the limited number of sensors in commercial light field cameras leads to a trade-off between spatial and angular resolutions. In this paper, an angular super-resolution framework is proposed to synthesize new views and overcome hardware restrictions. First, light field intrinsic feature convolution is proposed to extract intrinsic information, i.e., scene content, complete view correlations, and epipolar structures. Consequently, spatial, angular, and cross-domain information can be preserved in the extracted features. Second, the spatial-angular and depth streams are proposed based on the light field intrinsic feature convolution to synthesize high angular resolution light fields. The spatial-angular stream utilizes the light field intrinsic information to improve the angular resolution, whereas the depth stream disentangles the geometric information from the extracted light field intrinsic features, which is used to warp the given sub-aperture images to the new view positions. Both streams can synthesize high-quality intermediate results, where the intrinsic and geometric information are utilized separately. Finally, a confidence-based stream fusion module is proposed to fuse the outputs from the two streams, achieving the joint employment of light field intrinsic and geometric information and solving the problem of insufficient information exploration in the current methods. We conduct a series of experiments to validate the effectiveness of each component in the framework and demonstrate that our method can achieve state-of-the-art performance in various scenes.(c) 2023 Elsevier B.V. All rights reserved.
Computer modeling and simulation of light field camera and digital refocusing with attenuating mask

XiuBao Zhang,Yan Yuan,Zhiliang Zhou,Chengming Sun

DOI: https://doi.org/10.1117/12.836543

2009-01-01

Abstract:This paper presents a kind of light field camera with a patterned mask inserted between lens and sensor in a conventional camera. The mask can attenuate the incident light rays as well as the baseband signal is carried by carrier signal during remote transmission to reduce energy loss in radio and telecommunication field. Linearly independent weighted sums of light rays can be recorded by the sensor and rays combined in coding way can be decoded according to modulation-demodulation theory. Through rearranging the tile of the 2D Fourier transform of image on sensor into 4D planes, the light field can be reconstructed. Although light field image is captured by the camera in single photographic exposure time, sharp images focused at different depths could be obtained by computation. This paper researches on the principle, modeling and simulation of the camera according to ray transmission path. Two defocused light field images are simulated by using some supposed camera parameters and the digital refocused images are reconstructed respectively. © 2009 SPIE.
Mobile Camera Array Calibration for Light Field Acquisition

Yichao Xu,Kazuki Maeno,Hajime Nagahara,Rin-ichiro Taniguchi

DOI: https://doi.org/10.48550/arXiv.1407.4206

2014-07-16

Computer Vision and Pattern Recognition

Abstract:The light field camera is useful for computer graphics and vision applications. Calibration is an essential step for these applications. After calibration, we can rectify the captured image by using the calibrated camera parameters. However, the large camera array calibration method, which assumes that all cameras are on the same plane, ignores the orientation and intrinsic parameters. The multi-camera calibration technique usually assumes that the working volume and viewpoints are fixed. In this paper, we describe a calibration algorithm suitable for a mobile camera array based light field acquisition system. The algorithm performs in Zhang's style by moving a checkerboard, and computes the initial parameters in closed form. Global optimization is then applied to refine all the parameters simultaneously. Our implementation is rather flexible in that users can assign the number of viewpoints and refinement of intrinsic parameters is optional. Experiments on both simulated data and real data acquired by a commercial product show that our method yields good results. Digital refocusing application shows the calibrated light field can well focus to the target object we desired.
Lensless light-field imaging through diffuser encoding

Zewei Cai,Jiawei Chen,Giancarlo Pedrini,Wolfgang Osten,Xiaoli Liu,Xiang Peng

DOI: https://doi.org/10.1038/s41377-020-00380-x

2020-08-19

Abstract:Microlens array-based light-field imaging has been one of the most commonly used and effective technologies to record high-dimensional optical signals for developing various potential high-performance applications in many fields. However, the use of a microlens array generally suffers from an intrinsic trade-off between the spatial and angular resolutions. In this paper, we concentrate on exploiting a diffuser to explore a novel modality for light-field imaging. We demonstrate that the diffuser can efficiently angularly couple incident light rays into a detected image without needing any lens. To characterize and analyse this phenomenon, we establish a diffuser-encoding light-field transmission model, in which four-dimensional light fields are mapped into two-dimensional images via a transmission matrix describing the light propagation through the diffuser. Correspondingly, a calibration strategy is designed to flexibly determine the transmission matrix, so that light rays can be computationally decoupled from a detected image with adjustable spatio-angular resolutions, which are unshackled from the resolution limitation of the sensor. The proof-of-concept approach indicates the possibility of using scattering media for lensless four-dimensional light-field recording and processing, not just for two- or three-dimensional imaging.
The Light Field 3D Scanner.

Yingliang Zhang,Zhong Li,Wei Yang,Peihong Yu,Haiting Lin,Jingyi Yu

DOI: https://doi.org/10.1109/iccphot.2017.7951484

2017-01-01

Abstract:We present a novel light field structure-from-motion (SfM) framework for reliable 3D object reconstruction. Specifically, we use the light field (LF) camera such as Lytro and Raytrix as a virtual 3D scanner. We move an LF camera around the object and register between multiple LF shots. We show that applying conventional SfM on sub-aperture images is not only expensive but also unreliable due to ultra-small baseline and low image resolution. Instead, our LF-SfM scheme maps ray manifolds across LFs. Specifically, we show how rays passing through a common 3D point transform between two LFs and we develop reliable technique for extracting extrinsic parameters from this ray transform. Next, we apply a new edge-preserving stereo matching technique on individual LFs and conduct LF bundle adjustment to jointly optimize pose and geometry. Comprehensive experiments show our solution outperforms many state-of-the-art passive and even active techniques especially on topologically complex objects.
Design of a Novel Microlens Array and Imaging System for Light Fields

Yifeng Li,Pangyue Li,Xinyan Zheng,Huachen Liu,Yiran Zhao,Xueping Sun,Weiguo Liu,Shun Zhou

DOI: https://doi.org/10.3390/mi15091166

IF: 3.4

2024-09-22

Micromachines

Abstract:Light field cameras are unsuitable for further acquisition of high-quality images due to their small depth of field, insufficient spatial resolution, and poor imaging quality. To address these issues, we proposed a novel four-focal-square microlens and light field system. A square aspheric microlens array with four orthogonal focal lengths was designed, in which the aperture of a single lens was 100 μm. The square arrangement improves pixel utilization, the four focal lengths increase the depth of field, and the aspheric improves image quality. The simulations demonstrate pixel utilization rates exceeding 90%, depth-of-field ranges 6.57 times that of a single focal length, and image quality is significantly improved. We have provided a potential solution for improving the depth of field and image quality of the light field imaging system.

chemistry, analytical,nanoscience & nanotechnology,instruments & instrumentation,physics, applied
Light Field Acquisition Using Microlens Based on 3ds Max and Image Reconstruction Technology

Chuntao Xu,Axiu Cao,Lifang Shi,Hui Pang,Liwei Liu,Wenjing Liu,Yingfei Pang,Yongqi Fu,Qiling Deng

DOI: https://doi.org/10.1117/12.2500967

2018-01-01

Abstract:The traditional imaging method can only obtain the two-dimensional information of the object space in lateral resolution through a single exposure, but cannot obtain the longitudinal depth information. The depth information of the object space will be lost because the object cannot be reconstructed in three dimensions. The light field imaging technology enables reconstruction of three-dimensional objects by means of adding microlens arrays into a conventional camera system. The technology has a wide range of applications in medical, military, and entertainment. In this paper, a light field acquisition technology using microlens based on 3ds Max is proposed. A 3D object model was established using 3ds Max. By establishing a virtual microlens array, the crosstalk-free, high resolution and fast acquisition of the light field image by the microlens can be realized. Simulation study of the light field imaging technology can provide a high-efficiency computational study. The acquired images are processed to reconstruct images from different perspectives. Finally, the light field imaging experiments based on microlens arrays is carried out to realize the image reconstruction in different perspective images. Reliability of the algorithm is verified.
Self‐calibration Three‐dimensional Light Field Display Based on Scalable Multi‐lcds

Yifan Peng,Haifeng Li,Rui Wang,Qing Zhong,Xiang Han,Zisheng Cao,Xu Liu

DOI: https://doi.org/10.1002/jsid.133

2012-01-01

Journal of the Society for Information Display

Abstract:Approach to achieve self-calibration three-dimensional (3D) light field display is investigated in this paper. The proposed 3D light field display is constructed up on spliced multi-LCDs, lens and diaphragm arrays, and directional diffuser. The light field imaging principle, hardware configuration, diffuser characteristic, and image reconstruction simulation are described and analyzed, respectively. Besides the light field imaging, a self-calibration method is proposed to improve the imaging performance. An image sensor is deployed to capture calibration patterns projected onto and then reflected by the polymer dispersed liquid crystal film, which is attached to and shaped the diffuser. These calibration components are assembled with the display unit and can be switched between display mode and calibration mode. In the calibration mode, the imperfect imaging relations of optical components are captured and calibrated automatically. We demonstrate our design by implementing the prototype of proposed 3D light field display by using modified off-the-shelf products. The proposed approach successfully meets the requirement of real application on scalable configuration, fast calibration, large viewing angular range, and smooth motion parallax.
Light field camera all-in-focus image acquisition based on angular information

Yingchun Wu,Xing Cheng,Jie Liang,Anhong Wang,Xianling Zhao

DOI: https://doi.org/10.37190/oa210212

IF: 0.6

2021-01-01

Optica Applicata

Abstract:Traditional light field all-in-focus image fusion algorithms are based on the digital refocusing technique. Multi-focused images converted from one single light field image are used to calculate the all-in-focus image and the light field spatial information is used to accomplish the sharpness evaluation. Analyzing the 4D light field from another perspective, an all-in-focus image fusion algorithm based on angular information is presented in this paper. In the proposed method, the 4D light field data are fused directly and a macro-pixel energy difference function based on angular information is established to accomplish the sharpness evaluation. Then the fused 4D data is guided by the dimension increased central sub-aperture image to obtain the refined 4D data. Finally, the all-in-focus image is calculated by integrating the refined 4D light field data. Experimental results show that the fused images calculated by the proposed method have higher visual quality. Quantitative evaluation results also demonstrate the performance of the proposed algorithm. With the light field angular information, the image feature-based index and human perception inspired index of the fused image are improved.
Computer simulation for digital refocusing imager based on light field photography

Zhiliang Zhou,Yan Yuan,Bin Xiangli

DOI: https://doi.org/10.1117/12.836679

2009-01-01

Abstract:This paper presents a computer simulation of light field photography that records the light field by inserting a microlens array in a conventional camera. A computational model is configured to emulate how the 4D light field is distributed in the camera and then captured on a 2D sensor. Based on the recorded light field, refocused images are calculated by spatial integration at different depths. In the Fourier domain, a refocused photograph can be obtained by taking an appropriate 2D slice in the 4D light field. Due to this theorem, another refocusing algorithm in the Fourier domain is particularly explored in this paper. After reconstructing a focal stack of images at all depths in the scene, a photograph with extended depth of field can be calculated by wavelet based image fusion methods. © 2009 SPIE.
Automatic Geometrical Calibration for Multiprojector-Type Light Field Three-Dimensional Display

Bei-shi Chen,Qing Zhong,Hai-feng Li,Xu Liu,Hai-song Xu

DOI: https://doi.org/10.1117/1.oe.53.7.073107

IF: 1.3

2014-01-01

Optical Engineering

Abstract:Abstract. An automatic geometrical calibration approach has been developed to calibrate a multiprojector-type light field (LF) display automatically and accurately. The calibration framework based on image mapping is detailed, which transfers the calibration of three-dimensional (3-D) scene into the calibration of two-dimensional image in the diffuser interface. A multiprojectors-type LF display prototype is applied to implement the experimental calibration. Comparison results of the reconstructed 3-D scene before and after calibration show that a better overall performance is obtained through the proposed calibration approach.
Stray light simulation and control in imaging systems including diffraction elements

Xiaotong Li,Zhaofeng Cen

DOI: https://doi.org/10.1117/12.2312444

2018-01-01

Abstract:Some imaging optical systems include diffraction elements, both the even reflected stray light and multi-order diffraction stray light need to be considered at the image. They are related to the reflectivity and diffraction efficiency of optical surfaces. In this paper the binary tree is employed to describe the multifurcating tree in which each sampling ray's information is kept. According to the multi-order diffraction efficiency, paraxial ray and real sampling rays are traced, and the calculating results are applied to analyze the stray light in optical systems including diffraction components, and then the stray light energy distribution at any space especially at the image can be obtained. A new calculated wavelength is used to replace the designing wavelength for binary optical elements to calculate the reflected diffraction stray light. Based on the principles above, the stray light analysis software is developed. By tracing and analyzing all sampling rays the positions of ghosts are shown, especially at image surface. In this procedure, the first surface of the system serves as the entrance pupil or stop, instead of the entrance pupil in original imaging system, so as to collect all sampling rays from all direction, in or out the field of view, and then all possible ghosts are obtained. According to the simulation, the glare shields or control methods can be designed.
Camera Array Based Light Field Microscopy

Xing Lin,Jiamin Wu,Guoan Zheng,Qionghai Dai

DOI: https://doi.org/10.1364/boe.6.003179

IF: 3.4

2015-01-01

Biomedical Optics Express

Abstract:This paper proposes a novel approach for high-resolution light field microscopy imaging by using a camera array. In this approach, we apply a two-stage relay system for expanding the aperture plane of the microscope into the size of an imaging lens array, and utilize a sensor array for acquiring different sub-apertures images formed by corresponding imaging lenses. By combining the rectified and synchronized images from 5 × 5 viewpoints with our prototype system, we successfully recovered color light field videos for various fast-moving microscopic specimens with a spatial resolution of 0.79 megapixels at 30 frames per second, corresponding to an unprecedented data throughput of 562.5 MB/s for light field microscopy. We also demonstrated the use of the reported platform for different applications, including post-capture refocusing, phase reconstruction, 3D imaging, and optical metrology.
Light Field Measurement Based On The Single-Lens Coherent Diffraction Imaging

Cheng Shen,Jiubin Tan,Zhengjun Liu

DOI: https://doi.org/10.1117/12.2295577

2017-01-01

Abstract:Plenoptic camera and holography are popular light field measurement techniques. However, the low resolution or the complex apparatus hinders their widespread application. In this paper, we put forward a new light field measurement scheme. The lens is introduced into coherent diffraction imaging to operate an optical transform, extended fractional Fourier transform. Combined with the multi-image phase retrieval algorithm, the scheme is proved to hold several advantages. It gets rid of the support requirement and is much easier to implement while keeping a high resolution by making full use of the detector plane. Also, it is verified that our scheme has a superiority over the direct lens focusing imaging in amplitude measurement accuracy and phase retrieval ability.

Light field camera calibration and point spread function calculation based on differentiable ray tracing

Three Dimensional Surface Reconstruction Based on Light Field Imaging

Simulation Of Imaging And Refocusing For Light Field Photography

Calibration of a Camera-Array-based Microscopic System with Spatiotemporal Structured Light Encoding

Three Dimensional Reconstruction Using a Lenslet Light Field Camera

Simulation of Light-Field Camera Imaging Based on Ray Splitting Monte Carlo Method

Development of a Light Field Measuring System with Corrected Distortion Based on Imaging Analysis.

Light field angular super-resolution based on intrinsic and geometric information.

Computer modeling and simulation of light field camera and digital refocusing with attenuating mask

Mobile Camera Array Calibration for Light Field Acquisition

Lensless light-field imaging through diffuser encoding

The Light Field 3D Scanner.

Design of a Novel Microlens Array and Imaging System for Light Fields

Light Field Acquisition Using Microlens Based on 3ds Max and Image Reconstruction Technology

Self‐calibration Three‐dimensional Light Field Display Based on Scalable Multi‐lcds

Light field camera all-in-focus image acquisition based on angular information

Computer simulation for digital refocusing imager based on light field photography

Automatic Geometrical Calibration for Multiprojector-Type Light Field Three-Dimensional Display

Stray light simulation and control in imaging systems including diffraction elements

Camera Array Based Light Field Microscopy

Light Field Measurement Based On The Single-Lens Coherent Diffraction Imaging