LiFCal: Online Light Field Camera Calibration via Bundle Adjustment

Aymeric Fleith,Doaa Ahmed,Daniel Cremers,Niclas Zeller
2024-08-21
Abstract:We propose LiFCal, a novel geometric online calibration pipeline for MLA-based light field cameras. LiFCal accurately determines model parameters from a moving camera sequence without precise calibration targets, integrating arbitrary metric scaling constraints. It optimizes intrinsic parameters of the light field camera model, the 3D coordinates of a sparse set of scene points and camera poses in a single bundle adjustment defined directly on micro image points. We show that LiFCal can reliably and repeatably calibrate a focused plenoptic camera using different input sequences, providing intrinsic camera parameters extremely close to state-of-the-art methods, while offering two main advantages: it can be applied in a target-free scene, and it is implemented online in a complete and continuous pipeline. Furthermore, we demonstrate the quality of the obtained camera parameters in downstream tasks like depth estimation and SLAM. Webpage: <a class="link-external link-https" href="https://lifcal.github.io/" rel="external noopener nofollow">this https URL</a>
Image and Video Processing,Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the online calibration problem of micro - lens array (MLA) - based light - field cameras, especially for the situation without an accurate calibration target. Specifically, the paper proposes a new method named LiFCal, which can accurately determine the parameters of the light - field camera model from a sequence of moving cameras without a calibration target and integrates an arbitrary metric scale constraint. LiFCal optimizes the internal parameters of the light - field camera model, the 3D coordinates of scene points, and the camera pose, all of which are completed in a bundle adjustment process that directly acts on micro - image points. ### Main contributions: 1. **Online calibration process**: It can accurately determine all internal parameters of the light - field camera without a calibration target in the scene. 2. **Bundle adjustment formula**: Minimize the reprojection error of micro - image coordinates on the light - field camera and integrate an arbitrary metric scale constraint as a complete structure - from - motion (SfM) process. 3. **Depth estimation integration**: Integrate the light - field camera model into the micro - image - based depth estimation method to provide undistorted metric depth maps and synthesized intensity images. ### Problem background: - **Visual odometry (VO) and simultaneous localization and mapping (SLAM)**: These techniques are widely used in fields such as virtual reality, augmented reality, and autonomous driving, mainly using monocular, stereo, or RGB - D cameras. - **Advantages of light - field cameras**: Light - field cameras perform well in depth estimation, super - resolution, VO, and SLAM, etc., and have a very large depth of field. However, the complex calibration process limits its mainstream applications. - **Requirement for online calibration**: With the change of camera configurations, target - free online calibration is crucial for accurate data (such as accurate 3D reconstruction). ### Method overview: - **Light - field camera model**: LiFCal uses a thin - lens model to describe the main lens, and a pinhole model for each micro - lens. Distortion is directly defined on the MLA and original image coordinates. - **Calibration process**: It includes initial calibration and light - field bundle adjustment. Initial calibration uses the pinhole camera model to estimate all camera poses, the internal parameters of the pinhole model, and 3D point coordinates. Light - field bundle adjustment further optimizes these parameters. - **Optimization process**: Through minimizing the cost function (see Equation (7)), nonlinear optimization is carried out using the Levenberg - Marquardt algorithm. ### Evaluation results: - **Comparison with reference calibration**: On public datasets and self - recorded images, the results of LiFCal are very close to the reference calibration method [42] based on a complex 3D calibration target, with good accuracy and repeatability. - **Online calibration in target - free scenes**: In different scenes, LiFCal can reliably perform online calibration, and the deviation of the results from the reference calibration is within 0.6%. - **Generalization ability for different cameras**: The experiment also shows the generalization ability of LiFCal for different sensors, such as the Raytrix R25 camera, and the results indicate that this method has good consistency and stability. ### Formula examples: - **Projection relationship**: \[ \lambda\cdot\begin{bmatrix} x_V\\ y_V\\ z_V\\ 1 \end{bmatrix}=\begin{bmatrix} b_{L0}&0&0&0\\ 0&b_{L0}&0&0\\ 0&0&b&0\\ 0&0&1&0 \end{bmatrix}\cdot\begin{bmatrix} x_C\\ y_C\\ z_C\\ 1 \end{bmatrix} \] - **Virtual depth**: \[ v = \frac{b}{B}=\frac{b_L - b_{L0}}{B}\Rightarrow b_L=v\cdot B + b_{L0} \] Through these methods and evaluations, the paper shows the efficient online calibration ability of LiFCal in target - free scenes and its application potential in downstream tasks.