Abstract:Plenoptic cameras have been increasingly applied in three-dimensional (3D) optical imaging and measurements. Micro-lens array is the key component to capture the images at different depths simultaneously to realize the 3D reconstruction of the scenery. However, only a part of the captured information will be used eventually when focusing on specific objects, which implies a waste of time and resources that are incompetent for fast-response tasks like gesture recognition. Furthermore, it is still quite challenging to accurately calibrate the depths under complex circumstances, even sometimes losing a part of the target information due to the low signal-to-noise ratio. In this paper, a new hybrid optical system is developed combining a geometrical waveguide and micro-lens array to project a virtual scale network for quantitative depth calibration and fast-tracking of targets. Such a methodology based on the augmented reality (AR) mechanism helps to rapidly scope the targeted objects/features without reconstructing the full range model, significantly saving the processing time for a low-latency response. By establishing a geometrical model to quantitatively correlate the images in the auxiliary coordinate system and the virtual scale coordinate system, the depth error caused by the plenoptic camera can be calibrated and corrected. The coefficient of determination R 2 is used to evaluate the depth accuracy of 3D images and acts as the threshold to control the depth correction iterations. The closer the value of R 2 is to 1, the more accurate the depth information is. Experiments proved that even under complex backgrounds or insufficient light, in virtue of the virtual coordinate networks, backgrounds could be rapidly filtered, and targets were effectively identified with accurate depth information. The algorithm flow chart for the correction of depth error is given. In such a way, the system can achieve faster and more accurate capturing of 3D objects in a real-time manner than conventional plenoptic cameras.

Accurate Depth Recovery Method Based on the Fusion of Time-of-Flight and Dot-Coded Structured Light

Reliable Fusion of ToF and Stereo Data Based on Joint Depth Filter

K-nearest Neighborhood Based Integration of Time-of-flight Cameras and Passive Stereo for High-Accuracy Depth Maps.

DELTAR: Depth Estimation from a Light-Weight ToF Sensor and RGB Image

Iterative Error Removal for Time-of-Flight Depth Imaging

A Novel Depth Recovery Approach from Multi-View Stereo Based Focusing

K-nearestneighborhood Basedintegrationoftime-of-flight Camerasandpassivestereoforhigh-Accuracydepthmaps ∗

High quality depth maps from stereo matching and ToF camera.

A Fused Estimation Method of Depth Based on Depth from Stereo and Depth from Focus

Accurate 3D reconstruction using a multi-phase ToF camera

Spatio-Temporal Depth Recovery of Dynamic Scenes with Multiple Handheld Cameras

Real-Time Artifact Compensation for Depth Images of Multi-Frequency ToF

High-Quality Depth Recovery Via Interactive Multi-view Stereo

Expanding Sparse LiDAR Depth and Guiding Stereo Matching for Robust Dense Depth Estimation

Dense Depth Recovery Based on Adaptive Image Segmentation

Hybrid structured light for scalable depth sensing

LF-fusion: Dense and Accurate 3D Reconstruction from Light Field Images

Fusion of Time-of-Flight and Phase Shifting for High-Resolution and Low-Latency Depth Sensing

Restoration of Motion Blur in Time-of-Flight Depth Image Using Data Alignment.

Exploiting Temporal–Spatial Feature Correlations for Sequential Spacecraft Depth Completion

Depth error correction for plenoptic cameras based on an innovative AR system combining geometrical waveguide and micro-lens array