Abstract:Consumer-level RGB-D cameras have been widely used for dense 3D reconstruction of scenes. Especially for textureless or non-lambertian surfaces, consumer RGB-D cameras can ensure completeness of the reconstructed models at a low cost. However, the reconstruction quality relies heavily on the accuracy of the depth sensors. Digital cameras are also used popularly for capturing high-resolution pictures to achieve high-quality dense reconstruction of the scenes, but cannot handle textureless or non-lambertian regions well due to the visual ambiguity problem. To ensure both completeness and accuracy of the reconstructed 3D models, we propose a hybrid multi-view reconstruction pipeline named Hybrid-MVS, which combines the high-resolution images taken by a digital camera and the low-resolution RGB-D frames captured by a consumer RGB-D camera for robust reconstruction of complicated scenes with challenging textureless and non-lambertian surfaces. Unlike most existing multi-sensor systems which require explicit hardware calibration and synchronization of various sensors, the calibration and synchronization problems between the digital camera and RGB-D camera are implicitly solved for compositing reliable depth prior of the digital images in our pipeline. Especially, we propose a hybrid MVS framework for robust PatchMatch stereo and Delaunay meshing, which tightly couples both visual cues given by the digital images and depth cues from the RGB-D frames to maximize the complementary advantages. The experiments with quantitative and qualitative evaluations demonstrate the effectiveness of the proposed Hybrid-MVS framework, which can successfully achieve high-quality 3D reconstruction of complicated natural scenes with robustness to weakly textured and non-lambertian areas.

End-to-end optimization of single-shot monocular metasurface camera for RGBD imaging

High-performance full-color imaging system based on end-to-end joint optimization of computer-generated holography and metalens.

DoCam: Depth Sensing with an Optical Image Stabilization Supported RGB Camera.

Machine-learning Reprogrammable Metasurface Imager.

Calibration-free Deep Optics for Depth Estimation with Precise Simulation

A Compact Compressive Hyperspectral Imaging System Based on Metasurface

Metalens-integrated Compact Imaging Devices for Wide-Field Microscopy

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

Single-shot three-dimensional imaging with a metasurface depth camera

Monocular metasurface camera for passive single-shot 4D imaging

End-to-end joint optimization of metasurface and image processing for compact snapshot hyperspectral imaging

End-to-end metasurface inverse design for single-shot multi-channel imaging

Aberration-robust monocular passive depth sensing using a meta-imaging camera

Single-shot 3D imaging with point cloud projection based on metadevice

End-to-End Optimization of Metasurfaces for Imaging with Compressed Sensing

Hybrid-MVS: Robust Multi-View Reconstruction with Hybrid Optimization of Visual and Depth Cues

Single-shot Passive 3D and Polarization Vision Using a Monocular Metasurface Camera

Single-shot and single-sensor high/super-resolution microwave imaging based on metasurface

Active 3D positioning and imaging modulated by single fringe projection with compact metasurface device

High-fidelity achromatic metalens imaging via deep neural network

Miniature optical planar camera based on a wide-angle metasurface doublet corrected for monochromatic aberrations