Abstract:Consumer-level RGB-D cameras have been widely used for dense 3D reconstruction of scenes. Especially for textureless or non-lambertian surfaces, consumer RGB-D cameras can ensure completeness of the reconstructed models at a low cost. However, the reconstruction quality relies heavily on the accuracy of the depth sensors. Digital cameras are also used popularly for capturing high-resolution pictures to achieve high-quality dense reconstruction of the scenes, but cannot handle textureless or non-lambertian regions well due to the visual ambiguity problem. To ensure both completeness and accuracy of the reconstructed 3D models, we propose a hybrid multi-view reconstruction pipeline named Hybrid-MVS, which combines the high-resolution images taken by a digital camera and the low-resolution RGB-D frames captured by a consumer RGB-D camera for robust reconstruction of complicated scenes with challenging textureless and non-lambertian surfaces. Unlike most existing multi-sensor systems which require explicit hardware calibration and synchronization of various sensors, the calibration and synchronization problems between the digital camera and RGB-D camera are implicitly solved for compositing reliable depth prior of the digital images in our pipeline. Especially, we propose a hybrid MVS framework for robust PatchMatch stereo and Delaunay meshing, which tightly couples both visual cues given by the digital images and depth cues from the RGB-D frames to maximize the complementary advantages. The experiments with quantitative and qualitative evaluations demonstrate the effectiveness of the proposed Hybrid-MVS framework, which can successfully achieve high-quality 3D reconstruction of complicated natural scenes with robustness to weakly textured and non-lambertian areas.

A multi-layer occlusion handling method based on stereo vision and pose estimation for real-time augmented reality

Handling Occlusions in Video-Based Augmented Reality Using Depth Information

Real time stereo rendering for augmented reality on 3DTV system

Recurrent Volume-based 3D Feature Fusion for Real-time Multi-view Object Pose Estimation

Two-Stage Multi-Camera Constrain Mapping Pipeline for Large-Scale 3D Reconstruction

Real-time occlusion handling for augmented reality assistance assembly systems with monocular images

Recurrent Volume-Based 3-D Feature Fusion for Real-Time Multiview Object Pose Estimation.

Dynamic Occlusion Handling for Real-Time AR Applications

A touchable virtual screen interaction system with handheld Kinect camera

High-Quality Depth Recovery Via Interactive Multi-view Stereo

Mobile3DRecon: Real-time Monocular 3D Reconstruction on a Mobile Phone

A Temporally Streamlined Optimization Method for Stereo Video Correspondence

Robust Keyframe-Based Monocular SLAM for Augmented Reality

Accurate Real-Time Stereo Correspondence Using Intra- and Inter-Scanline Optimization

Real-time non-rigid shape recovery via active appearance models for augmented reality

Real-time Stereo to Multi-View Conversion System Based on Adaptive Meshing

Mixed Reality Depth Contour Occlusion Using Binocular Similarity Matching and Three-dimensional Contour Optimisation

Hybrid-MVS: Robust Multi-View Reconstruction with Hybrid Optimization of Visual and Depth Cues

Limited changes in severe functional mitral regurgitation and pulmonary hypertension after left ventricular assist device implantation: a clue to consider concurrent mitral correction?

Active stereo vision three-dimensional reconstruction by RGB dot pattern projection and ray intersection

Real-Time 3D Reconstruction Method Based on Monocular Vision