Abstract:PurposeTo make an augmented image realistic, the virtual objects should be correctly occluded by foreground objects. The purpose of this paper is to propose a new approach that resolves occlusion problems in augmented reality (AR). The main interest is that it can automatically obtain the proper spatial relationship between virtual and real objects in real time.Design/methodology/approachThe approach is divided into two steps: off‐line disparity map constructing and on‐line occlusion handling. In the off‐line stage, the disparity map of the real scene is constructed using the global stereo matching method prior and then the disparities are refined by means of the fast mean shift method. Since the depth values of objects in different positions are different, the real object that occludes the virtual object can be specified according to the depth value. In the on‐line stage, the contour of the specified object is tracked using the real time object tracking method with the combination of feature tracking method and minimum s‐t cut method. The augmented image with correct occlusions is produced by redrawing all the tracked object pixels on the augmented image.FindingsCompared with the existing methods, the proposed approach can automatically resolve occlusion problem in real time. The effectiveness of the method is demonstrated with several experimental results.Originality/valueThis paper makes three contributions. First, a novel framework is proposed to handle occlusion problem in AR. This framework is different to the previously proposed methods. The main procedure includes: obtain occluding real object, track the object, and redraw the pixels of the object on the composed image. It is much easier to implement and can achieve satisfactory results. Second, the disparity map is used to automatically obtain the contour of the occluding real object. To get the contour of the occluding real object precisely, the mean shift method is used to refine the disparity map. By comparing the depth value, the occluding real object can be extracted automatically. Third, the tracking method combining feature tracking method and minimum s‐t cut method ensures the real‐time requirement. The occlusion problem can be handled in real‐time.

Automatical Reconstruction of an Occluded Scene Plane Based on a Camera Array

A Simple 3D Reconstruction Method Based on Single Camera

Occluded-Object 3D Reconstruction Using Camera Array Synthetic Aperture Imaging.

Reconstructing Scene Depth and Appearance Behind Foreground Occlusion Using Camera Array

Automatic Reconstruction of Distant Scenes from Wide Baseline Images

Planar Reconstruction of Indoor Scenes from Sparse Views and Relative Camera Poses

Outdoor Scene 3D Reconstruction from Multiple Point Cloud

A Novel Method for Detecting Occluded Object by Multiple Camera Arrays.

Projective reconstruction with occlusions

3D Reconstruction Approach for Outdoor Scene Based on Multiple Point Cloud Fusion

Reconstruction of Indoor Scene from A Single Image

Simultaneous Active Camera Array Focus Plane Estimation and Occluded Moving Object Imaging.

Occluded Object Imaging Via Optimal Camera Selection.

Occluded Object Imaging Based on Collaborative Synthetic Aperture Photography

Algorithm for Projective Reconstruction with Occlusions

Scene Reconstruction Based on View Clustering via Camera Auxiliary Information

Synthetic Aperture Photography Using a Moving Camera-Imu System.

Efficient 3D Reconstruction Using Monocular Vision

An Automatic Occlusion Handling Method in Augmented Reality

Real-time indoor scene reconstruction with Manhattan assumption

Rapid and Robust Piecewise Planar Reconstruction of Urban Scenes