Abstract:The application of Augmented Reality (AR) can effectively reduce the cognitive load of operators by presenting complex assembly work instruction with virtual augmented contents. If the virtual augmented contents superimposed on physical product without handling occlusion, the distortion may appear in AR guidance video. This paper proposes a monocular image based real-time occlusion handling method for virtual part model and physical product. In AR assistance assembly process, the virtual part model can be occluded by assembly scene or assembly objects. First, the monocular SLAM is used to reconstruct the assembly scene, and the reconstructed sparse 3D points are converted into depth points in the depth map. We use the color information of the assembly scene to control the propagation of parse depth points to the remaining pixels. With GPU acceleration calculation, a densification depth map of the assembly scene can be obtained. Then, the 3D models of the assembly objects are rendered as depth map according to the registration coordinates in AR assistance assembly system. The depth map of the assembly scene and assembly objects are synthesized as the final depth map for AR occlusion handling. Finally, the depth relationship between real assembly scenes, assembly objects, and virtual augmented contents are compared to determine the display pixels for AR effect. The results show that the proposed occlusion handling method shows high accuracy and fast speed in comparison with conventional methods, which demonstrates that our method can effectively solve the problem of occlusion handing for AR assistance assembly systems.

Real-time Hand-object Occlusion for Augmented Reality Using Hand Segmentation and Depth Correction

In-Hand 3D Object Reconstruction from a Monocular RGB Video

Handling Occlusions in Video-Based Augmented Reality Using Depth Information

Resolving hand‐object occlusion for mixed reality with joint deep learning and model optimization

GrabAR: Occlusion-aware Grabbing Virtual Objects in AR

CAMInterHand: Cooperative Attention for Multi-View Interactive Hand Pose and Mesh Reconstruction

2D Motion Detection Bounded Hand 3D Trajectory Tracking and Gesture Recognition under Complex Background

A touchable virtual screen interaction system with handheld Kinect camera

Real-time occlusion handling for augmented reality assistance assembly systems with monocular images

Single Depth View Based Real-Time Reconstruction of Hand-Object Interactions

Hand Segmentation for Hand-Object Interaction from Depth map

Depth Restoration of Hand-Held Transparent Objects for Human-to-Robot Handover

Dynamic Occlusion Handling for Real-Time AR Applications

HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network

HandGCAT: Occlusion-Robust 3D Hand Mesh Reconstruction from Monocular Images.

Real-Time Hand Locating By Monocular Vision

The Effects of Incorrect Occlusion Cues on the Understanding of Barehanded Referencing in Collaborative Augmented Reality

Hand recognition method based on color and depth information from first-person view

Integrated Registration and Occlusion Handling Based on Deep Learning for Augmented Reality Assisted Assembly Instruction

Point Cloud Hand–Object Segmentation Using Multimodal Imaging with Thermal and Color Data for Safe Robotic Object Handover

3D Hand Mesh Recovery from Monocular RGB in Camera Space