A Monocular-Based Tracking Framework for Industrial Augmented Reality Applications

Xu Yang,Junqi Cai,Kunbo Li,Xiumin Fan,Hengling Cao

DOI: https://doi.org/10.1007/s00170-023-12082-w

2023-01-01

Abstract:Augmented reality tracking is the core problem of augmented reality, which has not been well resolved so far, and limits the wider application of industrial augmented reality in actual industrial scenarios to a certain extent. In this paper, we propose a general pose estimation, refinement, and tracking framework for industrial augmented reality system. First, we propose a deep convolutional neural network for texture-less object pose estimation at first look. And a synthetic data generation pipeline is utilized to synthesize photorealistic training images for the network. Then, we implement the metric initialization of a monocular visual simultaneous localization and mapping (SLAM) through object pose, and the real-time camera pose is continuously tracked by the monocular visual SLAM. During the tracking process, a back-end pose optimization thread is proposed to calculate the pose residual between the model and real scene, and correct the pose residual in real time. Finally, the entire assembly models are superimposed on the real scene through transformation relationships included in the CAD models. The experiments on a gearbox assembly scenario show that the proposed method gives a highest tracking accuracy in compared with a marker-based methods and two state-of-the-art CNN-based object pose estimation methods. The whole proposed pipeline runs at 25 FPS for real-time tracking. Moreover, the proposed method is also applied to a shipbuilding outfitting construction scenario, and shows good scalability and generality for the complex and large-scale industrial scenario and different mobile terminals.