Telepresence augmentation for visual and haptic guided immersive teleoperation of industrial manipulator
Fanghao Huang,Xiao Yang,Tuoyu Yan,Zheng Chen
DOI: https://doi.org/10.1016/j.isatra.2024.05.003
IF: 7.3
2024-05-10
ISA Transactions
Abstract:Highlights • A visual-based telepresence augmentation method to offer real-time additional depth information. • Provide fusion haptic force based on visual information, including the virtual guiding force, virtual repulsive force and real-time interactive force. • Visual/haptic integrated perception and reconstruction system is proposed for immersive telepresence, with consideration of heterogeneous characteristics and precise control. • A user study with a grab-place task that commonly appeared in industry is implemented to verify the effectiveness of the proposed system. Teleoperation under human guidance has become an effective solution to extend human's reach in various environments. However, the teleoperation system still faces challenges of insufficient sense of both visual and haptic feedback from remote environments, which results in the inadequate guidance for the operator. In this paper, a visual/haptic integrated perception and reconstruction system (VHI-PRS) is developed to provide the operator with 3D visual information and effective haptic guidance. Specifically, a visual-based telepresence augmentation method is proposed to provide the operator with virtual-reality combined visual telepresence, where the real point cloud model is directly superimposed on virtual manipulator to avoid the time-consuming process of mesh model rendering. With the utilization of visual information, a haptic-based telepresence augmentation method is proposed to provide the operator with comprehensive force feedback, including the virtual guiding force, virtual repulsive force and real-time interactive force, which greatly helps reduce the workload of operator. Finally, a user study on grab-place task is carried out to verify the effectiveness of proposed system.
automation & control systems,instruments & instrumentation,engineering, multidisciplinary