Abstract:Human-robot collaborative assembly (HRCA) is one of the current trends of intelligent manufacturing, and assembly action recognition is the basis of and the key to HRCA. A multi-scale and multi-stream graph convolutional network (2MSGCN) for assembly action recognition is proposed in this paper. 2MSGCN takes the temporal skeleton sample as input and outputs the class of the assembly action to which the sample belongs. RGBD images of the operator performing the assembly actions are captured by three RGBD cameras mounted at different viewpoints and pre-processed to generate the complete human skeleton. A multi-scale and multi-stream (2MS) mechanism and a feature fusion mechanism are proposed to improve the recognition accuracy of 2MSGCN. The 2MS mechanism is designed to input the skeleton data to 2MSGCN in the form of a joint stream, a bone stream and a motion stream, while the joint stream further generates two sets of input with rough scales to represent features in higher dimensional human skeleton, which obtains information of different scales and streams in temporal skeleton samples. And the feature fusion mechanism enables the fused feature to retain the information of the sub-feature while incorporating union information between the sub-features. Also, the improved convolution operation based on Ghost module is introduced to the 2MSGCN to reduce the number of the parameters and floating-point operations (FLOPs) and improve the real-time performance. Considering that there will be transitional actions when the operator switches between assembly actions in the continuous assembly process, a transitional action classification (TAC) method is proposed to distinguish the transitional actions from the assembly actions. Experiments on the public dataset NTU RGB+D +D 60 (NTU 60) and a self-built assembly action dataset indicate that the proposed 2MSGCN outperforms the mainstream models in recognition accuracy and real-time performance.

Equipment Assembly Recognition for Augmented Reality Guidance

AR-Assisted Guidance for Assembly and Maintenance of Avionics Equipment

Learning-Based Hand Motion Capture and Understanding in Assembly Process

Deep learning-based augmented reality work instruction assistance system for complex manual assembly

Guided assembly using deep learning-based object recognition and augmented reality instructions

AR-assisted assembly method based on instance segmentation

A posture detection method for augmented reality–aided assembly based on YOLO-6D

A Novel Method of Multitarget Augmented Reality Assembly Result Inspection for Large Complex Scenes

Development and Application of Parts Assembly Guidance System Based on Augmented Reality

Research and Implementation on Key Technologies of Assembly Assistance System based on Augmented Reality

Integrated Registration and Occlusion Handling Based on Deep Learning for Augmented Reality Assisted Assembly Instruction

Efficient evaluation of misalignment between real and virtual objects for HMD-Based AR assembly assistance system

Rapid Offline Detection and 3D Annotation of Assembly Elements in the Augmented Assembly

Real-time occlusion handling for augmented reality assistance assembly systems with monocular images

Multi-3D-Models Registration-Based Augmented Reality (AR) Instructions for Assembly

A Skeleton-Based Assembly Action Recognition Method with Feature Fusion for Human-Robot Collaborative Assembly

Augmented Reality Assisted Assembly Training Oriented Dynamic Gesture Recognition and Prediction

A natural bare-hand interface-enabled interactive AR assembly guidance

Active Assembly Guidance with Online Video Parsing

A Digital Twin-Driven Guidance Method for Human-Machine Collaborative Assembly Operations Based on Machine Learning and Computer Vision

A 3D Tracking and Registration Method Based on Point Cloud and Visual Features for Augmented Reality Aided Assembly System