Abstract:Human-robot collaborative assembly (HRCA) is one of the current trends of intelligent manufacturing, and assembly action recognition is the basis of and the key to HRCA. A multi-scale and multi-stream graph convolutional network (2MSGCN) for assembly action recognition is proposed in this paper. 2MSGCN takes the temporal skeleton sample as input and outputs the class of the assembly action to which the sample belongs. RGBD images of the operator performing the assembly actions are captured by three RGBD cameras mounted at different viewpoints and pre-processed to generate the complete human skeleton. A multi-scale and multi-stream (2MS) mechanism and a feature fusion mechanism are proposed to improve the recognition accuracy of 2MSGCN. The 2MS mechanism is designed to input the skeleton data to 2MSGCN in the form of a joint stream, a bone stream and a motion stream, while the joint stream further generates two sets of input with rough scales to represent features in higher dimensional human skeleton, which obtains information of different scales and streams in temporal skeleton samples. And the feature fusion mechanism enables the fused feature to retain the information of the sub-feature while incorporating union information between the sub-features. Also, the improved convolution operation based on Ghost module is introduced to the 2MSGCN to reduce the number of the parameters and floating-point operations (FLOPs) and improve the real-time performance. Considering that there will be transitional actions when the operator switches between assembly actions in the continuous assembly process, a transitional action classification (TAC) method is proposed to distinguish the transitional actions from the assembly actions. Experiments on the public dataset NTU RGB+D +D 60 (NTU 60) and a self-built assembly action dataset indicate that the proposed 2MSGCN outperforms the mainstream models in recognition accuracy and real-time performance.

Multi-Feature Fusion Real-Time Action Recognition Based on 2D to 3D Skeleton

Action Recognition Based on 3D Skeleton and RGB Frame Fusion

Multi-source Learning for Skeleton -Based Action Recognition Using Deep LSTM Networks

Skeleton-Based Multifeatures and Multistream Network for Real-Time Action Recognition

Skeleton Sequence and RGB Frame Based Multi-Modality Feature Fusion Network for Action Recognition

A Skeleton-Based Assembly Action Recognition Method with Feature Fusion for Human-Robot Collaborative Assembly

Accurate And Real-Time Human Action Recognition Based On 3d Skeleton

Skeleton-Based Human Action Recognition Using Spatial Temporal 3D Convolutional Neural Networks

3D Action Recognition Using Multi-Temporal Skeleton Visualization.

Skeleton Feature Fusion Based on Multi-Stream LSTM for Action Recognition.

Two-Stream 3D Convolutional Neural Network for Skeleton-Based Action Recognition

Skeleton-Indexed Deep Multi-Modal Feature Learning for High Performance Human Action Recognition

A New Representation of Skeleton Sequences for 3D Action Recognition

Multisource Learning for Skeleton-Based Action Recognition Using Deep LSTM and CNN

Skeleton-based Action Recognition Using LSTM and CNN

Attention-Based Multiview Re-Observation Fusion Network for Skeletal Action Recognition.

Multi-Modality Adaptive Feature Fusion Graph Convolutional Network for Skeleton-Based Action Recognition

Channel attention and multi-scale graph neural networks for skeleton-based action recognition

A Fine-to-Coarse Convolutional Neural Network for 3D Human Action Recognition

Symmetrical Enhanced Fusion Network for Skeleton-Based Action Recognition

Multi‐temporal scale aggregation refinement graph convolutional network for skeleton‐based action recognition