Abstract:Human-robot collaborative assembly (HRCA) is one of the current trends of intelligent manufacturing, and assembly action recognition is the basis of and the key to HRCA. A multi-scale and multi-stream graph convolutional network (2MSGCN) for assembly action recognition is proposed in this paper. 2MSGCN takes the temporal skeleton sample as input and outputs the class of the assembly action to which the sample belongs. RGBD images of the operator performing the assembly actions are captured by three RGBD cameras mounted at different viewpoints and pre-processed to generate the complete human skeleton. A multi-scale and multi-stream (2MS) mechanism and a feature fusion mechanism are proposed to improve the recognition accuracy of 2MSGCN. The 2MS mechanism is designed to input the skeleton data to 2MSGCN in the form of a joint stream, a bone stream and a motion stream, while the joint stream further generates two sets of input with rough scales to represent features in higher dimensional human skeleton, which obtains information of different scales and streams in temporal skeleton samples. And the feature fusion mechanism enables the fused feature to retain the information of the sub-feature while incorporating union information between the sub-features. Also, the improved convolution operation based on Ghost module is introduced to the 2MSGCN to reduce the number of the parameters and floating-point operations (FLOPs) and improve the real-time performance. Considering that there will be transitional actions when the operator switches between assembly actions in the continuous assembly process, a transitional action classification (TAC) method is proposed to distinguish the transitional actions from the assembly actions. Experiments on the public dataset NTU RGB+D +D 60 (NTU 60) and a self-built assembly action dataset indicate that the proposed 2MSGCN outperforms the mainstream models in recognition accuracy and real-time performance.

Unsupervised exceptional human action detection from repetition of human assembling tasks using entropy signal clustering

Based on cluster tree human action recognition algorithm for monocular video

Human Action Recognition Using Deep Learning Methods.

A Skeleton-Based Assembly Action Recognition Method with Feature Fusion for Human-Robot Collaborative Assembly

Online Robust Action Recognition Based on a Hierarchical Model

Memory Clustering Autoencoder Method for Human Action Anomaly Detection on Surveillance Camera Video

HAR-time: human action recognition with time factor analysis on worker operating time

Human Action Recognition Based on Three-Stream Network with Frame Sequence Features

Human Action Recognition with Contextual Constraints Using a RGB-D Sensor

Human Action Recognition From Digital Videos Based on Deep Learning.

Action Recognition by Exploring Data Distribution and Feature Correlation

Combining Sparse And Dense Descriptors With Temporal Semantic Structures For Robust Human Action Recognition

Elastic sequence correlation for human action analysis

Hybrid machine learning for human action recognition and prediction in assembly

A Matrix-Based Approach to Unsupervised Human Action Categorization

Unsupervised Temporal Segmentation of Repetitive Human Actions Based on Kinematic Modeling and Frequency Analysis

Unmanned aerial vehicles for human detection and recognition using neural-network model

Deep Learning-Based Human Action Recognition in Videos

Action Recognition for Human–Robot Teaming: Exploring Mutual Performance Monitoring Possibilities

Representing Videos As Discriminative Sub-graphs for Action Recognition*

Human Activity Recognition based on Dynamic Spatio-Temporal Relations