Abstract:Human-robot collaborative assembly (HRCA) is one of the current trends of intelligent manufacturing, and assembly action recognition is the basis of and the key to HRCA. A multi-scale and multi-stream graph convolutional network (2MSGCN) for assembly action recognition is proposed in this paper. 2MSGCN takes the temporal skeleton sample as input and outputs the class of the assembly action to which the sample belongs. RGBD images of the operator performing the assembly actions are captured by three RGBD cameras mounted at different viewpoints and pre-processed to generate the complete human skeleton. A multi-scale and multi-stream (2MS) mechanism and a feature fusion mechanism are proposed to improve the recognition accuracy of 2MSGCN. The 2MS mechanism is designed to input the skeleton data to 2MSGCN in the form of a joint stream, a bone stream and a motion stream, while the joint stream further generates two sets of input with rough scales to represent features in higher dimensional human skeleton, which obtains information of different scales and streams in temporal skeleton samples. And the feature fusion mechanism enables the fused feature to retain the information of the sub-feature while incorporating union information between the sub-features. Also, the improved convolution operation based on Ghost module is introduced to the 2MSGCN to reduce the number of the parameters and floating-point operations (FLOPs) and improve the real-time performance. Considering that there will be transitional actions when the operator switches between assembly actions in the continuous assembly process, a transitional action classification (TAC) method is proposed to distinguish the transitional actions from the assembly actions. Experiments on the public dataset NTU RGB+D +D 60 (NTU 60) and a self-built assembly action dataset indicate that the proposed 2MSGCN outperforms the mainstream models in recognition accuracy and real-time performance.

A Real-Time and Hardware-Efficient Processor for Skeleton-Based Action Recognition with Lightweight Convolutional Neural Network

Cloud-Edge Collaborative Human Activity Recognition System Design and Implementation.

Deploying human activity recognition in embedded RISC-V processors

A Neural Network-Based ECG Classification Processor with Exploitation of Heartbeat Similarity

MC-LSTM: Real-Time 3D Human Action Detection System for Intelligent Healthcare Applications

A Skeleton-based Action Recognition System for Medical Condition Detection

Design and Hardware Implementation of CNN-GCN Model for Skeleton-Based Human Action Recognition

An Energy-Efficient Convolutional Neural Network Processor Architecture Based on a Systolic Array

A 3.89-Gops/mw Scalable Recurrent Neural Network Processor with Improved Efficiency on Memory and Computation

A 0.99-to-4.38 Uj/class Event-Driven Hybrid Neural Network Processor for Full-Spectrum Neural Signal Analyses

A Skeleton-Based Assembly Action Recognition Method with Feature Fusion for Human-Robot Collaborative Assembly

A Novel Instruction Driven 1-D CNN Processor for ECG Classification

An Ultra-low Power TinyML System for Real-time Visual Processing at Edge

Low-Power and Low-Cost AI Processor with Distributed-Aggregated Classification Architecture for Wearable Epilepsy Seizure Detection

Dynamic Edge Convolutional Neural Network for Skeleton-Based Human Action Recognition

Simulation for a Mems-Based CTRNN Ultra-Low Power Implementation of Human Activity Recognition

Real-time Human Activity Recognition Using Conditionally Parametrized Convolutions on Mobile and Wearable Devices

Human Activity Recognition on Microcontrollers with Quantized and Adaptive Deep Neural Networks

An Energy-Efficient Deep Belief Network Processor Based on Heterogeneous Multi-Core Architecture With Transposable Memory and On-Chip Learning

Action Recognition Based on Temporal Skeleton Analysis: a New Method for IoT Scenarios

Two-stage Human Activity Recognition on Microcontrollers with Decision Trees and CNNs