Abstract:Purpose Assembly action recognition plays an important role in assembly process monitoring and human-robot collaborative assembly. Previous works overlook the interaction relationship between hands and operated objects and lack the modeling of subtle hand motions, which leads to a decline in accuracy for fine-grained action recognition. This paper aims to model the hand-object interactions and hand movements to realize high-accuracy assembly action recognition. Design/methodology/approach In this paper, a novel multi-stream hand-object interaction network (MHOINet) is proposed for assembly action recognition. To learn the hand-object interaction relationship in assembly sequence, an interaction modeling network (IMN) comprising both geometric and visual modeling is exploited in the interaction stream. The former captures the spatial location relation of hand and interacted parts/tools according to their detected bounding boxes, and the latter focuses on mining the visual context of hand and object at pixel level through a position attention model. To model the hand movements, a temporal enhancement module (TEM) with multiple convolution kernels is developed in the hand stream, which captures the temporal dependences of hand sequences in short and long ranges. Finally, assembly action prediction is accomplished by merging the outputs of different streams through a weighted score-level fusion. A robotic arm component assembly dataset is created to evaluate the effectiveness of the proposed method. Findings The method can achieve the recognition accuracy of 97.31% and 95.32% for coarse and fine assembly actions, which outperforms other comparative methods. Experiments on human-robot collaboration prove that our method can be applied to industrial production. Originality/value The author proposes a novel framework for assembly action recognition, which simultaneously leverages the features of hands, objects and hand-object interactions. The TEM enhances the representation of dynamics of hands and facilitates the recognition of assembly actions with various time spans. The IMN learns the semantic information from hand-object interactions, which is significant for distinguishing fine assembly actions.

A Skeleton-Based Assembly Action Recognition Method with Feature Fusion for Human-Robot Collaborative Assembly

Assembly Action Recognition Based on Dual Stream Fusion of Skeleton and Video Data

Hybrid Convolutional Neural Network Approaches for Recognizing Collaborative Actions in Human–Robot Assembly Tasks

Action Fusion Recognition Model Based on GAT-GRU Binary Classification Networks for Human-Robot Collaborative Assembly

A Novel Multi-Stream Hand-Object Interaction Network for Assembly Action Recognition

Online Robust Action Recognition Based on a Hierarchical Model

Skeleton-based Action Recognition with Multi-Stream, Multi-Scale Dilated Spatial-Temporal Graph Convolution Network

Human-Robot Collaboration Through a Multi-Scale Graph Convolution Neural Network With Temporal Attention

Human-centric multimodal fusion network for robust action recognition

Skeleton-based Human Action Recognition by Fusing Attention Based Three-Stream Convolutional Neural Network and SVM.

Data-efficient Multimodal Human Action Recognition for Proactive Human–robot Collaborative Assembly: A Cross-Domain Few-Shot Learning Approach

Human-object integrated assembly intention recognition for context-aware human-robot collaborative assembly

Joints-Centered Spatial-Temporal Features Fused Skeleton Convolution Network for Action Recognition

Hybrid machine learning for human action recognition and prediction in assembly

Multi‐temporal scale aggregation refinement graph convolutional network for skeleton‐based action recognition

A Lightweight Hierarchical Model with Frame-Level Joints Adaptive Graph Convolution for Skeleton- Based Action Recognition

Channel attention and multi-scale graph neural networks for skeleton-based action recognition

Attention-Based Multilevel Co-Occurrence Graph Convolutional LSTM for 3-D Action Recognition

HDBN: A Novel Hybrid Dual-branch Network for Robust Skeleton-based Action Recognition

Adaptive Channel-Enhanced Graph Convolution for Skeleton-Based Human Action Recognition

Skeleton-based Action Recognition with Multi-stream, Temporal-Channel Enhanced Graph Convolution Network