Abstract:Purpose Assembly action recognition plays an important role in assembly process monitoring and human-robot collaborative assembly. Previous works overlook the interaction relationship between hands and operated objects and lack the modeling of subtle hand motions, which leads to a decline in accuracy for fine-grained action recognition. This paper aims to model the hand-object interactions and hand movements to realize high-accuracy assembly action recognition. Design/methodology/approach In this paper, a novel multi-stream hand-object interaction network (MHOINet) is proposed for assembly action recognition. To learn the hand-object interaction relationship in assembly sequence, an interaction modeling network (IMN) comprising both geometric and visual modeling is exploited in the interaction stream. The former captures the spatial location relation of hand and interacted parts/tools according to their detected bounding boxes, and the latter focuses on mining the visual context of hand and object at pixel level through a position attention model. To model the hand movements, a temporal enhancement module (TEM) with multiple convolution kernels is developed in the hand stream, which captures the temporal dependences of hand sequences in short and long ranges. Finally, assembly action prediction is accomplished by merging the outputs of different streams through a weighted score-level fusion. A robotic arm component assembly dataset is created to evaluate the effectiveness of the proposed method. Findings The method can achieve the recognition accuracy of 97.31% and 95.32% for coarse and fine assembly actions, which outperforms other comparative methods. Experiments on human-robot collaboration prove that our method can be applied to industrial production. Originality/value The author proposes a novel framework for assembly action recognition, which simultaneously leverages the features of hands, objects and hand-object interactions. The TEM enhances the representation of dynamics of hands and facilitates the recognition of assembly actions with various time spans. The IMN learns the semantic information from hand-object interactions, which is significant for distinguishing fine assembly actions.

A Two-Step Shapelets Based Framework for Interactional Activities Recognition

Hierarchical Complex Activity Representation and Recognition Using Topic Model and Classifier Level Fusion.

Two-person interaction recognition using a two-step sequential pattern classification

Sensor-based Human Activity Recognition System with a Multilayered Model Using Time Series Shapelets

Recognition and Detection of Two-Person Interactive Actions Using Automatically Selected Skeleton Features

Human Interaction Recognition Framework based on Interacting Body Part Attention

Human Interaction Recognition Based on Whole-Individual Detection.

B2C-AFM: Bi-Directional Co-Temporal and Cross-Spatial Attention Fusion Model for Human Action Recognition.

Two-stream Multi-level Dynamic Point Transformer for Two-person Interaction Recognition

A novel two-level interactive action recognition model based on inertial data fusion

A Novel Multi-Stream Hand-Object Interaction Network for Assembly Action Recognition

Recognizing Interaction Activities using Dynamic Bayesian Network

Human Interaction Recognition Method Based on Parallel Multi-Feature Fusion Network

Dezert-Smarandache Theory-Based Fusion for Human Activity Recognition in Body Sensor Networks

An LSTM-Based Approach for Understanding Human Interactions Using Hybrid Feature Descriptors Over Depth Sensors

Two-Stream Modality-Based Deep Learning Approach for Enhanced Two-Person Human Interaction Recognition in Videos

Interaction Relational Network for Mutual Action Recognition

Human Activity Recognition based on Dynamic Spatio-Temporal Relations

Exploiting Human Pose and Scene Information for Interaction Detection

Effective approaches in human action recognition

Indoor Human Action Recognition Based on Dual Kinect V2 and Improved Ensemble Learning Method