Abstract:Purpose Assembly action recognition plays an important role in assembly process monitoring and human-robot collaborative assembly. Previous works overlook the interaction relationship between hands and operated objects and lack the modeling of subtle hand motions, which leads to a decline in accuracy for fine-grained action recognition. This paper aims to model the hand-object interactions and hand movements to realize high-accuracy assembly action recognition. Design/methodology/approach In this paper, a novel multi-stream hand-object interaction network (MHOINet) is proposed for assembly action recognition. To learn the hand-object interaction relationship in assembly sequence, an interaction modeling network (IMN) comprising both geometric and visual modeling is exploited in the interaction stream. The former captures the spatial location relation of hand and interacted parts/tools according to their detected bounding boxes, and the latter focuses on mining the visual context of hand and object at pixel level through a position attention model. To model the hand movements, a temporal enhancement module (TEM) with multiple convolution kernels is developed in the hand stream, which captures the temporal dependences of hand sequences in short and long ranges. Finally, assembly action prediction is accomplished by merging the outputs of different streams through a weighted score-level fusion. A robotic arm component assembly dataset is created to evaluate the effectiveness of the proposed method. Findings The method can achieve the recognition accuracy of 97.31% and 95.32% for coarse and fine assembly actions, which outperforms other comparative methods. Experiments on human-robot collaboration prove that our method can be applied to industrial production. Originality/value The author proposes a novel framework for assembly action recognition, which simultaneously leverages the features of hands, objects and hand-object interactions. The TEM enhances the representation of dynamics of hands and facilitates the recognition of assembly actions with various time spans. The IMN learns the semantic information from hand-object interactions, which is significant for distinguishing fine assembly actions.

A multimedia art interaction model for motion recognition based on data-driven model

Learning a Deep Motion Interpolation Network for Human Skeleton Animations

The Design and Implementation of Natural Human-Robot Interaction System Based on Kinect Sensor

A High‐Precision Dynamic Movement Recognition Algorithm Using Multimodal Biological Signals for Human–Machine Interaction

A Motion-based User Interface for the Control of Virtual Humans Performing Sports.

Construction of Human Digital Twin Model Based on Multimodal Data and Its Application in Locomotion Mode Identification

An Attention-Based Deep Learning Approach for Inertial Motion Recognition and Estimation in Human-Robot Collaboration

A Novel Multi-Stream Hand-Object Interaction Network for Assembly Action Recognition

Multimodal Sensor Motion Intention Recognition Based on Three-Dimensional Convolutional Neural Network Algorithm

A motion recognition model for upper-limb rehabilitation exercises

Human Motion Pattern Recognition and Feature Extraction: An Approach Using Multi-Information Fusion

Human motion intention recognition method with visual, audio, and surface electromyography modalities for a mechanical hand in different environments

Machine Learning-Based Human Motion Recognition via Wearable plastic Fiber Sensing System

Research on the Human Motion Recognition Method Based on Wearable

Research on Multimodal Fusion Recognition Method of Upper Limb Motion Patterns

WMNN: Wearables-Based Multi-Column Neural Network for Human Activity Recognition

High Efficient LSTM-based Network for Human Interaction Understanding

A Multi-Scale and Multi-Stage Human Pose Recognition Method Based on Convolutional Neural Networks for Non-Wearable Ergonomic Evaluation

Motion intensity modeling and trajectory control of upper limb rehabilitation exoskeleton robot based on multi-modal information

Visualization of movements in sports training based on multimedia information processing technology

Leveraging Digital Perceptual Technologies for Remote Perception and Analysis of Human Biomechanical Processes: A Contactless Approach for Workload and Joint Force Assessment