Abstract:Purpose Assembly action recognition plays an important role in assembly process monitoring and human-robot collaborative assembly. Previous works overlook the interaction relationship between hands and operated objects and lack the modeling of subtle hand motions, which leads to a decline in accuracy for fine-grained action recognition. This paper aims to model the hand-object interactions and hand movements to realize high-accuracy assembly action recognition. Design/methodology/approach In this paper, a novel multi-stream hand-object interaction network (MHOINet) is proposed for assembly action recognition. To learn the hand-object interaction relationship in assembly sequence, an interaction modeling network (IMN) comprising both geometric and visual modeling is exploited in the interaction stream. The former captures the spatial location relation of hand and interacted parts/tools according to their detected bounding boxes, and the latter focuses on mining the visual context of hand and object at pixel level through a position attention model. To model the hand movements, a temporal enhancement module (TEM) with multiple convolution kernels is developed in the hand stream, which captures the temporal dependences of hand sequences in short and long ranges. Finally, assembly action prediction is accomplished by merging the outputs of different streams through a weighted score-level fusion. A robotic arm component assembly dataset is created to evaluate the effectiveness of the proposed method. Findings The method can achieve the recognition accuracy of 97.31% and 95.32% for coarse and fine assembly actions, which outperforms other comparative methods. Experiments on human-robot collaboration prove that our method can be applied to industrial production. Originality/value The author proposes a novel framework for assembly action recognition, which simultaneously leverages the features of hands, objects and hand-object interactions. The TEM enhances the representation of dynamics of hands and facilitates the recognition of assembly actions with various time spans. The IMN learns the semantic information from hand-object interactions, which is significant for distinguishing fine assembly actions.

Agricultural Machinery Virtual Assembly System Using Dynamic Gesture Recognitive Interaction Based on a CNN and LSTM Network

Learning-Based Hand Motion Capture and Understanding in Assembly Process

Research On Aircraft Virtual Assembly Technology Based On Gesture Recognition

Augmented Reality Assisted Assembly Training Oriented Dynamic Gesture Recognition and Prediction

The Virtual Assembly Technology Based on Natural Gesture

MIVAS: A Multi-Modal Immersive Virtual Assembly System

A Novel Multi-Stream Hand-Object Interaction Network for Assembly Action Recognition

Gesture Classification in Electromyography Signals for Real-Time Prosthetic Hand Control Using a Convolutional Neural Network-Enhanced Channel Attention Model

Mid-Air Gestures for Multi-Fingered Virtual Assembly System with Leap Motion.

Deep learning-based augmented reality work instruction assistance system for complex manual assembly

Gesture-Controlled Robotic Arm for Agricultural Harvesting Using a Data Glove with Bending Sensor and OptiTrack Systems

Hybrid Convolutional Neural Network Approaches for Recognizing Collaborative Actions in Human–Robot Assembly Tasks

Deep Convolutional Neural Network enabled unmanned agricultural machine visual navigation system: architecture design, model optimization and empirical evaluation

Real-time 3D unstructured environment reconstruction utilizing VR and Kinect-based immersive teleoperation for agricultural field robots

HRC of intelligent assembly system based on multimodal gesture control

Immersive Human-Machine Teleoperation Framework for Precision Agriculture: Integrating UAV-based Digital Mapping and Virtual Reality Control

Gesture-driven interaction service system for complex operations in digital twin manufacturing cells

Interaction Control System of Agricultural Machinery Virtual Test

Critical Techniques of Virtual Assembly for Combine Harvester Chassis Based on Virtual Reality

Sequential dynamic gesture recognition controlled poka-yoke system for manual assembly

AN AUTOMATIC INTERACTION METHOD USING PART RECOGNITION BASED ON DEEP NETWORK FOR AUGMENTED REALITY ASSEMBLY GUIDANCE