Abstract:Purpose Assembly action recognition plays an important role in assembly process monitoring and human-robot collaborative assembly. Previous works overlook the interaction relationship between hands and operated objects and lack the modeling of subtle hand motions, which leads to a decline in accuracy for fine-grained action recognition. This paper aims to model the hand-object interactions and hand movements to realize high-accuracy assembly action recognition. Design/methodology/approach In this paper, a novel multi-stream hand-object interaction network (MHOINet) is proposed for assembly action recognition. To learn the hand-object interaction relationship in assembly sequence, an interaction modeling network (IMN) comprising both geometric and visual modeling is exploited in the interaction stream. The former captures the spatial location relation of hand and interacted parts/tools according to their detected bounding boxes, and the latter focuses on mining the visual context of hand and object at pixel level through a position attention model. To model the hand movements, a temporal enhancement module (TEM) with multiple convolution kernels is developed in the hand stream, which captures the temporal dependences of hand sequences in short and long ranges. Finally, assembly action prediction is accomplished by merging the outputs of different streams through a weighted score-level fusion. A robotic arm component assembly dataset is created to evaluate the effectiveness of the proposed method. Findings The method can achieve the recognition accuracy of 97.31% and 95.32% for coarse and fine assembly actions, which outperforms other comparative methods. Experiments on human-robot collaboration prove that our method can be applied to industrial production. Originality/value The author proposes a novel framework for assembly action recognition, which simultaneously leverages the features of hands, objects and hand-object interactions. The TEM enhances the representation of dynamics of hands and facilitates the recognition of assembly actions with various time spans. The IMN learns the semantic information from hand-object interactions, which is significant for distinguishing fine assembly actions.

3D Modelling of a Frame Assembly Using Deep Learning and the Chu–Liu–Edmonds Algorithm

Assembly Sequence Generation for New Objects Via Experience Learned from Similar Object

A Surrogate Modeling Framework for Aircraft Assembly Deformation Using Triplet Attention-Enhanced Conditional Autoencoder

Learning-Based Hand Motion Capture and Understanding in Assembly Process

A Modeling Method of Complex Assembly Based on Digital Twin

Monitoring of Assembly Process Using Deep Learning Technology.

Realistic Geometry Based Feature Modeling of Complex Part and Its Application in Assembly Quality Analysis

Study on Generation of 3D Assembly Dimension Chain

Automatic Generation of 3D Assembly Dimension Chain Based on Feature Model

Deep learning aided inverse design of the buckling-guided assembly for 3D frame structures

A Digital Twin-Driven Guidance Method for Human-Machine Collaborative Assembly Operations Based on Machine Learning and Computer Vision

A Novel Multi-Stream Hand-Object Interaction Network for Assembly Action Recognition

Learning Part Generation and Assembly for Sketching Man‐Made Objects

Digital twin-driven multi-dimensional assembly error modeling and control for complex assembly process in Industry 4.0

Generative 3D Part Assembly via Dynamic Graph Learning

Research on intelligent auxiliary assembly technology based on deep learning

Domain adaptive segmentation method for mechanical assembly based on iterative loops

Category-Level Multi-Part Multi-Joint 3D Shape Assembly

A flexible manufacturing assembly system with deep reinforcement learning

Research on Intelligent Assembly Strategy and Workpiece Grasping Method for Industrial Robots Based on Deep Learning

3D Part Assembly Generation With Instance Encoded Transformer