Abstract:Purpose Assembly action recognition plays an important role in assembly process monitoring and human-robot collaborative assembly. Previous works overlook the interaction relationship between hands and operated objects and lack the modeling of subtle hand motions, which leads to a decline in accuracy for fine-grained action recognition. This paper aims to model the hand-object interactions and hand movements to realize high-accuracy assembly action recognition. Design/methodology/approach In this paper, a novel multi-stream hand-object interaction network (MHOINet) is proposed for assembly action recognition. To learn the hand-object interaction relationship in assembly sequence, an interaction modeling network (IMN) comprising both geometric and visual modeling is exploited in the interaction stream. The former captures the spatial location relation of hand and interacted parts/tools according to their detected bounding boxes, and the latter focuses on mining the visual context of hand and object at pixel level through a position attention model. To model the hand movements, a temporal enhancement module (TEM) with multiple convolution kernels is developed in the hand stream, which captures the temporal dependences of hand sequences in short and long ranges. Finally, assembly action prediction is accomplished by merging the outputs of different streams through a weighted score-level fusion. A robotic arm component assembly dataset is created to evaluate the effectiveness of the proposed method. Findings The method can achieve the recognition accuracy of 97.31% and 95.32% for coarse and fine assembly actions, which outperforms other comparative methods. Experiments on human-robot collaboration prove that our method can be applied to industrial production. Originality/value The author proposes a novel framework for assembly action recognition, which simultaneously leverages the features of hands, objects and hand-object interactions. The TEM enhances the representation of dynamics of hands and facilitates the recognition of assembly actions with various time spans. The IMN learns the semantic information from hand-object interactions, which is significant for distinguishing fine assembly actions.

InteractionFusion

MoreFusion: Multi-object Reasoning for 6D Pose Estimation from Volumetric Fusion

CAMInterHand: Cooperative Attention for Multi-View Interactive Hand Pose and Mesh Reconstruction

Single Depth View Based Real-Time Reconstruction of Hand-Object Interactions

Multi-Level Fusion Net for Hand Pose Estimation in Hand-Object Interaction

Multi-Modal Hand-Object Pose Estimation With Adaptive Fusion and Interaction Learning

A Novel Multi-Stream Hand-Object Interaction Network for Assembly Action Recognition

RobustFusion: Robust Volumetric Performance Reconstruction under Human-object Interactions from Monocular RGBD Stream

Joint Hand-Object 3D Reconstruction From a Single Image With Cross-Branch Feature Fusion

Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wild.

Resolving hand‐object occlusion for mixed reality with joint deep learning and model optimization

Real-Time Hand Gesture Recognition Based on A Fusion Learning Method

Bodyfusion: Real-Time Capture Of Human Motion And Surface Geometry Using A Single Depth Camera

Harmonious Feature Learning for Interactive Hand-Object Pose Estimation

Novel-view Synthesis and Pose Estimation for Hand-Object Interaction from Sparse Views

Fusion of Independent and Interactive Features for Human-Object Interaction Detection

Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Image

TemporalFusion: Temporal Motion Reasoning with Multi-Frame Fusion for 6D Object Pose Estimation

Decoupling Heterogeneous Features for Robust 3D Interacting Hand Poses Estimation

PoseFusion: Robust Object-in-Hand Pose Estimation with SelectLSTM

Human Interaction Recognition Fusing Multiple Features of Depth Sequences