Abstract:Purpose Aiming at presenting an interaction-free assembly assistance tool, the purpose of this paper is to propose a synchronous augmented reality (AR) assembly assistance and monitoring system. The system monitors operator’s hands activity and process completeness to recognize the assembly state, then display the AR contents contextually. Design/methodology/approach An assembly behavior recognition method is proposed based on gesture recognition. An assembly completeness inspection method is proposed based on SURF feature matching. Assembly state and AR display state are solved by a novel sequential hybrid AR display control strategy. A synchronous multi-channel AR view output strategy is proposed based on QR matrix decomposition. Findings A prototype system has been developed, and case study is performed on an industrial product. Experiments are performed to verify the feasibility, efficiency and recognition accuracy of the proposed methods. Research limitations/implications The proposed system assists users to perform assembly tasks with automatic visual guidance and vision monitoring, avoiding distractions caused by redundant human–computer interactions. Practical implications All methods are integrated to work on only one head-worn device, making the proposed system portable and cheaper. The vision processing pipelines and the view output channels are reconfigurable for customization. Originality/value This paper proposes an interaction-free AR assembly assistance and monitoring system. Assembly behavior recognition and assembly completeness inspection methods are integrated to monitor the assembly state. A sequential hybrid AR display control strategy is proposed to contextually update the AR contents. A synchronous multi-channel AR view output strategy is proposed to fulfill different visualization needs.

An Environment State Perception Method Based on Knowledge Representation in Dual-Arm Robot Assembly Tasks.

Perception of Demonstration for Automatic Programing of Robotic Assembly: Framework, Algorithm, and Validation

Combining Vision Sensing with Knowledge Database for Environmental Modeling in Dual-arm Robot Assembly Task

Towards Learning from Demonstration System for Parts Assembly: A Graph Based Representation for Knowledge

A novel task-oriented framework for dual-arm robotic assembly task

A Robot Assembly Framework with “Perception-Action” Mapping Cognitive Learning

Integrating Knowledge and Augmented Reality: A Novel Framework for Intelligent Assembly Assistance

Probabilistic Graph Based Spatial Assembly Relation Inference for Programming of Assembly Task by Demonstration.

Human–Robot Collaborative Assembly Based on Eye-Hand and a Finite State Machine in a Virtual Environment

Multi-Robot Cross-Station 3D Perception System for Automatic Assembly

Skill learning for robotic assembly based on visual perspectives and force sensing

Assembly-Oriented Task Sequence Planning for a Dual-Arm Robot

Teaching robots to do object assembly using multi-modal 3D vision

State Estimation and Environment Recognition for Articulated Structures via Proximity Sensors Distributed over the Whole Body

Task Representation in Robots for Robust Coupling of Perception to Action in Dynamic Scenes

HRC for dual-robot intelligent assembly system based on multimodal perception

Robotic Fasten Assembly Using Vision and Force Sensing.

A Residual Reinforcement Learning Method for Robotic Assembly Using Visual and Force Information

Synchronous AR Assembly Assistance and Monitoring System Based on Ego-Centric Vision

Manipulation Skill Acquisition for Robotic Assembly Based on Multi-Modal Information Description

Imagination-enabled Robot Perception