Abstract:Augmented reality (AR) can benefit manual assembly operations by superimposing intuitive visual guidance on the workbench directly, alleviating the mental load to determine the next operation while improving assembly performance. Where the bare-hand gesture interface is considered a promising manner to manipulate the virtual assembly model, nevertheless, it usually suffers from unreliable recognition results and poor adaptability due to the unfamiliarity with gesture libraries in actual AR assembly tasks. In this paper, we try to establish a robust and easy-to-use natural bare-hand interactive AR assembly system, enabling the operator to interact against the ongoing virtual assembly models intuitively and resulting in a human-centric AR assembly performance. Firstly, a normalized feature information extraction for real-time bare-hand gestures is presented, which can establish a robust gesture model expression when encountering different sensing distances and varying operators. Followed by the definition of some typical bare-hand interaction functions related to manual assembly operations, and thus reliable interactions between hand gestures and virtual components are available. Then, a bare-hand-enabled interactive AR assembly system is established, where operators can explore and manipulate a large amount of assembly-related information with the help of real-time bare-hand interaction. Finally, extensive experiments are carried out to evaluate the proposed bare-hand interactive AR assembly system, and results demonstrate that it can alleviate the cognitive load for the operators while providing a human-centric assembly performance.

A multimodal interface to situated assembly robot systems

Situated Multimodal Control of a Mobile Robot: Navigation through a Virtual Environment

Exploitation of Multiplayer Interaction and Development of Virtual Puppetry Storytelling Using Gesture Control and Stereoscopic Devices.

Perception of Demonstration for Automatic Programing of Robotic Assembly: Framework, Algorithm, and Validation

MIVAS: A Multi-Modal Immersive Virtual Assembly System

Multi-Modal Human-Machine Communication for Instructing Robot Grasping Tasks

Enhancing Human–Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition

Function Block-Based Multimodal Control for Symbiotic Human–Robot Collaborative Assembly

Interaction With Gaze, Gesture, and Speech in a Flexibly Configurable Augmented Reality System

A multi-modal augmented reality based virtual assembly system

Gaze-Based Intention Recognition for Human-Robot Collaboration

A Multimodal Emotional Communication Based Humans-Robots Interaction System

A Multi‐Module Sensing and Bi‐Directional HMI Integrating Interaction, Recognition, and Feedback for Intelligent Robots

Projected augmented reality assembly assistance system supporting multi-modal interaction

Communicating human intent to a robotic companion by multi-type gesture sentences

Robi Butler: Remote Multimodal Interactions with Household Robot Assistant

Teaching robots to do object assembly using multi-modal 3D vision

Towards seamless human robot collaboration: integrating multimodal interaction

A Natural Bare-Hand Interaction Method With Augmented Reality for Constraint-Based Virtual Assembly

A natural bare-hand interface-enabled interactive AR assembly guidance

HRC for dual-robot intelligent assembly system based on multimodal perception