Abstract:In this work, we present a reconfigurable data glove design to capture different modes of human hand-object interactions, which are critical in training embodied artificial intelligence (AI) agents for fine manipulation tasks. To achieve various downstream tasks with distinct features, our reconfigurable data glove operates in three modes sharing a unified backbone design that reconstructs hand gestures in real time. In the tactile-sensing mode, the glove system aggregates manipulation force via customized force sensors made from a soft and thin piezoresistive material; this design minimizes interference during complex hand movements. The virtual reality (VR) mode enables real-time interaction in a physically plausible fashion: A caging-based approach is devised to determine stable grasps by detecting collision events. Leveraging a state-of-the-art finite element method (FEM), the simulation mode collects data on fine-grained 4D manipulation events comprising hand and object motions in 3D space and how the object's physical properties (e.g., stress and energy) change in accordance with manipulation over time. Notably, the glove system presented here is the first to use high-fidelity simulation to investigate the unobservable physical and causal factors behind manipulation actions. In a series of experiments, we characterize our data glove in terms of individual sensors and the overall system. More specifically, we evaluate the system's three modes by (i) recording hand gestures and associated forces, (ii) improving manipulation fluency in VR, and (iii) producing realistic simulation effects of various tool uses, respectively. Based on these three modes, our reconfigurable data glove collects and reconstructs fine-grained human grasp data in both physical and virtual environments, thereby opening up new avenues for the learning of manipulation skills for embodied AI agents.

Robust vision-based glove pose estimation for both hands in virtual reality

2D Motion Detection Bounded Hand 3D Trajectory Tracking and Gesture Recognition under Complex Background

Advancing Virtual Reality Interaction: A Ring-Shaped Controller and Pose Tracking

High-Fidelity Grasping In Virtual Reality Using A Glove-Based System

A Sensor Glove Based on Inertial Measurement Unit for Robot Teleoperetion.

Multi-modal Sensing-Based Interactive Glove System for Teleoperation and VR/AR

Interaction and control with the auxiliary of hand gesture

Realistic Virtual Hand Modeling with Applications for Virtual Grasping

Dual Quaternion Based Virtual Hand Interaction Modeling

Soft Robotic Glove with Sensing and Force Feedback for Rehabilitation in Virtual Reality

UltraGlove: Hand Pose Estimation with Mems-Ultrasonic Sensors

Stretchable glove for accurate and robust hand pose reconstruction based on comprehensive motion data

Lightweight Soft Robotic Glove with Whole-Hand Finger Motion Tracking for Hand Rehabilitation in Virtual Reality

3D Hand Motion Generation for VR Interactions Using a Haptic Data Glove

A Reconfigurable Data Glove for Reconstructing Physical and Virtual Grasps

Evaluation of Machine Learning Techniques for Hand Pose Estimation on Handheld Device with Proximity Sensor

A Novel Sensor Fusion Approach for Precise Hand Tracking in Virtual Reality-Based Human—Computer Interaction

Towards natural interaction in immersive reality with a cyber-glove

HaptGlove—Untethered Pneumatic Glove for Multimode Haptic Feedback in Reality–Virtuality Continuum

Visual-inertial hand motion tracking with robustness against occlusion, interference, and contact

VR-HandNet: A Visually and Physically Plausible Hand Manipulation System in Virtual Reality