Abstract:In this work, we present a reconfigurable data glove design to capture different modes of human hand-object interactions, which are critical in training embodied artificial intelligence (AI) agents for fine manipulation tasks. To achieve various downstream tasks with distinct features, our reconfigurable data glove operates in three modes sharing a unified backbone design that reconstructs hand gestures in real time. In the tactile-sensing mode, the glove system aggregates manipulation force via customized force sensors made from a soft and thin piezoresistive material; this design minimizes interference during complex hand movements. The virtual reality (VR) mode enables real-time interaction in a physically plausible fashion: A caging-based approach is devised to determine stable grasps by detecting collision events. Leveraging a state-of-the-art finite element method (FEM), the simulation mode collects data on fine-grained 4D manipulation events comprising hand and object motions in 3D space and how the object's physical properties (e.g., stress and energy) change in accordance with manipulation over time. Notably, the glove system presented here is the first to use high-fidelity simulation to investigate the unobservable physical and causal factors behind manipulation actions. In a series of experiments, we characterize our data glove in terms of individual sensors and the overall system. More specifically, we evaluate the system's three modes by (i) recording hand gestures and associated forces, (ii) improving manipulation fluency in VR, and (iii) producing realistic simulation effects of various tool uses, respectively. Based on these three modes, our reconfigurable data glove collects and reconstructs fine-grained human grasp data in both physical and virtual environments, thereby opening up new avenues for the learning of manipulation skills for embodied AI agents.

A Compact Gesture Sensing Glove for Digital Twin of Hand Motion and Robot Teleoperation

A Sensor Glove Based on Inertial Measurement Unit for Robot Teleoperetion.

Stretchable glove for accurate and robust hand pose reconstruction based on comprehensive motion data

A Robust Data-Driven Soft Sensory Glove for Human Hand Motions Identification and Replication

A Reconfigurable Data Glove for Reconstructing Physical and Virtual Grasps

Elastic Tactile Sensor Glove for Dexterous Teaching by Demonstration

Development of Flexible Tactile Sensor for the Envelop of Curved Robotic Hand Finger in Grasping Force Sensing

A Novel Data Glove Using Inertial and Magnetic Sensors for Motion Capture and Robotic Arm-Hand Teleoperation

Multi-modal Sensing-Based Interactive Glove System for Teleoperation and VR/AR

An Intelligent Glove System for Real‐Time Multiple Electrical Signal Acquisition and Display

Development of Robotic Hand Tactile Sensing System for Distributed Contact Force Sensing in Robotic Dexterous Multimodal Grasping

A Glove-Based System For Studying Hand-Object Manipulation Via Joint Pose And Force Sensing

Capturing complex hand movements and object interactions using machine learning-powered stretchable smart textile gloves

A Physiology-Based Flexible Strap Sensor for Gesture Recognition by Sensing Tendon Deformation

A Robotic Hand-Arm Teleoperation System Using Human Arm/Hand with A Novel Data Glove

Design of a lightweight and easy‐to‐wear hand glove with multi‐modal tactile perception for digital human

Low-Cost Self-Calibration Data Glove Based on Space-Division Multiplexed Flexible Optical Fiber Sensor

GelSight Svelte Hand: A Three-finger, Two-DoF, Tactile-rich, Low-cost Robot Hand for Dexterous Manipulation

Real-Time Hand Gesture Tracking for Human–Computer Interface Based on Multi-Sensor Data Fusion

Data Glove with Self-Compensation Mechanism Based on High-Sensitive Elastic Fiber-Optic Sensor