Abstract:Training robot policies in simulation is becoming increasingly popular; nevertheless, a precise, reliable, and easy-to-use tactile simulator for contact-rich manipulation tasks is still missing. To close this gap, we develop TacEx -- a modular tactile simulation framework. We embed a state-of-the-art soft-body simulator for contacts named GIPC and vision-based tactile simulators Taxim and FOTS into Isaac Sim to achieve robust and plausible simulation of the visuotactile sensor GelSight Mini. We implement several Isaac Lab environments for Reinforcement Learning (RL) leveraging our TacEx simulation, including object pushing, lifting, and pole balancing. We validate that the simulation is stable and that the high-dimensional observations, such as the gel deformation and the RGB images from the GelSight camera, can be used for training. The code, videos, and additional results will be released online <a class="link-external link-https" href="https://sites.google.com/view/tacex" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the lack of an accurate, reliable, and easy - to - use haptic simulator, especially in robot manipulation tasks involving rich contacts. Specifically, the existing haptic simulators have deficiencies in handling soft - body interactions and haptic perception, resulting in difficulties in high - quality simulations. To address these issues, the authors developed TacEx - a modular haptic simulation framework, aiming to provide more stable and realistic haptic simulations, especially for the GelSight Mini sensor. ### Main Contributions of the Paper 1. **Modular and Extensible Framework**: TacEx is a modular haptic simulation framework embedded in NVIDIA's Isaac Sim. It combines state - of - the - art soft - body simulators (such as GIPC) and vision - based haptic simulators (such as Taxim and FOTS) to achieve robust and reliable simulations of the GelSight Mini sensor. 2. **GPU - Accelerated Soft - Body Simulation**: TacEx integrates GIPC, a GPU - accelerated and non - inversion - required soft - body simulator, which can efficiently simulate soft - body contacts. This enables TacEx to have higher precision and stability when dealing with complex contact scenarios. 3. **Support for Reinforcement Learning Environments**: TacEx supports multiple reinforcement learning (RL) tasks, including object pushing, grasping, and pole - balancing. These environments demonstrate TacEx's training ability in high - dimensional observation spaces, such as gel deformation and RGB images. 4. **Multimodal Perception**: TacEx not only simulates physical contacts but also combines optical simulation (generating perceived RGB images) and marker simulation (generating marker displacement fields) to comprehensively simulate the behavior of the GelSight sensor. ### Experimental Verification To verify the effectiveness of TacEx, the authors conducted multiple experiments: - **Ball Rolling Experiment**: Demonstrated the simulation behavior in a dynamic setting, using a single GelSight sensor for ball rolling. - **Object Grasping and Lifting**: Demonstrated robust grasping ability using two GelSight sensors. - **Beam Torsion Experiment**: Tested the stability of GIPC simulation under extreme deformations. - **Reinforcement Learning Tasks**: Implemented three RL environments (object pushing, object lifting, and pole - balancing), and trained policies using the PPO algorithm to verify the application of TacEx in RL. ### Future Work The authors point out that the current work mainly focuses on qualitative experiments and demonstrations of the simulator. In the future, they plan to conduct more quantitative experiments, compare different haptic simulation methods, and explore the possibility of Sim2Real transfer. In addition, they also plan to expand the framework to include more RL environments and haptic simulation methods, with the ultimate goal of creating a benchmark platform for haptic simulators and algorithms. Through these efforts, TacEx is expected to become a powerful tool in the field of haptic simulation, promoting the development of robot haptic perception and control technologies.

TacEx: GelSight Tactile Simulation in Isaac Sim -- Combining Soft-Body and Visuotactile Simulators

TacSL: A Library for Visuotactile Sensor Simulation and Learning

Taxim: An Example-based Simulation Model for GelSight Tactile Sensors

Simulation, Learning, and Application of Vision-Based Tactile Sensing at Large Scale

Tacchi: A Pluggable and Low Computational Cost Elastomer Deformation Simulator for Optical Tactile Sensors

Generation of GelSight Tactile Images for Sim2Real Learning

TacIPC: Intersection- and Inversion-free FEM-based Elastomer Simulation For Optical Tactile Sensors

DIFFTACTILE: A Physics-based Differentiable Tactile Simulator for Contact-rich Robotic Manipulation

Elastic Tactile Simulation Towards Tactile-Visual Perception.

Elastic Interaction of Particles for Robotic Tactile Simulation

General-Purpose Sim2Real Protocol for Learning Contact-Rich Manipulation With Marker-Based Visuotactile Sensors

Optimizing BioTac Simulation for Realistic Tactile Perception

3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing

Crossing the Reality Gap in Tactile-Based Learning

RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator

FOTS: A Fast Optical Tactile Simulator for Sim2Real Learning of Tactile-motor Robot Manipulation Skills

Interpreting and Predicting Tactile Signals for the SynTouch BioTac

Sim2Real Manipulation on Unknown Objects with Tactile-based Reinforcement Learning

Beyond Flat GelSight Sensors: Simulation of Optical Tactile Sensors of Complex Morphologies for Sim2Real Learning

9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation

MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation