Abstract:Synthetic data and novel rendering techniques have greatly influenced computer vision research in tasks like target tracking and human pose estimation. However, robotics research has lagged behind in leveraging it due to the limitations of most simulation frameworks, including the lack of low-level software control and flexibility, Robot Operating System integration, realistic physics, or photorealism. This hindered progress in (visual-)perception research, e.g. in autonomous robotics, especially in dynamic environments. Visual Simultaneous Localization and Mapping (V-SLAM), for instance, has been mostly developed passively, in static environments, and evaluated on few pre-recorded dynamic datasets due to the difficulties of realistically simulating dynamic worlds and the huge sim-to-real gap. To address these challenges, we present GRADE (Generating Realistic and Dynamic Environments), a highly customizable framework built upon NVIDIA Isaac Sim. We leverage Isaac's rendering capabilities and low-level APIs to populate and control the simulation, collect ground-truth data, and test online and offline approaches. Importantly, we introduce a new way to precisely repeat a recorded experiment within a physically enabled simulation while allowing environmental and simulation changes. Next, we collect a synthetic dataset of richly annotated videos in dynamic environments with a flying drone. Using that, we train detection and segmentation models for humans, closing the syn-to-real gap. Finally, we benchmark state-of-the-art dynamic V-SLAM algorithms, revealing their short tracking times and low generalization capabilities. We also show for the first time that the top-performing deep learning models do not achieve the best SLAM performance. Code and data are provided as open-source at <a class="link-external link-https" href="https://grade.is.tue.mpg.de" rel="external noopener nofollow">this https URL</a>.

Robotic Learning in your Backyard: A Neural Simulator from Open Source Components

Robot Simulation and Reinforcement Learning Training Platform Based on Distributed Architecture.

RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator

Learning Visual Parkour from Generated Images

Aerial Gym -- Isaac Gym Simulator for Aerial Robots

Reinforcement Learning with Generalizable Gaussian Splatting

IRASim: Learning Interactive Real-Robot Action Simulators

Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks

Air Learning: A Deep Reinforcement Learning Gym for Autonomous Aerial Robot Visual Navigation

RL-GSBridge: 3D Gaussian Splatting Based Real2Sim2Real Method for Robotic Manipulation Learning

The Radiance of Neural Fields: Democratizing Photorealistic and Dynamic Robotic Simulation

SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting

Physically Embodied Gaussian Splatting: A Realtime Correctable World Model for Robotics

GRADE: Generating Realistic And Dynamic Environments for Robotics Research with Isaac Sim

An open-source software framework for reinforcement learning-based control of tracked robots in simulated indoor environments

A novel simulation reality closed loop learning framework for autonomous robot skill learning

Learning to Fly -- a Gym Environment with PyBullet Physics for Reinforcement Learning of Multi-agent Quadcopter Control

Spatially Visual Perception for End-to-End Robotic Learning

Learning 3D Particle-based Simulators from RGB-D Videos

Learning Interactive Real-World Simulators

OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research