Abstract:One of the major problems that cause continual trouble in deep learning networks is that training a large network requires massive labelled datasets. The preparation of a massive labelled dataset is a cumbersome task and requires lot of human interventions. This paper proposes a novel generator network 'Sim2Real' transfer is a recent and fast-developing field in machine learning used to bridge the gap between simulated and real data. Training with simulated datasets often converges due to its size but fails to generalize real-world applications. Simulated datasets can be used to train and test deep learning models, enables the development and evaluation of new algorithms and architectures. By simulating road dataset, researchers can generate large amounts of realistic road-traffic dataset that can be used to study and understand several problems such as vehicular object tracking and classification, traffic situation analysis etc. The main advantage of such a transfer algorithm is to use the abundance of a simulated dataset to generate huge realistic-looking datasets to solve data-intense tasks. This work presents a novel, robust sim2real algorithm that converts the labels of a semantic segmentation map to a realistic-looking street view using the Cityscapes dataset and aims to achieve robust urban mobility for smart cities. Further, the generalizability of the Cycle Generative Adversarial Network (CycleGAN) architecture was tested by using an origami robot dataset for sim2real transfer. We show that the results were found to be qualitatively satisfactory for different traffic analysis applications. In addition, road perception was done using a lightweight SVM pipeline and evaluated on the KITTI dataset. We have incorporated Cycle Consistency Loss and Identity Loss as the metrics to evaluate the performance of the proposed Cycle GAN model. We inferred that the proposed Cycle GAN model provides an Identity loss of less than 0.2 in both the Cityscapes dataset and KITTI datasets. Also, we understand that the super-pixel resolution has a good impact on the quantitative results of the proposed Cycle GAN models.

Digital Twin (DT)-CycleGAN: Enabling Zero-Shot Sim-to-Real Transfer of Visual Grasping Models

Ensemble Bootstrapped Deep Deterministic Policy Gradient For Vision-Based Robotic Grasping

A digital twin-based sim-to-real transfer for deep reinforcement learning-enabled industrial robot grasping

Virtual View Generation Based on 3D-Dense-attentive GAN Networks

A Cascaded Deep Learning Framework for Real-time and Robust Grasp Planning

ALDM-Grasping: Diffusion-aided Zero-Shot Sim-to-Real Transfer for Robot Grasping

A Grasps-Generation-and-selection Convolutional Neural Network for a Digital Twin of Intelligent Robotic Grasping.

RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real

Modular Deep Q Networks for Sim-to-real Transfer of Visuo-motor Policies

Bidirectional Sim-to-Real Transfer for GelSight Tactile Sensors With CycleGAN

A digital twin of intelligent robotic grasping based on single-loop-optimized differentiable architecture search and sim-real collaborative learning

SingleS2R: Single sample driven Sim-to-Real transfer for Multi-Source Visual-Tactile Information Understanding using multi-scale vision transformers

DT/MARS-CycleGAN: Improved Object Detection for MARS Phenotyping Robot

Towards Precise Model-free Robotic Grasping with Sim-to-Real Transfer Learning

Augmenting Tactile Simulators with Real-like and Zero-Shot Capabilities

Automated Creation of Digital Cousins for Robust Policy Learning

Deep Reinforcement Learning Enhanced Convolutional Neural Networks for Robotic Grasping

How to Close Sim-Real Gap? Transfer with Segmentation!

A hybrid Cycle GAN-based lightweight road perception pipeline for road dataset generation for Urban mobility

Adversarial Discriminative Sim-to-real Transfer of Visuo-motor Policies

Triple Regression for Camera Agnostic Sim2Real Robot Grasping and Manipulation Tasks