Abstract:Navigating complex indoor environments requires a deep understanding of the space the robotic agent is acting into to correctly inform the navigation process of the agent towards the goal location. In recent learning-based navigation approaches, the scene understanding and navigation abilities of the agent are achieved simultaneously by collecting the required experience in simulation. Unfortunately, even if simulators represent an efficient tool to train navigation policies, the resulting models often fail when transferred into the real world. One possible solution is to provide the navigation model with mid-level visual representations containing important domain-invariant properties of the scene. But, what are the best representations that facilitate the transfer of a model to the real-world? How can they be combined? In this work we address these issues by proposing a benchmark of Deep Learning architectures to combine a range of mid-level visual representations, to perform a PointGoal navigation task following a Reinforcement Learning setup. All the proposed navigation models have been trained with the Habitat simulator on a synthetic office environment and have been tested on the same real-world environment using a real robotic platform. To efficiently assess their performance in a real context, a validation tool has been proposed to generate realistic navigation episodes inside the simulator. Our experiments showed that navigation models can benefit from the multi-modal input and that our validation tool can provide good estimation of the expected navigation performance in the real world, while saving time and resources. The acquired synthetic and real 3D models of the environment, together with the code of our validation tool built on top of Habitat, are publicly available at the following link: <a class="link-external link-https" href="https://iplab.dmi.unict.it/EmbodiedVN/" rel="external noopener nofollow">this https URL</a>

Embodied Learning for Lifelong Visual Perception

Think Holistically, Act Down-to-Earth: A Semantic Navigation Strategy with Continuous Environmental Representation and Multi-step Forward Planning

Active Perception for Visual-Language Navigation

SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency

Learning and Planning with a Semantic Model

Self-Supervised Embodied Learning for Semantic Segmentation

Learning Efficient Multi-Agent Cooperative Visual Exploration

Deep Learning for Embodied Vision Navigation: A Survey

Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Following

Learning Navigational Visual Representations with Semantic Map Supervision

Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks

Learning Autonomous Exploration and Mapping with Semantic Vision

Embodied Contrastive Learning with Geometric Consistency and Behavioral Awareness for Object Navigation

Learning with a Mole: Transferable latent spatial representations for navigation without reconstruction

ALP: Action-Aware Embodied Learning for Perception

Learning a Semantic Prior for Guided Navigation

Image-based Navigation in Real-World Environments via Multiple Mid-level Representations: Fusion Models, Benchmark and Efficient Evaluation

Learning to Navigate in Complex Environments

Agent Journey Beyond RGB: Unveiling Hybrid Semantic-Spatial Environmental Representations for Vision-and-Language Navigation

Improving Vision-and-Language Navigation by Generating Future-View Image Semantics