Abstract:The Object goal Navigation (ObjectNav) task requires an agent to navigate through a previously unknown domestic scenario using spatial and semantic contextual information, where the goal is specified by a semantic label (e.g., find a TV). Such a task is especially challenging as it requires formulating and understanding the complex co-occurrence relations among objects in diverse settings, which is critical for long-sequence navigational decision-making. Existing methods learn to either explicitly represent co-occurrence relationships as discrete semantic priors, or implicitly encode them from raw observations, thus can not benefit from the rich environmental semantics. In this work, we propose a novel Deep Reinforcement Learning (DRL) based ObjectNav strategy by actively imagining spatial and semantic clues outside the agent’s Field of View (FoV) and further mining Continuous Environmental Representations (CER) using self-supervised learning. Additionally, the illusion of spatial and semantic patterns allows the agent to perform Multi-Step Forward-Looking Planning (MSFLP) by considering the temporal evolution of egocentric local observations. Our approach is thoroughly evaluated and ablated in the visually realistic environments of the Matterport3D (MP3D) dataset. The experimental results reflect that our method combining CER and imagination-based MSFLP facilitates learning complicated semantic priors and navigation skills, thus achieving state-of-the-art performance on the ObjectNav task. In addition, adequate quantitative and qualitative analyses validate the excellent generalization ability and superiority of our method.

Multi goals and multi scenes visual mapless navigation in indoor using meta-learning and scene priors

Context vector-based visual mapless navigation in indoor using hierarchical semantic information and meta-learning

Unifying Visual Localization and Scene Recognition for People with Visual Impairment

MVSSC: Meta-reinforcement Learning Based Visual Indoor Navigation Using Multi-View Semantic Spatial Context

Visual Navigation with Multiple Goals Based on Deep Reinforcement Learning

HSPNav: Hierarchical Scene Prior Learning for Visual Semantic Navigation Towards Real Settings

Learning Navigational Visual Representations with Semantic Map Supervision

Multi-Agent Embodied Visual Semantic Navigation with Scene Prior Knowledge

Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Grounding

DivScene: Benchmarking LVLMs for Object Navigation with Diverse Scenes and Objects

Visual Semantic Navigation using Scene Priors

MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigation

MGRL: Graph neural network based inference in a Markov network with reinforcement learning for visual navigation

Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation

Visual Perception Generalization for Vision-and-Language Navigation via Meta-Learning

Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigation

Seeing is Believing? Enhancing Vision-Language Navigation using Visual Perturbations

Think Holistically, Act Down-to-Earth: A Semantic Navigation Strategy with Continuous Environmental Representation and Multi-step Forward Planning

NAVS: A Neural Attention-Based Visual SLAM for Autonomous Navigation in Unknown 3D Environments

Vision and Language Navigation Using Multi-head Attention Mechanism

A Novel Neural Multi-Store Memory Network for Autonomous Visual Navigation in Unknown Environment