Abstract:We study lifelong visual perception in an embodied setup, where we develop new models and compare various agents that navigate in buildings and occasionally request annotations which, in turn, are used to refine their visual perception capabilities. The purpose of the agents is to recognize objects and other semantic classes in the whole building at the end of a process that combines exploration and active visual learning. As we study this task in a lifelong learning context, the agents should use knowledge gained in earlier visited environments in order to guide their exploration and active learning strategy in successively visited buildings. We use the semantic segmentation performance as a proxy for general visual perception and study this novel task for several exploration and annotation methods, ranging from frontier exploration baselines which use heuristic active learning, to a fully learnable approach. For the latter, we introduce a deep reinforcement learning (RL) based agent which jointly learns both navigation and active learning. A point goal navigation formulation, coupled with a global planner which supplies goals, is integrated into the RL model in order to provide further incentives for systematic exploration of novel scenes. By performing extensive experiments on the Matterport3D dataset, we show how the proposed agents can utilize knowledge from previously explored scenes when exploring new ones, e.g. through less granular exploration and less frequent requests for annotations. The results also suggest that a learning-based agent is able to use its prior visual knowledge more effectively than heuristic alternatives.

Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks

Learning Exploration Policies for Navigation

Look Around and Learn: Self-Training Object Detection by Exploration

Embodied Learning for Lifelong Visual Perception

Robot Active Neural Sensing and Planning in Unknown Cluttered Environments

Learning to explore by reinforcement over high-level options

Graph-based Cluttered Scene Generation and Interactive Exploration using Deep Reinforcement Learning

Learning Human Search Behavior from Egocentric Visual Inputs

Interesting Object, Curious Agent: Learning Task-Agnostic Exploration

Learning Efficient Multi-Agent Cooperative Visual Exploration

Learning to Explore Informative Trajectories and Samples for Embodied Perception

Learning to Explore using Active Neural SLAM

Learning to Look around Objects for Top-View Representations of Outdoor Scenes

Look-ahead before you leap: end-to-end active recognition by forecasting the effect of motion

Learning and Planning with a Semantic Model

LeLaN: Learning A Language-Conditioned Navigation Policy from In-the-Wild Videos

Sidekick Policy Learning for Active Visual Exploration

Think Holistically, Act Down-to-Earth: A Semantic Navigation Strategy with Continuous Environmental Representation and Multi-step Forward Planning

NavDreams: Towards Camera-Only RL Navigation Among Humans

Attend Before you Act: Leveraging human visual attention for continual learning

Active Object Perceiver: Recognition-Guided Policy Learning for Object Searching on Mobile Robots