Abstract:In many real-world scenarios where extrinsic rewards to the agent are extremely sparse, curiosity has emerged as a useful concept providing intrinsic rewards that enable the agent to explore its environment and acquire information to achieve its goals. Despite their strong performance on many sparse-reward tasks, existing curiosity approaches rely on an overly holistic view of state transitions, and do not allow for a structured understanding of specific aspects of the environment. In this paper, we formulate curiosity based on grounded question answering by encouraging the agent to ask questions about the environment and be curious when the answers to these questions change. We show that natural language questions encourage the agent to uncover specific knowledge about their environment such as the physical properties of objects as well as their spatial relationships with other objects, which serve as valuable curiosity rewards to solve sparse-reward tasks more efficiently.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to make agents explore more effectively in sparse - reward environments. In many real - world scenarios, the external rewards received by agents are very scarce, which makes it difficult for agents to effectively explore the environment and achieve goals. Although existing curiosity - driven methods perform well on many sparse - reward tasks, they rely on an overall view of state transitions and cannot provide a structured understanding of specific aspects of the environment. Therefore, the paper proposes a new form of curiosity mechanism based on grounded question answering, namely **ASK&EXPLORE (ANE)**, which promotes agents' exploration of specific knowledge of the environment by encouraging agents to ask questions about the environment and generating curiosity when the answers to the questions change. This method can not only help agents discover the physical properties of objects, but also understand the spatial relationships between objects, thus providing valuable intrinsic rewards for solving sparse - reward tasks. Specifically, the paper achieves this goal in the following ways: 1. **Language - based curiosity mechanism**: The paper proposes using natural - language questions as a medium for exploration. These questions can capture the physical properties of objects (such as "Is the large sphere green?") and the spatial relationships between objects (such as "Is there a blue rubber ball behind the cyan ball?"). In this way, agents can obtain specific knowledge about the current state, and this knowledge, as a valuable intrinsic reward, helps agents solve long - term sparse - reward tasks more efficiently. 2. **Experimental verification**: The paper conducted experiments in the CLEVR - Robot environment to verify the effectiveness of the curiosity mechanism based on grounded question answering. The experimental results show that, in a sparse - reward setting, ANE significantly outperforms other benchmark methods, especially in the case of a single question. In addition, the paper also studied the impact of questions of different complexities on agent performance and found that even simple spatial - relationship queries are sufficient as curiosity signals in sparse - reward tasks. Overall, this paper aims to improve the exploration efficiency and task - completion ability of agents in sparse - reward environments by introducing a curiosity mechanism based on grounded question answering.

Ask & Explore: Grounded Question Answering for Curiosity-Driven Exploration

Enhancing Curiosity-driven Reinforcement Learning Through Historical State Information for Long-term Exploration

Curiosity-driven Exploration by Self-supervised Prediction

Understanding exploration in humans and machines by formalizing the function of curiosity

Curiosity-Driven Exploration via Latent Bayesian Surprise

Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environments

Self-Attention-Based Temporary Curiosity in Reinforcement Learning Exploration

CIExplore: Curiosity and Influence-based Exploration in Multi-Agent Cooperative Scenarios with Sparse Rewards

CMBE: Curiosity-driven Model-Based Exploration for Multi-Agent Reinforcement Learning in Sparse Reward Settings

Curiosity-driven Exploration in Sparse-reward Multi-agent Reinforcement Learning

Prototyping three key properties of specific curiosity in computational reinforcement learning

Intrinsic Rewards for Exploration Without Harm From Observational Noise: A Simulation Study Based on the Free Energy Principle

Dynamic Memory-Based Curiosity: A Bootstrap Approach for Exploration in Reinforcement Learning

Curiosity and the dynamics of optimal exploration

Attention-based Curiosity-driven Exploration in Deep Reinforcement Learning

Don’t Ignore the Drive of Curiosity: Rethinking Subtleties Between Universality of Commonsense Knowledge and Excellence of Large Language Models

Curiosity-Driven Rainbow Agent Exploratory Learning

Ask to Learn: A Study on Curiosity-driven Question Generation

Computational mechanisms of curiosity and goal-directed exploration

A unified strategy for implementing curiosity and empowerment driven reinforcement learning

Large-Scale Study of Curiosity-Driven Learning