Collaborative Instance Navigation: Leveraging Agent Self-Dialogue to Minimize User Input

Francesco Taioli,Edoardo Zorzi,Gianni Franchi,Alberto Castellini,Alessandro Farinelli,Marco Cristani,Yiming Wang

2024-12-02

Abstract:Existing embodied instance goal navigation tasks, driven by natural language, assume human users to provide complete and nuanced instance descriptions prior to the navigation, which can be impractical in the real world as human instructions might be brief and ambiguous. To bridge this gap, we propose a new task, Collaborative Instance Navigation (CoIN), with dynamic agent-human interaction during navigation to actively resolve uncertainties about the target instance in natural, template-free, open-ended dialogues. To address CoIN, we propose a novel method, Agent-user Interaction with UncerTainty Awareness (AIUTA), leveraging the perception capability of Vision Language Models (VLMs) and the capability of Large Language Models (LLMs). First, upon object detection, a Self-Questioner model initiates a self-dialogue to obtain a complete and accurate observation description, while a novel uncertainty estimation technique mitigates inaccurate VLM perception. Then, an Interaction Trigger module determines whether to ask a question to the user, continue or halt navigation, minimizing user input. For evaluation, we introduce CoIN-Bench, a benchmark supporting both real and simulated humans. AIUTA achieves competitive performance in instance navigation against state-of-the-art methods, demonstrating great flexibility in handling user inputs.

Artificial Intelligence

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to address the challenges in **Collaborative Instance Navigation (CoIN)**, especially in the instance - navigation tasks guided by natural language, how to reduce user input and accurately find the target object through dynamic human - machine interaction.

Specifically, the existing natural - language - based target - navigation tasks usually assume that the user provides a complete and detailed instance description before the navigation starts, which is often unrealistic in practical applications. The user may be unable or unwilling to provide all the detailed information, resulting in the need for additional information during the navigation process to eliminate uncertainties. To solve this problem, the paper proposes a new task - **Collaborative Instance Navigation (CoIN)**, which allows the agent to interact dynamically with the user through natural language during the navigation process to obtain more detailed information about the target object.

#### Main problems and solutions

1. **Problem description**:
- **Limitations of existing methods**: Current methods assume that the user provides a complete instance description before the navigation starts, which may be impractical in practical applications.
- **Real - world requirements**: In real - world scenarios, the user may only be able to provide short and ambiguous instructions, such as "find the picture", and the specific details need to be gradually confirmed during the navigation process.

2. **Solutions**:
- **Introducing the CoIN task**: Allows the agent to interact dynamically with the user through natural language during the navigation process to obtain more detailed information about the target object.
- **Proposing the AIUTA method**: AIUTA (Agent - user Interaction with UncerTainty Awareness) is a new method proposed for the CoIN task, which uses the Self - Questioner module and the Interaction Trigger module to reduce perceived uncertainty and minimize user input.

#### Key components of the AIUTA method

1. **Self - Questioner (self - dialogue module)**:
- Utilizes large - language models (LLM) and visual - language models (VLM) for self - dialogue to generate detailed descriptions of the detected objects.
- Proposes an entropy - based uncertainty - estimation technique to filter uncertain attributes, thereby obtaining more accurate descriptions.

2. **Interaction Trigger (interaction - trigger module)**:
- Based on the detected objects and known target facts, decides whether to ask the user questions, continue the navigation, or terminate the navigation.
- Calculates the similarity score to evaluate the matching degree between the current description and the target object, thereby deciding the next action.

#### Experimental verification

To evaluate the effectiveness of the CoIN task, the paper introduces the **CoIN - Bench benchmark**, which supports online evaluation of real - user and simulated - user, and introduces new performance metrics to measure the effectiveness of the agent - user interaction.

### Summary

This paper solves the problems of excessive user input and incomplete descriptions in the existing natural - language - guided instance - navigation tasks by introducing the CoIN task and the AIUTA method, achieving a more flexible and efficient navigation strategy.