Navigation World Models

Amir Bar,Gaoyue Zhou,Danny Tran,Trevor Darrell,Yann LeCun

2024-12-05

Abstract:Navigation is a fundamental skill of agents with visual-motor capabilities. We introduce a Navigation World Model (NWM), a controllable video generation model that predicts future visual observations based on past observations and navigation actions. To capture complex environment dynamics, NWM employs a Conditional Diffusion Transformer (CDiT), trained on a diverse collection of egocentric videos of both human and robotic agents, and scaled up to 1 billion parameters. In familiar environments, NWM can plan navigation trajectories by simulating them and evaluating whether they achieve the desired goal. Unlike supervised navigation policies with fixed behavior, NWM can dynamically incorporate constraints during planning. Experiments demonstrate its effectiveness in planning trajectories from scratch or by ranking trajectories sampled from an external policy. Furthermore, NWM leverages its learned visual priors to imagine trajectories in unfamiliar environments from a single input image, making it a flexible and powerful tool for next-generation navigation systems.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning,Robotics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the limitations of current robot navigation strategies. Specifically, existing navigation strategies are difficult to dynamically introduce new constraints (such as "no left turn") after training, and cannot dynamically allocate more computational resources according to the task difficulty. In addition, these strategies usually rely on fixed environmental data and cannot adapt well to unknown or changing environments.

To solve these problems, the paper proposes a **Navigation World Model (NWM)**. NWM is a controllable video - generation model that can predict future visual observations based on past observations and navigation actions. NWM captures the dynamics of complex environments by using the Conditional Diffusion Transformer (CDiT) and can be extended to 1 billion parameters. The main features of NWM include:

1. **Dynamic planning ability**: Unlike supervised - learning navigation strategies, NWM can dynamically introduce constraint conditions during the planning process, thus more flexibly dealing with different navigation tasks.
2. **Simulating and evaluating trajectories**: NWM can plan new navigation paths in familiar environments by simulating potential navigation plans and evaluating whether they reach the goal.
3. **Adapting to unknown environments**: NWM can use its learned visual prior knowledge to imagine navigation paths in unknown environments from a single input image, making it a flexible and powerful next - generation navigation tool.

### Main contributions
1. **Proposing a new Navigation World Model (NWM)**: This model can predict future video frames and can be used for independent planning or combined with other navigation strategies to improve visual navigation performance.
2. **Designing a new type of Conditional Diffusion Transformer (CDiT)**: This model is significantly more computationally efficient than the standard Diffusion Transformer (DiT) and can be effectively extended in multiple environments and different robot morphologies.
3. **Demonstrating performance improvement in unknown environments**: By training on unlabeled Ego4D video data, the video prediction and generation performance of NWM in unknown environments has been significantly improved.

### Experimental results
The paper verifies the effectiveness of NWM through multiple experiments:
- **Video prediction and synthesis**: The quality of video prediction and synthesis of NWM in known environments is significantly better than existing methods, such as DIAMOND.
- **Navigation planning**: NWM performs excellently when used for independent planning and when combined with other navigation strategies, especially in unknown environments.
- **Adapting to constraint conditions**: NWM can dynamically introduce various constraint conditions, such as "no left turn", during the planning process and effectively abide by these constraints.

In general, by proposing NWM, this paper solves the problems of the rigidity and insufficient adaptability of existing navigation strategies, providing a new research direction for future robot navigation systems.