Driv3R: Learning Dense 4D Reconstruction for Autonomous Driving

Xin Fei,Wenzhao Zheng,Yueqi Duan,Wei Zhan,Masayoshi Tomizuka,Kurt Keutzer,Jiwen Lu

2024-12-10

Abstract:Realtime 4D reconstruction for dynamic scenes remains a crucial challenge for autonomous driving perception. Most existing methods rely on depth estimation through self-supervision or multi-modality sensor fusion. In this paper, we propose Driv3R, a DUSt3R-based framework that directly regresses per-frame point maps from multi-view image sequences. To achieve streaming dense reconstruction, we maintain a memory pool to reason both spatial relationships across sensors and dynamic temporal contexts to enhance multi-view 3D consistency and temporal integration. Furthermore, we employ a 4D flow predictor to identify moving objects within the scene to direct our network focus more on reconstructing these dynamic regions. Finally, we align all per-frame pointmaps consistently to the world coordinate system in an optimization-free manner. We conduct extensive experiments on the large-scale nuScenes dataset to evaluate the effectiveness of our method. Driv3R outperforms previous frameworks in 4D dynamic scene reconstruction, achieving 15x faster inference speed compared to methods requiring global alignment. Code: <a class="link-external link-https" href="https://github.com/Barrybarry-Smith/Driv3R" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

### Problems the paper attempts to solve

The paper "Driv3R: Learning Dense 4D Reconstruction for Autonomous Driving" aims to solve the problem of dense reconstruction of real - time dynamic scenes in autonomous driving perception. Specifically, the paper proposes a framework named Driv3R, which can directly regress the point cloud map of each frame from multi - view image sequences and perform streaming dense reconstruction in the global world coordinate system. The main contributions of the paper are as follows:

1. **Real - time and accuracy**: Achieve real - time and accurate dense reconstruction of dynamic scenes, which is crucial for perception tasks in autonomous driving and robotics.
2. **No need for global alignment optimization**: By maintaining a spatio - temporal memory pool, Driv3R can achieve multi - view 3D consistency and temporal integration without relying on global alignment optimization.
3. **Dynamic object recognition**: Introduce a 4D flow predictor to identify moving objects in the scene, thereby guiding the network to pay more attention to the reconstruction of these dynamic regions.
4. **Efficiency**: Compared with methods that require global alignment optimization, the inference speed of Driv3R is increased by 15 times.

### Main methods and techniques

1. **Problem definition**:
- Given multi - view RGB camera images \(I_t=\{I_{t,c}\}_{c = 1}^C\) and their corresponding camera poses \(\{T_{t,c},K_{t,c}\}_{c = 1}^C\), the goal of Driv3R is to learn 4D dense pixel - level point cloud representations \(\{P_t|t\in T\}\) in the global world coordinate system.

2. **Spatio - temporal memory pool**:
- Maintain a spatio - temporal memory pool to handle the spatio - temporal relationships in the input sequence. Through the cross - attention mechanism, reduce unnecessary computational overhead and improve the efficiency of feature update.
- Use ViT to encode image features and update features through key - value pairs in the memory pool.

3. **4D flow predictor**:
- Based on the RAFT model, generate optical flow maps to identify dynamic objects.
- Use the pre - trained SAM2 model to further refine the masks of dynamic objects.

4. **Multi - view aligner**:
- Align the point cloud maps predicted by multiple sensors to the global world coordinate system to ensure the consistency of point clouds in time and space.

5. **Training strategy**:
- Use the depth estimation results generated by the pre - trained R3D3 model as supervision signals.
- Train the model in two stages. The first stage deals with the temporal information of a single sensor, and the second stage deals with the spatial information of multiple sensors.

### Experimental results

- **Depth estimation**: On the nuScenes dataset, Driv3R has achieved results comparable to existing methods in depth estimation, especially performing well in dynamic scenes.
- **Dynamic objects and scene reconstruction**: On subsets containing a large number of dynamic objects, Driv3R outperforms other methods in both reconstruction and depth estimation.
- **Efficiency analysis**: The inference speed of Driv3R is significantly faster than methods that require global alignment optimization, and it is also competitive in memory usage.

### Conclusion

Driv3R has successfully solved the problem of dense reconstruction of real - time dynamic scenes in autonomous driving through the innovative spatio - temporal memory pool and 4D flow predictor, achieving efficient and accurate 4D reconstruction.