Neural Eulerian Scene Flow Fields

Kyle Vedder,Neehar Peri,Ishan Khatri,Siyi Li,Eric Eaton,Mehmet Kocamaz,Yue Wang,Zhiding Yu,Deva Ramanan,Joachim Pehserl

2024-10-29

Abstract:We reframe scene flow as the task of estimating a continuous space-time ODE that describes motion for an entire observation sequence, represented with a neural prior. Our method, EulerFlow, optimizes this neural prior estimate against several multi-observation reconstruction objectives, enabling high quality scene flow estimation via pure self-supervision on real-world data. EulerFlow works out-of-the-box without tuning across multiple domains, including large-scale autonomous driving scenes and dynamic tabletop settings. Remarkably, EulerFlow produces high quality flow estimates on small, fast moving objects like birds and tennis balls, and exhibits emergent 3D point tracking behavior by solving its estimated ODE over long-time horizons. On the Argoverse 2 2024 Scene Flow Challenge, EulerFlow outperforms all prior art, surpassing the next-best unsupervised method by more than 2.5x, and even exceeding the next-best supervised method by over 10%.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to solve several key challenges in scene flow estimation, especially the performance issues when dealing with small and fast - moving objects. Specifically:

1. **Limitations of existing methods**: Current scene flow methods perform poorly when dealing with small, fast - moving objects (such as birds or tennis balls), especially in complex environments like autonomous vehicles. These methods usually rely on point cloud data within a limited time interval and have difficulty capturing small and rapidly changing motions.

2. **Multi - frame optimization and continuous - time modeling**: Traditional scene flow methods mainly focus on motion estimation between two frames, ignoring dynamic information over a longer time span. This limits their ability to understand complex scenes.

3. **Advantages of unsupervised learning**: To overcome the problem of insufficient labeled data in supervised learning, this paper proposes an unsupervised method that can directly learn from real - world data without the need for a large amount of labeled data.

4. **Cross - domain generalization ability**: Existing scene flow methods usually need to be tuned for specific domains and have difficulty maintaining consistent performance in different scenarios. This paper hopes to develop a method that can perform well in multiple domains (such as autonomous driving and desktop operations) without additional tuning.

To solve the above problems, the paper proposes EulerFlow, a neural scene - flow estimation method based on ordinary differential equations (ODE). EulerFlow improves scene - flow estimation in the following ways:

- **Continuous - time modeling**: Model the scene flow as a continuous ODE that describes the change in position over time throughout the entire observation sequence.

- **Multi - frame optimization objective**: Use multi - frame reconstruction loss and cycle - consistency loss to optimize ODE estimation, thereby improving the ability to capture motion over a long time span.

- **Simple and flexible formula**: Use a simple Euler integration method to extract 3D point trajectories, demonstrating its superior performance in handling complex dynamic scenes.

Through these improvements, EulerFlow not only achieves significantly better performance than existing methods in multiple benchmark tests but also shows strong capabilities when dealing with small and fast - moving objects.