Abstract:Inspired by frame-based methods, state-of-the-art event-based optical flow networks rely on the explicit construction of correlation volumes, which are expensive to compute and store, rendering them unsuitable for robotic applications with limited compute and energy budget. Moreover, correlation volumes scale poorly with resolution, prohibiting them from estimating high-resolution flow. We observe that the spatiotemporally continuous traces of events provide a natural search direction for seeking pixel correspondences, obviating the need to rely on gradients of explicit correlation volumes as such search directions. We introduce IDNet (Iterative Deblurring Network), a lightweight yet high-performing event-based optical flow network directly estimating flow from event traces without using correlation volumes. We further propose two iterative update schemes: "ID" which iterates over the same batch of events, and "TID" which iterates over time with streaming events in an online fashion. Our top-performing ID model sets a new state of the art on DSEC benchmark. Meanwhile, the base ID model is competitive with prior arts while using 80% fewer parameters, consuming 20x less memory footprint and running 40% faster on the NVidia Jetson Xavier NX. Furthermore, the TID model is even more efficient offering an additional 5x faster inference speed and 8 ms ultra-low latency at the cost of only a 9% performance drop, making it the only model among current literature capable of real-time operation while maintaining decent performance.

What problem does this paper attempt to address?

The paper attempts to address several major drawbacks present in existing event-based optical flow estimation methods, particularly the high computational and storage costs associated with correlation volumes, and the scalability issues of these methods when handling high-resolution inputs. Specifically: 1. **High Latency**: Constructing correlation volumes requires accumulating event data, which leads to high latency. 2. **High Computational and Storage Costs**: Calculating and storing correlation volumes is very expensive, limiting the deployment of these methods on systems with limited memory and computational resources. 3. **Poor Scalability**: The computational complexity of correlation volumes increases sharply with input resolution, making it difficult for the algorithm to handle high-resolution events or provide fine-grained optical flow estimation. To address these issues, the paper proposes a new event-based optical flow estimation network—IDNet (Iterative Deblurring Network), which estimates optical flow directly from event trajectories without the need for correlation volumes. Additionally, the paper introduces two iterative update schemes: ID (Iterative Deblurring) and TID (Temporal Iterative Deblurring) to improve estimation quality and processing speed. ### Main Contributions: 1. **Optical Flow Estimation without Correlation Volumes**: IDNet estimates optical flow by directly processing event trajectories, avoiding the high costs of constructing and storing correlation volumes. 2. **Efficient Iterative Deblurring Schemes**: The ID scheme improves estimation quality by iteratively deblurring the same batch of events multiple times, while the TID scheme achieves faster processing speed by iteratively processing streaming events over time. 3. **High Performance and Low Resource Consumption**: IDNet achieves new state-of-the-art performance on the DSEC benchmark, while reducing the number of parameters by 80%, memory usage by 20 times, and running speed by 40%. The TID model, while maintaining high performance, increases inference speed by 5 times, reduces latency to 8 milliseconds, making it suitable for real-time operations. ### Application Scenarios: - **Robotic Navigation**: IDNet and TID models are particularly suitable for resource-constrained agile robots, such as micro aerial/ground vehicles (MAVs/MGVs), which require efficient optical flow estimation under low power and low latency conditions. - **Real-time Vision Applications**: Due to their efficiency and low latency, these models are also applicable to real-time vision applications such as autonomous driving, augmented reality, etc. In summary, the paper proposes a lightweight and high-performance event-based optical flow estimation method, addressing the computational and storage cost bottlenecks of existing methods, and providing new solutions for real-time and resource-constrained applications.

Lightweight Event-based Optical Flow Estimation via Iterative Deblurring

A Dense Optical Flow Registration Algorithm Based on Deep Learning

FastFlowNet: A Lightweight Network for Fast Optical Flow Estimation

Dense Continuous-Time Optical Flow from Events and Frames

EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras

Fdflownet: Fast Optical Flow Estimation Using A Deep Lightweight Network

Dense Continuous-Time Optical Flow from Event Cameras

Towards Anytime Optical Flow Estimation with Event Cameras

Optical flow estimation from event-based cameras and spiking neural networks

Neuromorphic Optical Flow and Real-time Implementation with Event Cameras

Spatio-Temporal Recurrent Networks for Event-Based Optical Flow Estimation

Context-Aware Iteration Policy Network for Efficient Optical Flow Estimation

Event-based Temporally Dense Optical Flow Estimation with Sequential Learning

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

Real-Time Optical Flow for Vehicular Perception with Low- and High-Resolution Event Cameras

Fast Event-based Optical Flow Estimation by Triplet Matching

DIFT: Dynamic Iterative Field Transforms for Memory Efficient Optical Flow

SDformerFlow: Spatiotemporal swin spikeformer for event-based optical flow estimation

ResFlow: Fine-tuning Residual Optical Flow for Event-based High Temporal Resolution Motion Estimation

Efficient Meshflow and Optical Flow Estimation from Event Cameras