Abstract:We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. In this work, we show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To the best of our knowledge, our model is the first method that can regress dense pixel trajectories from event data. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments not only suggest that our method successfully predicts pixel trajectories in continuous time but also that it is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public.

Continuous-time Intensity Estimation Using Event Cameras

Intensity/Inertial Integration-Aided Feature Tracking on Event Cameras

Event-based Continuous Color Video Decompression from Single Frames

Event-Based Fusion for Motion Deblurring with Cross-modal Attention

Learning Dense and Continuous Optical Flow from an Event Camera

An Asynchronous Linear Filter Architecture for Hybrid Event-Frame Cameras

Generalized Event Cameras

Event-based High Dynamic Range Image and Very High Frame Rate Video Generation using Conditional Generative Adversarial Networks

Intensity-image reconstruction for event cameras using convolutional neural network

A Review of Image Reconstruction Based on Event Cameras

Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation

Dense Continuous-Time Optical Flow from Event Cameras

Bringing a Blurry Frame Alive at High Frame-Rate With an Event Camera

Single Image Optical Flow Estimation with an Event Camera.

High-speed Video Generation with an Event Camera

Joint Filtering of Intensity Images and Neuromorphic Events for High-Resolution Noise-Robust Imaging

Dense Continuous-Time Optical Flow from Events and Frames

Globally-Optimal Event Camera Motion Estimation

Eventmd: High-Speed Moving Object Detection Based on Event-Based Video Frames

Standard and Event Cameras Fusion for Feature Tracking

Temporal-Mapping Photography for Event Cameras