Abstract:Traditional frame-based video frame interpolation (VFI) methods rely on the linear motion assumption and brightness invariance assumption, which may lead to fatal errors confronting the scenarios with high-speed motions. To tackle the above challenge, inspired by the advantages of event cameras on asynchronously recording brightness changes at each pixel, we propose a Fast-Slow joint synthesis framework for event-enhanced high-speed video frame interpolation, named SuperFast, in this paper, which can generate high frame rate (5000 FPS, 200× faster) video from the input low frame rate (25 FPS) video and the corresponding event stream. In our framework, the task is divided into two sub-tasks, i.e., video frame interpolation for the contents with and without high-speed motions, which are tackled by two corresponding branches, i.e., the fast synthesis pathway and the slow synthesis pathway. The fast synthesis pathway leverages a spiking neural network to encode the input event stream, and combines boundary frames to generate intermediate results through synthesis and refinement, targeting on contents with high-speed motions. The slow synthesis pathway stacks the two input boundary frames and the event stream to synthesize intermediate results, focusing on relatively slow-motion contents. Finally, a fusion module with a comparison loss is utilized to generate the final video frame interpolation results. We also build a hybrid visual acquisition system containing an event camera and a high frame rate camera, and collect the first 5000 FPS High-Speed Event-enhanced Video frame Interpolation (THU[Formula: see text]) dataset. To evaluate the performance of our proposed framework, we have conducted experiments on our THU[Formula: see text] dataset and the existing HS-ERGB dataset. Experimental results demonstrate that our proposed framework can achieve state-of-the-art 200× video frame interpolation performance under high-speed motion scenarios.

High-speed Video Generation with an Event Camera

EventCap: Monocular 3D Capture of High-Speed Human Motions Using an Event Camera

EventHDR: from Event to High-Speed HDR Videos and Beyond

Learning to Reconstruct High Speed and High Dynamic Range Videos from Events

A Review of Image Reconstruction Based on Event Cameras

Eventmd: High-Speed Moving Object Detection Based on Event-Based Video Frames

Event-based High Dynamic Range Image and Very High Frame Rate Video Generation using Conditional Generative Adversarial Networks

Learning to Deblur and Generate High Frame Rate Video with an Event Camera

Globally-Optimal Event Camera Motion Estimation

Bringing a Blurry Frame Alive at High Frame-Rate With an Event Camera

EF-3DGS: Event-Aided Free-Trajectory 3D Gaussian Splatting

SuperFast: 200× Video Frame Interpolation Via Event Camera

Event-Enhanced Snapshot Compressive Videography at 10K FPS

TimeReplayer: Unlocking the Potential of Event Cameras for Video Interpolation

Motion-Aware Optical Camera Communication with Event Cameras

Event Camera Visualization

High Frame Rate Video Reconstruction Based on an Event Camera

High-Speed Depth Stream Generation From A Hybrid Camera

E2HQV: High-Quality Video Generation from Event Camera via Theory-Inspired Model-Aided Deep Learning

Speckle-Projection-Based High-Speed 3D Reconstruction Using Event Cameras.