Abstract:The accuracy of learning-based optical flow estimation models heavily relies on the realism of the training datasets. Current approaches for generating such datasets either employ synthetic data or generate images with limited realism. However, the domain gap of these data with real-world scenes constrains the generalization of the trained model to real-world applications. To address this issue, we investigate generating realistic optical flow datasets from real-world images. Firstly, to generate highly realistic new images, we construct a layered depth representation, known as multiplane images (MPI), from single-view images. This allows us to generate novel view images that are highly realistic. To generate optical flow maps that correspond accurately to the new image, we calculate the optical flows of each plane using the camera matrix and plane depths. We then project these layered optical flows into the output optical flow map with volume rendering. Secondly, to ensure the realism of motion, we present an independent object motion module that can separate the camera and dynamic object motion in MPI. This module addresses the deficiency in MPI-based single-view methods, where optical flow is generated only by camera motion and does not account for any object movement. We additionally devise a depth-aware inpainting module to merge new images with dynamic objects and address unnatural motion occlusions. We show the superior performance of our method through extensive experiments on real-world datasets. Moreover, our approach achieves state-of-the-art performance in both unsupervised and supervised training of learning-based models. The code will be made publicly available at: \url{<a class="link-external link-https" href="https://github.com/Sharpiless/MPI-Flow" rel="external noopener nofollow">this https URL</a>}.

RealFlow: EM-Based Realistic Optical Flow Dataset Generation from Videos

Unsupervised Learning of Scene Flow Estimation Fusing with Local Rigidity.

Optical Flow Dataset and Benchmark for Visual Crowd Analysis

MPI-Flow: Learning Realistic Optical Flow with Multiplane Images

A Dense Optical Flow Registration Algorithm Based on Deep Learning

Learning Optical Flow from Event Camera with Rendered Dataset

The devil in the details: simple and effective optical flow synthetic data generation

OmniFlow: Human Omnidirectional Optical Flow

FlowDA: Unsupervised Domain Adaptive Framework for Optical Flow Estimation

BlinkFlow: A Dataset to Push the Limits of Event-based Optical Flow Estimation

Skin the sheep not only once: Reusing Various Depth Datasets to Drive the Learning of Optical Flow

Nonrigid Optical Flow Ground Truth for Real-World Scenes with Time-Varying Shading Effects

FacialFlowNet: Advancing Facial Optical Flow Estimation with a Diverse Dataset and a Decomposed Model

FlowDiffuser: Advancing Optical Flow Estimation with Diffusion Models

Learning Omnidirectional Flow in 360-degree Video via Siamese Representation

CLIP-FLOW: CONTRASTIVE LEARNING WITH ITERATIVE PSEUDO LABELING FOR OPTICAL FLOW

Rethinking Optical Flow from Geometric Matching Consistent Perspective

Learning Multi-Human Optical Flow

STFlow: Self-Taught Optical Flow Estimation Using Pseudo Labels

CLIP-FLow: Contrastive Learning by semi-supervised Iterative Pseudo labeling for Optical Flow Estimation

DeepFaceFlow: In-the-wild Dense 3D Facial Motion Estimation