Abstract:The accuracy of learning-based optical flow estimation models heavily relies on the realism of the training datasets. Current approaches for generating such datasets either employ synthetic data or generate images with limited realism. However, the domain gap of these data with real-world scenes constrains the generalization of the trained model to real-world applications. To address this issue, we investigate generating realistic optical flow datasets from real-world images. Firstly, to generate highly realistic new images, we construct a layered depth representation, known as multiplane images (MPI), from single-view images. This allows us to generate novel view images that are highly realistic. To generate optical flow maps that correspond accurately to the new image, we calculate the optical flows of each plane using the camera matrix and plane depths. We then project these layered optical flows into the output optical flow map with volume rendering. Secondly, to ensure the realism of motion, we present an independent object motion module that can separate the camera and dynamic object motion in MPI. This module addresses the deficiency in MPI-based single-view methods, where optical flow is generated only by camera motion and does not account for any object movement. We additionally devise a depth-aware inpainting module to merge new images with dynamic objects and address unnatural motion occlusions. We show the superior performance of our method through extensive experiments on real-world datasets. Moreover, our approach achieves state-of-the-art performance in both unsupervised and supervised training of learning-based models. The code will be made publicly available at: \url{<a class="link-external link-https" href="https://github.com/Sharpiless/MPI-Flow" rel="external noopener nofollow">this https URL</a>}.

OptFlowCam: A 3D‐Image‐Flow‐Based Metric in Camera Space for Camera Paths in Scenes with Extreme Scale Variations

Unsupervised Learning of Scene Flow Estimation Fusing with Local Rigidity.

A Multiscale-Contour-based Interpolation Framework for Generating a Time-Varying Quasi-Dense Point Cloud Sequence.

FlowMap: High-Quality Camera Poses, Intrinsics, and Depth via Gradient Descent

ScaleFlow++: Robust and Accurate Estimation of 3D Motion from Video

CamLiFlow: Bidirectional Camera-LiDAR Fusion for Joint Optical Flow and Scene Flow Estimation

Computing Diffeomorphic Paths for Large Motion Interpolation

Metric Flow Matching for Smooth Interpolations on the Data Manifold

Optical Flow in Mostly Rigid Scenes

An Optical Flow-Based Approach for Minimally-Divergent Velocimetry Data Interpolation

Multi-Scale Generalized Plane Match for Optical Flow

Fast Multi-frame Stereo Scene Flow with Motion Segmentation

Full Flow: Optical Flow Estimation By Global Optimization over Regular Grids

MPI-Flow: Learning Realistic Optical Flow with Multiplane Images

Joint direct estimation of 3D geometry and 3D motion using spatio temporal gradients

Path‐space Motion Estimation and Decomposition for Robust Animation Filtering

Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation

HMAFlow: Learning More Accurate Optical Flow via Hierarchical Motion Field Alignment

Learning Optical Flow and Scene Flow with Bidirectional Camera-LiDAR Fusion

StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences