Abstract:The accuracy of learning-based optical flow estimation models heavily relies on the realism of the training datasets. Current approaches for generating such datasets either employ synthetic data or generate images with limited realism. However, the domain gap of these data with real-world scenes constrains the generalization of the trained model to real-world applications. To address this issue, we investigate generating realistic optical flow datasets from real-world images. Firstly, to generate highly realistic new images, we construct a layered depth representation, known as multiplane images (MPI), from single-view images. This allows us to generate novel view images that are highly realistic. To generate optical flow maps that correspond accurately to the new image, we calculate the optical flows of each plane using the camera matrix and plane depths. We then project these layered optical flows into the output optical flow map with volume rendering. Secondly, to ensure the realism of motion, we present an independent object motion module that can separate the camera and dynamic object motion in MPI. This module addresses the deficiency in MPI-based single-view methods, where optical flow is generated only by camera motion and does not account for any object movement. We additionally devise a depth-aware inpainting module to merge new images with dynamic objects and address unnatural motion occlusions. We show the superior performance of our method through extensive experiments on real-world datasets. Moreover, our approach achieves state-of-the-art performance in both unsupervised and supervised training of learning-based models. The code will be made publicly available at: \url{<a class="link-external link-https" href="https://github.com/Sharpiless/MPI-Flow" rel="external noopener nofollow">this https URL</a>}.

Full View Optical Flow Estimation Leveraged from Light Field Superpixel

Three Dimensional Reconstruction Using a Lenslet Light Field Camera

Unsupervised Learning of Scene Flow Estimation Fusing with Local Rigidity.

A Learning-Based Method Using Epipolar Geometry for Light Field Depth Estimation

Depth Estimation Algorithm for Light Field Data by Epipolar Image Analysis and Region Interpolation

Fast and Accurate Optical Flow based Depth Map Estimation from Light Fields

Light field angular super-resolution based on intrinsic and geometric information.

A New Parallel Intelligence Based Light Field Dataset for Depth Refinement and Scene Flow Estimation

Unsupervised Dense Light Field Reconstruction with Occlusion Awareness

Light Field Depth Estimation via Epipolar Plane Image Analysis and Locally Linear Embedding

Light Field Depth Estimation Exploiting Linear Structure in EPI

MPI-Flow: Learning Realistic Optical Flow with Multiplane Images

Robust depth estimation for light field via spinning parallelogram operator

Skin the sheep not only once: Reusing Various Depth Datasets to Drive the Learning of Optical Flow

PIV-Based 3D Fluid Flow Reconstruction Using Light Field Camera

An Unsupervised Optical Flow Estimation For LiDAR Image Sequences

Depth Fused from Intensity Range and Blur Estimation for Light-Field Cameras

Depth Estimation by Analyzing Intensity Distribution for Light-Field Cameras.

Plenoptic Reconstruction

Image Refined Disparity Our novel view Iterate Synthesized Light Field Novel view from Didyk

Light Field Reconstruction using Efficient Pseudo 4D Epipolar-Aware Structure