Abstract:Learning to estimate 3D geometry in a single frame and optical flow from consecutive frames by watching unlabeled videos via deep convolutional network has made significant progress recently. Current state-of-the-art (SoTA) methods treat the two tasks independently. One typical assumption of the existing depth estimation methods is that the scenes contain no independent moving objects. while object moving could be easily modeled using optical flow. In this paper, we propose to address the two tasks as a whole, i.e., to jointly understand per-pixel 3D geometry and motion. This eliminates the need of static scene assumption and enforces the inherent geometrical consistency during the learning process, yielding significantly improved results for both tasks. We call our method as "Every Pixel Counts++" or "EPC++". Specifically, during training, given two consecutive frames from a video, we adopt three parallel networks to predict the camera motion (MotionNet), dense depth map (DepthNet), and per-pixel optical flow between two frames (OptFlowNet) respectively. The three types of information, are fed into a holistic 3D motion parser (HMP), and per-pixel 3D motion of both rigid background and moving objects are disentangled and recovered. Various loss terms are formulated to jointly supervise the three networks. An effective adaptive training strategy is proposed to achieve better performance and more efficient convergence. Comprehensive experiments were conducted on datasets with different scenes, including driving scenario (KITTI 2012 and KITTI 2015 datasets), mixed outdoor/indoor scenes (Make3D) and synthetic animation (MPI Sintel dataset). Performance on the five tasks of depth estimation, optical flow estimation, odometry, moving object segmentation and scene flow estimation shows that our approach outperforms other SoTA methods, demonstrating the effectiveness of each module of our proposed -ethod. Code will be available at: https://github.com/chenxuluo/EPC.

FeatFlow: Learning Geometric Features for 3D Motion Estimation

Unsupervised Learning of Scene Flow Estimation Fusing with Local Rigidity.

FlowMOT: 3D Multi-Object Tracking by Scene Flow Association

Recurrent Graph Optimal Transport for Learning 3D Flow Motion in Particle Tracking

Bi-PointFlowNet: Bidirectional Learning for Point Cloud Based Scene Flow Estimation

ScaleFlow++: Robust and Accurate Estimation of 3D Motion from Video

Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding

Learning Optical Flow and Scene Flow with Bidirectional Camera-LiDAR Fusion

CamLiFlow: Bidirectional Camera-LiDAR Fusion for Joint Optical Flow and Scene Flow Estimation

DELFlow: Dense Efficient Learning of Scene Flow for Large-Scale Point Clouds

3D Scene Flow Estimation on Pseudo-LiDAR: Bridging the Gap on Estimating Point Motion

Hierarchical Attention Learning of Scene Flow in 3D Point Clouds

Self-Supervised 3D Scene Flow Estimation and Motion Prediction using Local Rigidity Prior

Flow4D: Leveraging 4D Voxel Network for LiDAR Scene Flow Estimation

DeepLiDARFlow: A Deep Learning Architecture For Scene Flow Estimation Using Monocular Camera and Sparse LiDAR

STARFlow: Spatial Temporal Feature Re-embedding with Attentive Learning for Real-world Scene Flow

Rethinking Optical Flow from Geometric Matching Consistent Perspective

3DSFLabelling: Boosting 3D Scene Flow Estimation by Pseudo Auto-labelling

GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose

EgoFlowNet: Non-Rigid Scene Flow from Point Clouds with Ego-Motion Support