Abstract:Learning to estimate 3D geometry in a single frame and optical flow from consecutive frames by watching unlabeled videos via deep convolutional network has made significant progress recently. Current state-of-the-art (SoTA) methods treat the two tasks independently. One typical assumption of the existing depth estimation methods is that the scenes contain no independent moving objects. while object moving could be easily modeled using optical flow. In this paper, we propose to address the two tasks as a whole, i.e., to jointly understand per-pixel 3D geometry and motion. This eliminates the need of static scene assumption and enforces the inherent geometrical consistency during the learning process, yielding significantly improved results for both tasks. We call our method as "Every Pixel Counts++" or "EPC++". Specifically, during training, given two consecutive frames from a video, we adopt three parallel networks to predict the camera motion (MotionNet), dense depth map (DepthNet), and per-pixel optical flow between two frames (OptFlowNet) respectively. The three types of information, are fed into a holistic 3D motion parser (HMP), and per-pixel 3D motion of both rigid background and moving objects are disentangled and recovered. Various loss terms are formulated to jointly supervise the three networks. An effective adaptive training strategy is proposed to achieve better performance and more efficient convergence. Comprehensive experiments were conducted on datasets with different scenes, including driving scenario (KITTI 2012 and KITTI 2015 datasets), mixed outdoor/indoor scenes (Make3D) and synthetic animation (MPI Sintel dataset). Performance on the five tasks of depth estimation, optical flow estimation, odometry, moving object segmentation and scene flow estimation shows that our approach outperforms other SoTA methods, demonstrating the effectiveness of each module of our proposed -ethod. Code will be available at: https://github.com/chenxuluo/EPC.

Geometry Understanding from Autonomous Driving Scenarios Based on Feature Refinement

Multi-Task Learning in Autonomous Driving Scenarios Via Adaptive Feature Refinement Networks

Unsupervised Learning of Scene Flow Estimation Fusing with Local Rigidity.

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

Unsupervised Learning of Depth, Optical Flow and Pose With Occlusion From 3D Geometry

Self-supervised Learning of Monocular 3D Geometry Understanding with Two- and Three-View Geometric Constraints

GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose

Unsupervised Joint Learning of Depth, Optical Flow, Ego-motion from Video

Self-Supervised Multi-Scale Hierarchical Refinement Method for Joint Learning of Optical Flow and Depth

Every Pixel Counts: Unsupervised Geometry Learning with Holistic 3D Motion Understanding

Feature-Level Collaboration: Joint Unsupervised Learning of Optical Flow, Stereo Depth and Camera Motion.

An Object Context Integrated Network for Joint Learning of Depth and Optical Flow.

3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose From Monocular Video

Learning Graph-Based Representations for Scene Flow Estimation

Learning Effective Geometry Representation from Videos for Self-Supervised Monocular Depth Estimation

Learning Geometry-Guided Depth via Projective Modeling for Monocular 3D Object Detection

Learning Monocular Depth in Dynamic Environment via Context-aware Temporal Attention

Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding

Cycle-SfM: Joint Self-Supervised Learning of Depth and Camera Motion from Monocular Image Sequences.

Unsupervised Monocular Depth Estimation for Monocular Visual SLAM Systems

Beyond Learning: Back to Geometric Essence of Visual Odometry via Fusion-Based Paradigm