Abstract:Learning to estimate 3D geometry in a single frame and optical flow from consecutive frames by watching unlabeled videos via deep convolutional network has made significant progress recently. Current state-of-the-art (SoTA) methods treat the two tasks independently. One typical assumption of the existing depth estimation methods is that the scenes contain no independent moving objects. while object moving could be easily modeled using optical flow. In this paper, we propose to address the two tasks as a whole, i.e., to jointly understand per-pixel 3D geometry and motion. This eliminates the need of static scene assumption and enforces the inherent geometrical consistency during the learning process, yielding significantly improved results for both tasks. We call our method as "Every Pixel Counts++" or "EPC++". Specifically, during training, given two consecutive frames from a video, we adopt three parallel networks to predict the camera motion (MotionNet), dense depth map (DepthNet), and per-pixel optical flow between two frames (OptFlowNet) respectively. The three types of information, are fed into a holistic 3D motion parser (HMP), and per-pixel 3D motion of both rigid background and moving objects are disentangled and recovered. Various loss terms are formulated to jointly supervise the three networks. An effective adaptive training strategy is proposed to achieve better performance and more efficient convergence. Comprehensive experiments were conducted on datasets with different scenes, including driving scenario (KITTI 2012 and KITTI 2015 datasets), mixed outdoor/indoor scenes (Make3D) and synthetic animation (MPI Sintel dataset). Performance on the five tasks of depth estimation, optical flow estimation, odometry, moving object segmentation and scene flow estimation shows that our approach outperforms other SoTA methods, demonstrating the effectiveness of each module of our proposed -ethod. Code will be available at: https://github.com/chenxuluo/EPC.

MOAC: Multi-level Perception Optimizer Based on Dual Augmented Cost for Structure- From-Motion

Bi-objective Optimization for Robust RGB-D Visual Odometry

DRO: Deep Recurrent Optimizer for Structure-from-Motion.

CAMO-MOT: Combined Appearance-Motion Optimization for 3D Multi-Object Tracking With Camera-LiDAR Fusion

Local Optimized and Scalable Frame-to-model SLAM

Boosting Object Representation Learning via Motion and Object Continuity

Structuring a Training Strategy to Robustify Perception Models with Realistic Image Augmentations

Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding

A target-free video structural motion estimation method based on multi-path optimization

Optimizing Camera Motion with MCTS and Target Motion Modeling in Multi-Target Active Object Tracking

Exploring Data Augmentation for Multi-Modality 3D Object Detection

MOPT: Multi-Object Panoptic Tracking

MOIPC-MAAC: Communication-Assisted Multi-Objective MARL for Trajectory Planning and Task Offloading in Multi-UAV Assisted MEC

RoCo:Robust Collaborative Perception By Iterative Object Matching and Pose Adjustment

Multi-object tracking with adaptive measurement noise and information fusion

Heuristic Search for Structural Constraints in Data Association

MCSfM: Multi-Camera-Based Incremental Structure-From-Motion

MASS: Mobility-Aware Sensor Scheduling of Cooperative Perception for Connected Automated Driving

CAMO: Correlation-Aware Mask Optimization with Modulated Reinforcement Learning

DR.CPO: Diversified and Realistic 3D Augmentation via Iterative Construction, Random Placement, and HPR Occlusion

Metric Learning Driven Multi-Task Structured Output Optimization for Robust Keypoint Tracking