Abstract:Learning to estimate 3D geometry in a single frame and optical flow from consecutive frames by watching unlabeled videos via deep convolutional network has made significant progress recently. Current state-of-the-art (SoTA) methods treat the two tasks independently. One typical assumption of the existing depth estimation methods is that the scenes contain no independent moving objects. while object moving could be easily modeled using optical flow. In this paper, we propose to address the two tasks as a whole, i.e., to jointly understand per-pixel 3D geometry and motion. This eliminates the need of static scene assumption and enforces the inherent geometrical consistency during the learning process, yielding significantly improved results for both tasks. We call our method as "Every Pixel Counts++" or "EPC++". Specifically, during training, given two consecutive frames from a video, we adopt three parallel networks to predict the camera motion (MotionNet), dense depth map (DepthNet), and per-pixel optical flow between two frames (OptFlowNet) respectively. The three types of information, are fed into a holistic 3D motion parser (HMP), and per-pixel 3D motion of both rigid background and moving objects are disentangled and recovered. Various loss terms are formulated to jointly supervise the three networks. An effective adaptive training strategy is proposed to achieve better performance and more efficient convergence. Comprehensive experiments were conducted on datasets with different scenes, including driving scenario (KITTI 2012 and KITTI 2015 datasets), mixed outdoor/indoor scenes (Make3D) and synthetic animation (MPI Sintel dataset). Performance on the five tasks of depth estimation, optical flow estimation, odometry, moving object segmentation and scene flow estimation shows that our approach outperforms other SoTA methods, demonstrating the effectiveness of each module of our proposed -ethod. Code will be available at: https://github.com/chenxuluo/EPC.

Flow-guided Feature Propagation with Occlusion Aware Detail Enhancement for Hand Segmentation in Egocentric Videos.

Robust Bilayer Segmentation and Motion/Depth Estimation with a Handheld Camera

Spatio-Temporal Segmentation with Depth-Inferred Videos of Static Scenes

Spatio-Temporal Video Segmentation of Static Scenes and Its Applications

Optical Flow-Guided Mask Generation Network For Video Segmentation

Coarse-to-Fine Video Instance Segmentation With Factorized Conditional Appearance Flows

Hand Segmentation With Dense Dilated U-Net and Structurally Incoherent Nonnegative Matrix Factorization-Based Gesture Recognition

Flow-based Video Segmentation for Human Head and Shoulders

Object Detection and Tracking under Occlusion for Object-Level RGB-D Video Segmentation

Generative Adversarial Network for Future Hand Segmentation from Egocentric Video

SPMHand: Segmentation-guided Progressive Multi-path 3D Hand Pose and Shape Estimation

Deep Features Homography Transformation Fusion Network-A Universal Foreground Segmentation Algorithm for PTZ Cameras and a Comparative Study

Fingertip in the Eye: A cascaded CNN pipeline for the real-time fingertip detection in egocentric videos

Moving Object Proposals with Deep Learned Optical Flow for Video Object Segmentation

Lightweight real-time hand segmentation leveraging MediaPipe landmark detection

Fast and Robust Dynamic Hand Gesture Recognition via Key Frames Extraction and Feature Fusion

USegScene: Unsupervised Learning of Depth, Optical Flow and Ego-Motion with Semantic Guidance and Coupled Networks

Flow-guided Feature Enhancement Network for Video-Based Person Re-Identification

Beyond One Glance: Gated Recurrent Architecture for Hand Segmentation

Hierarchical Feature Alignment Network for Unsupervised Video Object Segmentation

Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding