Abstract:Semantic segmentation and single-view depth estimation are two fundamental problems in computer vision. They exploit the semantic and geometric properties of images, respectively, and are thus complementary in scene understanding. In this paper, we propose a collaborative deconvolutional neural network (C-DCNN) to jointly model these two problems for mutual promotion. The C-DCNN consists of two DCNNs, of which each is for one task. The DCNNs provide a finer resolution reconstruction method and are pretrained with hierarchical supervision. The feature maps from these two DCNNs are integrated via a pointwise bilinear layer, which fuses the semantic and depth information and produces higher order features. Then, the integrated features are fed into two sibling classification layers to simultaneously learn for semantic segmentation and depth estimation. In this way, we combine the semantic and depth features in a unified deep network and jointly train them to benefit each other. Specifically, during network training, we process depth estimation as a classification problem where a soft mapping strategy is proposed to map the continuous depth values into discrete probability distributions and the cross entropy loss is used. Besides, a fully connected conditional random field is also used as postprocessing to further improve the performance of semantic segmentation, where the proximity relations of pixels on position, intensity, and depth are jointly considered. We evaluate our approach on two challenging benchmarks: NYU Depth V2 and SUN RGB-D. It is demonstrated that our approach effectively utilizes these two kinds of information and achieves state-of-the-art results on both the semantic segmentation and depth estimation tasks.

Joint Learning for Face Alignment and Face Transfer with Depth Image

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks

Multi-task Convolution Network for Face Alignment

Joint 3d Face Reconstruction And Dense Face Alignment Via Deep Face Feature Alignment

UFaceNet: Research on Multi-Task Face Recognition Algorithm Based on CNN.

Layer-wise Supervised Neural Network for Face Alignment with Multi-Task Regularization

Deep Multi-Center Learning for Face Alignment

Collaborative 3D face alignment and head pose estimation with frontal face constraint based on RGB and sparse depth

Face Alignment by Deep Convolutional Network with Adaptive Learning Rate.

A Unified Deep Model for Joint Facial Expression Recognition, Face Synthesis, and Face Alignment.

Joint Learning of Semantic Alignment and Object Landmark Detection

Collaborative Learning of Depth Estimation, Visual Odometry and Camera Relocalization from Monocular Videos.

Learning Deep Representation for Face Alignment with Auxiliary Attributes

Multitask learning and CNN for application of face recognition.

Joint Multi-View Face Alignment in the Wild

Collaborative Deconvolutional Neural Networks for Joint Depth Estimation and Semantic Segmentation

Deep Joint Task Learning for Generic Object Extraction.

Deep Adaptive Attention for Joint Facial Action Unit Detection and Face Alignment

Face Verification with Multi-Task and Multi-Scale Feature Fusion

Cascaded Deep Convolutional Neural Network for Robust Face Alignment.