Abstract:Many machine learning problems concern with discovering or associating common patterns in data of multiple views or modalities. Multi-view learning is of the methods to achieve such goals. Recent methods propose deep multi-view networks via adaptation of generic Deep Neural Networks (DNNs), which concatenate features of individual views at intermediate network layers (i.e., fusion layers). In this work, we study the problem of multi-view learning in such end-to-end networks. We take a regularization approach via multi-view learning criteria, and propose a novel, effective, and efficient neuron-wise correlation-maximizing regularizer. We implement our proposed regularizers collectively as a correlation-regularized network layer (CorrReg). CorrReg can be applied to either fully-connected or convolutional fusion layers, simply by replacing them with their CorrReg counterparts. By partitioning neurons of a hidden layer in generic DNNs into multiple subsets, we also consider a multi-view feature learning perspective of generic DNNs. Such a perspective enables us to study deep multi-view learning in the context of regularized network training, for which we present control experiments of benchmark image classification to show the efficacy of our proposed CorrReg. To investigate how CorrReg is useful for practical multi-view learning problems, we conduct experiments of RGB-D object/scene recognition and multi-view based 3D object recognition, using networks with fusion layers that concatenate intermediate features of individual modalities or views for subsequent classification. Applying CorrReg to fusion layers of these networks consistently improves classification performance. In particular, we achieve the new state of the art on the benchmark RGB-D object and RGB-D scene datasets. We make the implementation of CorrReg publicly available.

Learning Discriminated and Correlated Patches for Multi-View Object Detection Using Sparse Coding

Multi-View Correlated Feature Learning by Uncovering Shared Component.

Multiview Hessian Discriminative Sparse Coding for Image Annotation.

Feature-Correlation based multi-view detection

Novel Framework for Multi-view Object Detection through Combining Multiple Classifiers

Co-Learning Non-Negative Correlated and Uncorrelated Features for Multi-View Data

Multiple Discrimination and Pairwise CNN for view-based 3D object retrieval

3D Object Detection Based on Multi-View Feature Point Matching

Learning Coarse-To-Fine Sparselets For Efficient Object Detection And Scene Classification

Multiclass Object Detection by Combining Local Appearances and Context

Multi-View correlation distillation for incremental object detection

Multi-patch Feature Pyramid Network for Weakly Supervised Object Detection in Optical Remote Sensing Images

Deep Multi-View Learning using Neuron-Wise Correlation-Maximizing Regularizers

Learning Disentangled Representation for Multi-View 3D Object Recognition.

Multi-modal Deep Feature Learning for RGB-D Object Detection

Multi-view Latent Space Learning Based on Local Discriminant Embedding

Multi-view Common Component Discriminant Analysis for Cross-view Classification

Multi-level Discriminative Dictionary Learning with Application to Large Scale Image Classification.

Two-stage Patch-Based Sparse Multi-Value Descriptor for Face Recognition

Laplacian multiset canonical correlations for multiview feature extraction and image recognition

SketchDesc: Learning Local Sketch Descriptors for Multi-View Correspondence