Abstract:Many machine learning problems concern with discovering or associating common patterns in data of multiple views or modalities. Multi-view learning is of the methods to achieve such goals. Recent methods propose deep multi-view networks via adaptation of generic Deep Neural Networks (DNNs), which concatenate features of individual views at intermediate network layers (i.e., fusion layers). In this work, we study the problem of multi-view learning in such end-to-end networks. We take a regularization approach via multi-view learning criteria, and propose a novel, effective, and efficient neuron-wise correlation-maximizing regularizer. We implement our proposed regularizers collectively as a correlation-regularized network layer (CorrReg). CorrReg can be applied to either fully-connected or convolutional fusion layers, simply by replacing them with their CorrReg counterparts. By partitioning neurons of a hidden layer in generic DNNs into multiple subsets, we also consider a multi-view feature learning perspective of generic DNNs. Such a perspective enables us to study deep multi-view learning in the context of regularized network training, for which we present control experiments of benchmark image classification to show the efficacy of our proposed CorrReg. To investigate how CorrReg is useful for practical multi-view learning problems, we conduct experiments of RGB-D object/scene recognition and multi-view based 3D object recognition, using networks with fusion layers that concatenate intermediate features of individual modalities or views for subsequent classification. Applying CorrReg to fusion layers of these networks consistently improves classification performance. In particular, we achieve the new state of the art on the benchmark RGB-D object and RGB-D scene datasets. We make the implementation of CorrReg publicly available.

Diverse and consistent multi-view networks for semi-supervised regression

Multi-View Correlated Feature Learning by Uncovering Shared Component.

Deep Incomplete Multi-View Learning Network with Insufficient Label Information

UMCGL: Universal Multi-View Consensus Graph Learning With Consistency and Diversity

The similarity-consensus regularized multi-view learning for dimension reduction

Robust deep multi-view subspace clustering networks with a correntropy-induced metric

Asymmetric double-winged multi-view clustering network for exploring diverse and consistent information

Self-attention Multi-view Representation Learning with Diversity-promoting Complementarity

Multi-View Graph Learning by Joint Modeling of Consistency and Inconsistency

Deep Multi-View Learning using Neuron-Wise Correlation-Maximizing Regularizers

View-Driven Multi-View Clustering via Contrastive Double-Learning

MCoCo: Multi-level Consistency Collaborative multi-view clustering

A Survey on Multi-view Learning

Disentangling Multi-view Representations Beyond Inductive Bias

Dual Contrast-Driven Deep Multi-View Clustering

Co-Learning Non-Negative Correlated and Uncorrelated Features for Multi-View Data

Robust Multi-View Learning Via Adaptive Regression

Multi-view Collaborative Semi-Supervised Classification Algorithm Based on Diversity Measurers of Classifier with the Combination of Agreement and Disagreement Label Rules

Multi-view Deep Subspace Clustering Networks

Reliable Representations Learning for Incomplete Multi-View Partial Multi-Label Classification

Self-Supervised Discriminative Feature Learning for Deep Multi-View Clustering