Abstract:Many machine learning problems concern with discovering or associating common patterns in data of multiple views or modalities. Multi-view learning is of the methods to achieve such goals. Recent methods propose deep multi-view networks via adaptation of generic Deep Neural Networks (DNNs), which concatenate features of individual views at intermediate network layers (i.e., fusion layers). In this work, we study the problem of multi-view learning in such end-to-end networks. We take a regularization approach via multi-view learning criteria, and propose a novel, effective, and efficient neuron-wise correlation-maximizing regularizer. We implement our proposed regularizers collectively as a correlation-regularized network layer (CorrReg). CorrReg can be applied to either fully-connected or convolutional fusion layers, simply by replacing them with their CorrReg counterparts. By partitioning neurons of a hidden layer in generic DNNs into multiple subsets, we also consider a multi-view feature learning perspective of generic DNNs. Such a perspective enables us to study deep multi-view learning in the context of regularized network training, for which we present control experiments of benchmark image classification to show the efficacy of our proposed CorrReg. To investigate how CorrReg is useful for practical multi-view learning problems, we conduct experiments of RGB-D object/scene recognition and multi-view based 3D object recognition, using networks with fusion layers that concatenate intermediate features of individual modalities or views for subsequent classification. Applying CorrReg to fusion layers of these networks consistently improves classification performance. In particular, we achieve the new state of the art on the benchmark RGB-D object and RGB-D scene datasets. We make the implementation of CorrReg publicly available.

Regularizing Deep Convolutional Neural Networks with a Structured Decorrelation Constraint.

Structured Pruning for Efficient Convolutional Neural Networks Via Incremental Regularization

SparseConnect: Regularising CNNs on Fully Connected Layers

Wordreg: Mitigating the Gap Between Training and Inference with Worst-Case Drop Regularization

Subdomain contraction in deep networks for robust representation learning

Structured Deep Neural Network Pruning by Varying Regularization Parameters.

SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction

Overfitting Remedy by Sparsifying Regularization on Fully-Connected Layers of CNNs.

Improving Deep Neural Network Sparsity Through Decorrelation Regularization

Learning Structured Sparsity in Deep Neural Networks

Network as Regularization for Training Deep Neural Networks: Framework, Model and Performance

Reducing Overfitting in Deep Networks by Decorrelating Representations

Structured pruning for group regularized convolutional neural networks via dynamic regularization factor

Deep Multi-View Learning using Neuron-Wise Correlation-Maximizing Regularizers

Adaptive Structured Sparse Network for Efficient CNNs with Feature Regularization.

Structure Regularization for Structured Prediction.

62 43 v 2 [ cs . L G ] 30 J an 2 01 5 Structure Regularization for Structured Prediction : Theories and Experiments

Structure Regularization for Structured Prediction: Theories and Experiments

Correlation-based Structural Dropout for Convolutional Neural Networks

Regularization graph convolutional networks with data augmentation

Adaptive Structured Sparse Network for Efficient CNNs with Feature Regularization