Abstract:Many machine learning problems concern with discovering or associating common patterns in data of multiple views or modalities. Multi-view learning is of the methods to achieve such goals. Recent methods propose deep multi-view networks via adaptation of generic Deep Neural Networks (DNNs), which concatenate features of individual views at intermediate network layers (i.e., fusion layers). In this work, we study the problem of multi-view learning in such end-to-end networks. We take a regularization approach via multi-view learning criteria, and propose a novel, effective, and efficient neuron-wise correlation-maximizing regularizer. We implement our proposed regularizers collectively as a correlation-regularized network layer (CorrReg). CorrReg can be applied to either fully-connected or convolutional fusion layers, simply by replacing them with their CorrReg counterparts. By partitioning neurons of a hidden layer in generic DNNs into multiple subsets, we also consider a multi-view feature learning perspective of generic DNNs. Such a perspective enables us to study deep multi-view learning in the context of regularized network training, for which we present control experiments of benchmark image classification to show the efficacy of our proposed CorrReg. To investigate how CorrReg is useful for practical multi-view learning problems, we conduct experiments of RGB-D object/scene recognition and multi-view based 3D object recognition, using networks with fusion layers that concatenate intermediate features of individual modalities or views for subsequent classification. Applying CorrReg to fusion layers of these networks consistently improves classification performance. In particular, we achieve the new state of the art on the benchmark RGB-D object and RGB-D scene datasets. We make the implementation of CorrReg publicly available.

Correlative Filters For Convolutional Neural Networks

SparseConnect: Regularising CNNs on Fully Connected Layers

Research on Image Classification Method of Features of Combinatorial Convolution

Feature Correlation Loss in Convolutional Neural Networks for Image Classification

Filter Bank Regularization of Convolutional Neural Networks

Improving correlation method with convolutional neural networks

CFNN: Correlation Filter Neural Network for Visual Object Tracking

HCFNN: High-order Coverage Function Neural Network for Image Classification

Modeling Embedding Dimension Correlations Via Convolutional Neural Collaborative Filtering

Convolutional Channel Features

Correlation-based Structural Dropout for Convolutional Neural Networks

CorrNet: pearson correlation based pruning for efficient convolutional neural networks

Enhancing Generalization in Convolutional Neural Networks through Regularization with Edge and Line Features

Importance-Aware Filter Selection for Convolutional Neural Network Acceleration

When Correlation Filters Meet Convolutional Neural Networks for Visual Tracking

A Collaborative Visual Tracking Architecture for Correlation Filter and Convolutional Neural Network Learning

CovNN: A Covariance Neural Network Extended from CNN

Correlation Filters with Adaptive Convolution Response Fusion for Object Tracking.

Deep Multi-View Learning using Neuron-Wise Correlation-Maximizing Regularizers

Learning Filter Basis for Convolutional Neural Network Compression

Integrating Multi-Level Convolutional Features for Correlation Filter Tracking