Abstract:Orthogonal matrix has shown advantages in training Recurrent Neural Networks (RNNs), but such matrix is limited to be square for the hidden-to-hidden transformation in RNNs. In this paper, we generalize such square orthogonal matrix to orthogonal rectangular matrix and formulating this problem in feed-forward Neural Networks (FNNs) as Optimization over Multiple Dependent Stiefel Manifolds (OMDSM). We show that the rectangular orthogonal matrix can stabilize the distribution of network activations and regularize FNNs. We also propose a novel orthogonal weight normalization method to solve OMDSM. Particularly, it constructs orthogonal transformation over proxy parameters to ensure the weight matrix is orthogonal and back-propagates gradient information through the transformation during training. To guarantee stability, we minimize the distortions between proxy parameters and canonical weights over all tractable orthogonal transformations. In addition, we design an orthogonal linear module (OLM) to learn orthogonal filter banks in practice, which can be used as an alternative to standard linear module. Extensive experiments demonstrate that by simply substituting OLM for standard linear module without revising any experimental protocols, our method largely improves the performance of the state-of-the-art networks, including Inception and residual networks on CIFAR and ImageNet datasets. In particular, we have reduced the test error of wide residual network on CIFAR-100 from 20.04% to 18.61% with such simple substitution. Our code is available online for result reproduction.

Growing Algorithm of Laguerre Orthogonal Basis Neural Network with Weights Directly Determined

Weights immediate determination for feed-forward neural network with Legendre orthogonal basis function

Gegenbauer orthogonal basis neural network and its weights-direct-determination method

Growing-type Weights and Structure Determination of 2-Input Legendre Orthogonal Polynomial Neuronet.

A Chebyshev Orthogonal Basis Neural Network with Direct Weight Determination

Two-Input Gegenbauer Orthogonal Neural Network with Growing-and-Pruning Weights and Structure Determination.

Bernoulli Neural Network with Weights Directly Determined and with the Number of Hidden- Layer Neurons Automatically Determined

Padé Rational Neural Network and Its Weights-Direct-Determination Method

Weights and Structure Determination of Multi-input Laguerre-orthogonal-polynomial Feed-forward Neural Network

Improve Feed-Forward Neural Network for Weights Direct Determination: a Jacobi-Polynomial Example

Euler Neural Network with Its Weight-Direct-Determination and Structure-Automatic-Determination Algorithms

On Weights and Structure Determination of Hermite-interpolation Neural Network

Feed-Forward Neural Network Activated with Hermite Orthogonal Polynomials and Its Weights-Determination Method

Approximation-Performance and Global-Convergence Analysis of Basis-Function Feedforward Neural Network

A Novel Method to Compute the Weights of Neural Networks

RBF Neural Network Classifier with Weights and Structure Determination Method

Weights and Structure Determination Method of Multiple-Input Sigmoid Activation Function Neural Network

Weights-direct-determination and Structure-Adaptive-determination of Feed-Forward Neural Network Activated with the 2nd -Class Chebyshev Orthogonal Polynomials

MISO Multivariate Generalized Polynomials Neural Network and Its Weights-Direct-Determination

Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks

A Gegenbauer Neural Network with Regularized Weights Direct Determination for Classification