Abstract:It has been widely observed that many activation functions and pooling methods of neural network models have (positive-) rescaling-invariant property, including ReLU, PReLU, max-pooling, and average pooling, which makes fully-connected neural networks (FNNs) and convolutional neural networks (CNNs) invariant to (positive) rescaling operation across layers. This may cause unneglectable problems with their optimization: (1) different NN models could be equivalent, but their gradients can be very different from each other; (2) it can be proven that the loss functions may have many spurious critical points in the redundant weight space. To tackle these problems, in this paper, we first characterize the rescaling-invariant properties of NN models using equivalent classes and prove that the dimension of the equivalent class space is significantly smaller than the dimension of the original weight space. Then we represent the loss function in the compact equivalent class space and develop novel algorithms that conduct optimization of the NN models directly in the equivalent class space. We call these algorithms Equivalent Class Optimization (abbreviated as EC-Opt) algorithms. Moreover, we design efficient tricks to compute the gradients in the equivalent class, which almost have no extra computational complexity as compared to standard back-propagation (BP). We conducted experimental study to demonstrate the effectiveness of our proposed new optimization algorithms. In particular, we show that by using the idea of EC-Opt, we can significantly improve the accuracy of the learned model (for both FNN and CNN), as compared to using conventional stochastic gradient descent algorithms.

A Universal Approximation Result for Difference of log-sum-exp Neural Networks

Log-sum-exp neural networks and posynomial models for convex and log-log-convex data

Convergence Rates for A Class of Neural Networks with Logarithmic Function

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Why Deep Neural Networks for Function Approximation?

Universal approximation results for neural networks with non-polynomial activation function over non-compact domains

Lp Simultaneous Approximation by Neural Networks with One Hidden Layer

Universal Approximation with Convex Optimization: Gimmick or Reality? [discussion Forum].

A general approximation lower bound in $L^p$ norm, with applications to feed-forward neural networks

Optimal function approximation with ReLU neural networks

Universal Approximation of Parametric Optimization via Neural Networks with Piecewise Linear Policy Approximation

Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?

A Universal Approximation Theorem for Mixture of Experts Models

New advances in universal approximation with neural networks of minimal width

A new approach to generalisation error of machine learning algorithms: Estimates and convergence

A continuum among logarithmic, linear, and exponential functions, and its potential to improve generalization in neural networks

Universal Approximation of Multiple Nonlinear Operators by Neural Networks

Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators

Optimizing Neural Networks in the Equivalent Class Space.

On the approximation by single hidden layer feedforward neural networks with fixed weights