Abstract:Neural networks with rectified linear unit (ReLU) activation functions (a.k.a. ReLU networks) have achieved great empirical success in various domains. Nonetheless, existing results for learning ReLU networks either pose assumptions on the underlying data distribution being, e.g., Gaussian, or require the network size and/or training size to be sufficiently large. In this context, the problem of learning a two-layer ReLU network is approached in a binary classification setting, where the data are linearly separable and a hinge loss criterion is adopted. Leveraging the power of random noise perturbation, this paper presents a novel stochastic gradient descent (SGD) algorithm, which can provably train any single-hidden-layer ReLU network to attain global optimality, despite the presence of infinitely many bad local minima, maxima, and saddle points in general. This result is the first of its kind, requiring no assumptions on the data distribution, training/network size, or initialization. Convergence of the resultant iterative algorithm to a global minimum is analyzed by establishing both an upper bound and a lower bound on the number of non-zero updates to be performed. Moreover, generalization guarantees are developed for ReLU networks trained with the novel SGD leveraging classic compression bounds. These guarantees highlight a key difference (at least in the worst case) between reliably learning a ReLU network as well as a leaky ReLU network in terms of sample complexity. Numerical tests using both synthetic data and real images validate the effectiveness of the algorithm and the practical merits of the theory.

Stability and Performance Analysis of Discrete-Time ReLU Recurrent Neural Networks

Improving Fault Tolerance for Reliable DNN Using Boundary-Aware Activation

A Complete Set of Quadratic Constraints for Repeated ReLU and Generalizations

Stability Analysis of Feedback Systems with ReLU Nonlinearities via Semialgebraic Set Representation

ReLU soothes the NTK condition number and accelerates optimization for wide neural networks

Neural networks with ReLU powers need less depth

Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes

Stability analysis of a class of discrete-time recurrent neural networks: An LMI approach

Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalization

Stability Analysis of Recurrent Neural Networks by IQC with Copositive Mutipliers

Improved Stability Criteria of Generalized Recurrent Neural Networks with Time-varying Delays

Simplified Exponential Stability Analysis for Recurrent Neural Networks with Discrete and Distributed Time-Varying Delays

Improving performance of recurrent neural network with relu nonlinearity

Robust Stability Of Neural Networks With Discontinuous Activation Functions And Time-Varying Delays

Improved Delay-Dependent Stability Results of Recurrent Neural Networks

New Stability Results of Discrete-Time Recurrent Neural Networks with Time-Varying Delays

Stability Analysis of Recurrent Neural Networks with Time-Varying Delay and Disturbances Via Quadratic Convex Technique

Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networks

ReLUs Are Sufficient for Learning Implicit Neural Representations

Stability Analysis for Delayed Neural Networks via a Generalized Reciprocally Convex Inequality

Stability Analysis of Delayed Recurrent Neural Networks Based on a Flexible Terminal Inequality