Norm-Based Capacity Control in Neural Networks

Behnam Neyshabur,Ryota Tomioka,Nathan Srebro

DOI: https://doi.org/10.48550/arXiv.1503.00036

2015-04-15

Abstract:We investigate the capacity, convexity and characterization of a general family of norm-constrained feed-forward networks.

Machine Learning,Artificial Intelligence,Neural and Evolutionary Computing

What problem does this paper attempt to address?

Capacity Control of ReLU Neural Networks by Basis-path Norm

Shuxin Zheng,Qi Meng,Huishuai Zhang,Wei Chen,Nenghai Yu,Tie-Yan Liu

DOI: https://doi.org/10.48550/arXiv.1809.07122

2018-09-19

Abstract:Recently, path norm was proposed as a new capacity measure for neural networks with Rectified Linear Unit (ReLU) activation function, which takes the rescaling-invariant property of ReLU into account. It has been shown that the generalization error bound in terms of the path norm explains the empirical generalization behaviors of the ReLU neural networks better than that of other capacity measures. Moreover, optimization algorithms which take path norm as the regularization term to the loss function, like Path-SGD, have been shown to achieve better generalization performance. However, the path norm counts the values of all paths, and hence the capacity measure based on path norm could be improperly influenced by the dependency among different paths. It is also known that each path of a ReLU network can be represented by a small group of linearly independent basis paths with multiplication and division operation, which indicates that the generalization behavior of the network only depends on only a few basis paths. Motivated by this, we propose a new norm \emph{Basis-path Norm} based on a group of linearly independent paths to measure the capacity of neural networks more accurately. We establish a generalization error bound based on this basis path norm, and show it explains the generalization behaviors of ReLU networks more accurately than previous capacity measures via extensive experiments. In addition, we develop optimization algorithms which minimize the empirical risk regularized by the basis-path norm. Our experiments on benchmark datasets demonstrate that the proposed regularization method achieves clearly better performance on the test set than the previous regularization approaches.

Machine Learning
Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks

Fanghui Liu,Leello Dadi,Volkan Cevher

2024-06-26

Abstract:Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks as the curse of dimensionality (CoD) cannot be evaded when trying to approximate even a single ReLU neuron (Bach, 2017). In this paper, we study a suitable function space for over-parameterized two-layer neural networks with bounded norms (e.g., the path norm, the Barron norm) in the perspective of sample complexity and generalization properties. First, we show that the path norm (as well as the Barron norm) is able to obtain width-independence sample complexity bounds, which allows for uniform convergence guarantees. Based on this result, we derive the improved result of metric entropy for $\epsilon$-covering up to $O(\epsilon^{-\frac{2d}{d+2}})$ ($d$ is the input dimension and the depending constant is at most linear order of $d$) via the convex hull technique, which demonstrates the separation with kernel methods with $\Omega(\epsilon^{-d})$ to learn the target function in a Barron space. Second, this metric entropy result allows for building a sharper generalization bound under a general moment hypothesis setting, achieving the rate at $O(n^{-\frac{d+2}{2d+2}})$. Our analysis is novel in that it offers a sharper and refined estimation for metric entropy with a linear dimension dependence and unbounded sampling in the estimation of the sample error and the output error.

Machine Learning
A Capacity Scaling Law for Artificial Neural Networks

Gerald Friedland,Mario Krell

DOI: https://doi.org/10.48550/arXiv.1708.06019

2018-09-10

Abstract:We derive the calculation of two critical numbers predicting the behavior of perceptron networks. First, we derive the calculation of what we call the lossless memory (LM) dimension. The LM dimension is a generalization of the Vapnik--Chervonenkis (VC) dimension that avoids structured data and therefore provides an upper bound for perfectly fitting almost any training data. Second, we derive what we call the MacKay (MK) dimension. This limit indicates a 50% chance of not being able to train a given function. Our derivations are performed by embedding a neural network into Shannon's communication model which allows to interpret the two points as capacities measured in bits. We present a proof and practical experiments that validate our upper bounds with repeatable experiments using different network configurations, diverse implementations, varying activation functions, and several learning algorithms. The bottom line is that the two capacity points scale strictly linear with the number of weights. Among other practical applications, our result allows to compare and benchmark different neural network implementations independent of a concrete learning task. Our results provide insight into the capabilities and limits of neural networks and generate valuable know how for experimental design decisions.

Neural and Evolutionary Computing,Machine Learning
Hard-Constrained Neural Networks with Universal Approximation Guarantees

Youngjae Min,Anoopkumar Sonar,Navid Azizan

2024-10-15

Abstract:Incorporating prior knowledge or specifications of input-output relationships into machine learning models has gained significant attention, as it enhances generalization from limited data and leads to conforming outputs. However, most existing approaches use soft constraints by penalizing violations through regularization, which offers no guarantee of constraint satisfaction -- an essential requirement in safety-critical applications. On the other hand, imposing hard constraints on neural networks may hinder their representational power, adversely affecting performance. To address this, we propose HardNet, a practical framework for constructing neural networks that inherently satisfy hard constraints without sacrificing model capacity. Specifically, we encode affine and convex hard constraints, dependent on both inputs and outputs, by appending a differentiable projection layer to the network's output. This architecture allows unconstrained optimization of the network parameters using standard algorithms while ensuring constraint satisfaction by construction. Furthermore, we show that HardNet retains the universal approximation capabilities of neural networks. We demonstrate the versatility and effectiveness of HardNet across various applications: fitting functions under constraints, learning optimization solvers, optimizing control policies in safety-critical systems, and learning safe decision logic for aircraft systems.

Machine Learning,Artificial Intelligence
Capacity allocation analysis of neural networks: A tool for principled architecture design

Jonathan Donier

DOI: https://doi.org/10.48550/arXiv.1902.04485

2019-02-13

Abstract:Designing neural network architectures is a task that lies somewhere between science and art. For a given task, some architectures are eventually preferred over others, based on a mix of intuition, experience, experimentation and luck. For many tasks, the final word is attributed to the loss function, while for some others a further perceptual evaluation is necessary to assess and compare performance across models. In this paper, we introduce the concept of capacity allocation analysis, with the aim of shedding some light on what network architectures focus their modelling capacity on, when used on a given task. We focus more particularly on spatial capacity allocation, which analyzes a posteriori the effective number of parameters that a given model has allocated for modelling dependencies on a given point or region in the input space, in linear settings. We use this framework to perform a quantitative comparison between some classical architectures on various synthetic tasks. Finally, we consider how capacity allocation might translate in non-linear settings.

Machine Learning
Approximation Bounds for Norm Constrained Neural Networks with Applications to Regression and GANs

Yuling Jiao,Yang Wang,Yunfei Yang

DOI: https://doi.org/10.1016/j.acha.2023.03.004

2024-01-01

Abstract:This paper studies the approximation capacity of ReLU neural networks with norm constraint on the weights. We prove upper and lower bounds on the approximation error of these networks for smooth function classes. The lower bound is derived through the Rademacher complexity of neural networks, which may be of independent interest. We apply these approximation bounds to analyze the convergences of regression using norm constrained neural networks and distribution estimation by GANs. In particular, we obtain convergence rates for over-parameterized neural networks. It is also shown that GANs can achieve optimal rate of learning probability distributions, when the discriminator is a properly chosen norm constrained neural network.
Convex neural network synthesis for robustness in the 1-norm

Ross Drummond,Chris Guiver,Matthew C. Turner

2024-05-29

Abstract:With neural networks being used to control safety-critical systems, they increasingly have to be both accurate (in the sense of matching inputs to outputs) and robust. However, these two properties are often at odds with each other and a trade-off has to be navigated. To address this issue, this paper proposes a method to generate an approximation of a neural network which is certifiably more robust. Crucially, the method is fully convex and posed as a semi-definite programme. An application to robustifying model predictive control is used to demonstrate the results. The aim of this work is to introduce a method to navigate the neural network robustness/accuracy trade-off.

Systems and Control,Artificial Intelligence
Channel Capacity of Neural Networks

Gen Ye,Tong Lin

DOI: https://doi.org/10.1007/978-3-030-86380-7_21

2021-01-01

Abstract:Occam's Razor principle suggests preference for simpler models and triggers an enduring question: what is the proper definition of complexity of a model? In this work, we regard neural networks as communication channels and measure the complexity of neural networks by means of their channel capacity-the maximum information reserved in the output of a neural network. Furthermore, we show a connection between the L2-norm of the weight matrix of the linear model and its channel capacity through the singular values of the weight matrix. On image classification problems, we find regularizing different neural networks by constraining their channel capacity effectively boosts the generalization performance and outperforms other information-theoretic regularization methods.
A provable control of sensitivity of neural networks through a direct parameterization of the overall bi-Lipschitzness

Yuri Kinoshita,Taro Toyoizumi

2024-04-15

Abstract:While neural networks can enjoy an outstanding flexibility and exhibit unprecedented performance, the mechanism behind their behavior is still not well-understood. To tackle this fundamental challenge, researchers have tried to restrict and manipulate some of their properties in order to gain new insights and better control on them. Especially, throughout the past few years, the concept of \emph{bi-Lipschitzness} has been proved as a beneficial inductive bias in many areas. However, due to its complexity, the design and control of bi-Lipschitz architectures are falling behind, and a model that is precisely designed for bi-Lipschitzness realizing a direct and simple control of the constants along with solid theoretical analysis is lacking. In this work, we investigate and propose a novel framework for bi-Lipschitzness that can achieve such a clear and tight control based on convex neural networks and the Legendre-Fenchel duality. Its desirable properties are illustrated with concrete experiments. We also apply this framework to uncertainty estimation and monotone problem settings to illustrate its broad range of applications.

Machine Learning
Control of neural transport for normalizing flows

Domènec Ruiz-Balet,Enrique Zuazua

2023-08-02

Abstract:Inspired by normalizing flows, we analyze the bilinear control of neural transport equations by means of time-dependent velocity fields restricted to fulfill, at any time instance, a simple neural network ansatz. The L^1 approximate controllability property is proved, showing that any probability density can be driven arbitrarily close to any other one in any time horizon. The control vector fields are built explicitly and inductively and this provides quantitative estimates on their complexity and amplitude. This also leads to statistical error bounds when only random samples of the target probability density are available.

Optimization and Control
Tight Certified Robustness via Min-Max Representations of ReLU Neural Networks

Brendon G. Anderson,Samuel Pfrommer,Somayeh Sojoudi

2023-10-08

Abstract:The reliable deployment of neural networks in control systems requires rigorous robustness guarantees. In this paper, we obtain tight robustness certificates over convex attack sets for min-max representations of ReLU neural networks by developing a convex reformulation of the nonconvex certification problem. This is done by "lifting" the problem to an infinite-dimensional optimization over probability measures, leveraging recent results in distributionally robust optimization to solve for an optimal discrete distribution, and proving that solutions of the original nonconvex problem are generated by the discrete distribution under mild boundedness, nonredundancy, and Slater conditions. As a consequence, optimal (worst-case) attacks against the model may be solved for exactly. This contrasts prior state-of-the-art that either requires expensive branch-and-bound schemes or loose relaxation techniques. Experiments on robust control and MNIST image classification examples highlight the benefits of our approach.

Optimization and Control,Machine Learning
Polysemanticity and Capacity in Neural Networks

Adam Scherlis,Kshitij Sachan,Adam S. Jermyn,Joe Benton,Buck Shlegeris

2023-07-12

Abstract:Individual neurons in neural networks often represent a mixture of unrelated features. This phenomenon, called polysemanticity, can make interpreting neural networks more difficult and so we aim to understand its causes. We propose doing so through the lens of feature \emph{capacity}, which is the fractional dimension each feature consumes in the embedding space. We show that in a toy model the optimal capacity allocation tends to monosemantically represent the most important features, polysemantically represent less important features (in proportion to their impact on the loss), and entirely ignore the least important features. Polysemanticity is more prevalent when the inputs have higher kurtosis or sparsity and more prevalent in some architectures than others. Given an optimal allocation of capacity, we go on to study the geometry of the embedding space. We find a block-semi-orthogonal structure, with differing block sizes in different models, highlighting the impact of model architecture on the interpretability of its neurons.

Neural and Evolutionary Computing,Artificial Intelligence,Machine Learning
AdaCap: Adaptive Capacity control for Feed-Forward Neural Networks

Katia Meziani,Karim Lounici,Benjamin Riu

DOI: https://doi.org/10.48550/arXiv.2205.07860

IF: 5.414

2022-05-09

Machine Learning

Abstract:The capacity of a ML model refers to the range of functions this model can approximate. It impacts both the complexity of the patterns a model can learn but also memorization, the ability of a model to fit arbitrary labels. We propose Adaptive Capacity (AdaCap), a training scheme for Feed-Forward Neural Networks (FFNN). AdaCap optimizes the capacity of FFNN so it can capture the high-level abstract representations underlying the problem at hand without memorizing the training dataset. AdaCap is the combination of two novel ingredients, the Muddling labels for Regularization (MLR) loss and the Tikhonov operator training scheme. The MLR loss leverages randomly generated labels to quantify the propensity of a model to memorize. We prove that the MLR loss is an accurate in-sample estimator for out-of-sample generalization performance and that it can be used to perform Hyper-Parameter Optimization provided a Signal-to-Noise Ratio condition is met. The Tikhonov operator training scheme modulates the capacity of a FFNN in an adaptive, differentiable and data-dependent manner. We assess the effectiveness of AdaCap in a setting where DNN are typically prone to memorization, small tabular datasets, and benchmark its performance against popular machine learning methods.
Bounding the Rademacher complexity of Fourier neural operators

Taeyoung Kim,Myungjoo Kang

DOI: https://doi.org/10.1007/s10994-024-06533-y

IF: 5.414

2024-03-28

Machine Learning

Abstract:Recently, several types of neural operators have been developed, including deep operator networks, graph neural operators, and Multiwavelet-based operators. Compared with these models, the Fourier neural operator (FNO), a physics-inspired machine learning method, is computationally efficient and can learn nonlinear operators between function spaces independent of a certain finite basis. This study investigated the bounding of the Rademacher complexity of the FNO based on specific group norms. Using capacity based on these norms, we bound the generalization error of the model. In addition, we investigate the correlation between the empirical generalization error and the proposed capacity of FNO. We infer that the type of group norm determines the information about the weights and architecture of the FNO model stored in capacity. The experimental results offer insight into the impact of the number of modes used in the FNO model on the generalization error. The results confirm that our capacity is an effective index for estimating generalization errors.

computer science, artificial intelligence
The Sample Complexity of One-Hidden-Layer Neural Networks

Gal Vardi,Ohad Shamir,Nathan Srebro

DOI: https://doi.org/10.48550/arXiv.2202.06233

2022-09-22

Abstract:We study norm-based uniform convergence bounds for neural networks, aiming at a tight understanding of how these are affected by the architecture and type of norm constraint, for the simple class of scalar-valued one-hidden-layer networks, and inputs bounded in Euclidean norm. We begin by proving that in general, controlling the spectral norm of the hidden layer weight matrix is insufficient to get uniform convergence guarantees (independent of the network width), while a stronger Frobenius norm control is sufficient, extending and improving on previous work. Motivated by the proof constructions, we identify and analyze two important settings where (perhaps surprisingly) a mere spectral norm control turns out to be sufficient: First, when the network's activation functions are sufficiently smooth (with the result extending to deeper networks); and second, for certain types of convolutional networks. In the latter setting, we study how the sample complexity is additionally affected by parameters such as the amount of overlap between patches and the overall number of patches.

Machine Learning
Network properties determine neural network performance

Chunheng Jiang,Zhenhan Huang,Tejaswini Pedapati,Pin-Yu Chen,Yizhou Sun,Jianxi Gao

DOI: https://doi.org/10.1038/s41467-024-48069-8

2024-07-08

Abstract:Machine learning influences numerous aspects of modern society, empowers new technologies, from Alphago to ChatGPT, and increasingly materializes in consumer products such as smartphones and self-driving cars. Despite the vital role and broad applications of artificial neural networks, we lack systematic approaches, such as network science, to understand their underlying mechanism. The difficulty is rooted in many possible model configurations, each with different hyper-parameters and weighted architectures determined by noisy data. We bridge the gap by developing a mathematical framework that maps the neural network's performance to the network characters of the line graph governed by the edge dynamics of stochastic gradient descent differential equations. This framework enables us to derive a neural capacitance metric to universally capture a model's generalization capability on a downstream task and predict model performance using only early training results. The numerical results on 17 pre-trained ImageNet models across five benchmark datasets and one NAS benchmark indicate that our neural capacitance metric is a powerful indicator for model selection based only on early training results and is more efficient than state-of-the-art methods.
A Convex Relaxation Approach to Generalization Analysis for Parallel Positively Homogeneous Networks

Uday Kiran Reddy Tadipatri,Benjamin D. Haeffele,Joshua Agterberg,René Vidal

2024-11-05

Abstract:We propose a general framework for deriving generalization bounds for parallel positively homogeneous neural networks--a class of neural networks whose input-output map decomposes as the sum of positively homogeneous maps. Examples of such networks include matrix factorization and sensing, single-layer multi-head attention mechanisms, tensor factorization, deep linear and ReLU networks, and more. Our general framework is based on linking the non-convex empirical risk minimization (ERM) problem to a closely related convex optimization problem over prediction functions, which provides a global, achievable lower-bound to the ERM problem. We exploit this convex lower-bound to perform generalization analysis in the convex space while controlling the discrepancy between the convex model and its non-convex counterpart. We apply our general framework to a wide variety of models ranging from low-rank matrix sensing, to structured matrix sensing, two-layer linear networks, two-layer ReLU networks, and single-layer multi-head attention mechanisms, achieving generalization bounds with a sample complexity that scales almost linearly with the network width.

Machine Learning,Signal Processing
Robust and Provably Monotonic Networks

Ouail Kitouni,Niklas Nolte,Mike Williams

DOI: https://doi.org/10.1088/2632-2153/aced80

2023-03-16

Abstract:The Lipschitz constant of the map between the input and output space represented by a neural network is a natural metric for assessing the robustness of the model. We present a new method to constrain the Lipschitz constant of dense deep learning models that can also be generalized to other architectures. The method relies on a simple weight normalization scheme during training that ensures the Lipschitz constant of every layer is below an upper limit specified by the analyst. A simple monotonic residual connection can then be used to make the model monotonic in any subset of its inputs, which is useful in scenarios where domain knowledge dictates such dependence. Examples can be found in algorithmic fairness requirements or, as presented here, in the classification of the decays of subatomic particles produced at the CERN Large Hadron Collider. Our normalization is minimally constraining and allows the underlying architecture to maintain higher expressiveness compared to other techniques which aim to either control the Lipschitz constant of the model or ensure its monotonicity. We show how the algorithm was used to train a powerful, robust, and interpretable discriminator for heavy-flavor-quark decays, which has been adopted for use as the primary data-selection algorithm in the LHCb real-time data-processing system in the current LHC data-taking period known as Run 3. In addition, our algorithm has also achieved state-of-the-art performance on benchmarks in medicine, finance, and other applications.

Machine Learning,High Energy Physics - Experiment
Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?

Li Liu,Tengchao Yu,Heng Yong

2024-09-25

Abstract:The Universal Approximation Theorem posits that neural networks can theoretically possess unlimited approximation capacity with a suitable activation function and a freely chosen or trained set of parameters. However, a more practical scenario arises when these neural parameters, especially the nonlinear weights and biases, are bounded. This leads us to question: \textbf{Does the approximation capacity of a neural network remain universal, or does it have a limit when the parameters are practically bounded? And if it has a limit, how can it be measured?} Our theoretical study indicates that while universal approximation is theoretically feasible, in practical numerical scenarios, Deep Neural Networks (DNNs) with any analytic activation functions (such as Tanh and Sigmoid) can only be approximated by a finite-dimensional vector space under a bounded nonlinear parameter space (NP space), whether in a continuous or discrete sense. Based on this study, we introduce the concepts of \textit{$\epsilon$ outer measure} and \textit{Numerical Span Dimension (NSdim)} to quantify the approximation capacity limit of a family of networks both theoretically and practically. Furthermore, drawing on our new theoretical study and adopting a fresh perspective, we strive to understand the relationship between back-propagation neural networks and random parameter networks (such as the Extreme Learning Machine (ELM)) with both finite and infinite width. We also aim to provide fresh insights into regularization, the trade-off between width and depth, parameter space, width redundancy, condensation, and other related important issues.

Machine Learning
Adaptive neural-based control for non-strict feedback systems with full-state constraints and unmodeled dynamics

Dan Ye,Yujin Cai,Haijiao Yang,Xingang Zhao

DOI: https://doi.org/10.1007/s11071-019-05008-3

IF: 5.741

2019-05-23

Nonlinear Dynamics

Abstract:In this paper, an adaptive neural network controller is designed for non-strict feedback systems with full-state constraints. According to practical applications, both input saturation and unmodeled dynamics are also taken into account. By using a logarithm nonlinear mapping, non-strict feedback systems with full-state constraints can be converted to unconstrained ones, which may result in some exponential terms. Here, a new variable separation method is proposed based on Taylor's formula to cope with the exponential terms and non-strict structure. Then, the relationship between the norm of state vector and error functions is established. A hyperbolic tangent function and a dynamic signal are introduced to deal with input saturation and unmodeled dynamics, respectively. It is proved that all signals of the closed-loop system are uniformly ultimately bounded and the requirement of full-state constraints is satisfied. Two illustrative examples are provided to demonstrate the effectiveness of the presented method.

engineering, mechanical,mechanics

Norm-Based Capacity Control in Neural Networks

Capacity Control of ReLU Neural Networks by Basis-path Norm

Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks

A Capacity Scaling Law for Artificial Neural Networks

Hard-Constrained Neural Networks with Universal Approximation Guarantees

Capacity allocation analysis of neural networks: A tool for principled architecture design

Approximation Bounds for Norm Constrained Neural Networks with Applications to Regression and GANs

Convex neural network synthesis for robustness in the 1-norm

Channel Capacity of Neural Networks

A provable control of sensitivity of neural networks through a direct parameterization of the overall bi-Lipschitzness

Control of neural transport for normalizing flows

Tight Certified Robustness via Min-Max Representations of ReLU Neural Networks

Polysemanticity and Capacity in Neural Networks

AdaCap: Adaptive Capacity control for Feed-Forward Neural Networks

Bounding the Rademacher complexity of Fourier neural operators

The Sample Complexity of One-Hidden-Layer Neural Networks

Network properties determine neural network performance

A Convex Relaxation Approach to Generalization Analysis for Parallel Positively Homogeneous Networks

Robust and Provably Monotonic Networks

Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?

Adaptive neural-based control for non-strict feedback systems with full-state constraints and unmodeled dynamics