Abstract:Feedforward neural network is one of the most commonly used function approximation techniques and has been applied to a wide variety of problems arising from various disciplines. However, neural networks are black-box models having multiple challenges/difficulties associated with training and generalization. This paper initially looks into the internal behavior of neural networks and develops a detailed interpretation of the neural network functional geometry. Based on this geometrical interpretation, a new set of variables describing neural networks is proposed as a more effective and geometrically interpretable alternative to the traditional set of network weights and biases. Then, this paper develops a new formulation for neural networks with respect to the newly defined variables; this reformulated neural network (ReNN) is equivalent to the common feedforward neural network but has a less complex error response surface. To demonstrate the learning ability of ReNN, in this paper, two training methods involving a derivative-based (a variation of backpropagation) and a derivative-free optimization algorithms are employed. Moreover, a new measure of regularization on the basis of the developed geometrical interpretation is proposed to evaluate and improve the generalization ability of neural networks. The value of the proposed geometrical interpretation, the ReNN approach, and the new regularization measure are demonstrated across multiple test problems. Results show that ReNN can be trained more effectively and efficiently compared to the common neural networks and the proposed regularization measure is an effective indicator of how a network would perform in terms of generalization.

A reformulation to Embedding a Neural Network in a linear program without integer variables

Neural Network Solution for General Nonlinear Optimization Problems

A discrete-time neural network for optimization problems with hybrid constraints.

Modeling Design and Control Problems Involving Neural Network Surrogates

Learning to Reformulate for Linear Programming

A Novel Neural Network for Nonlinear Convex Programming

A Mixed Integer Linear Programming Formulation to Artificial Neural Networks

Convex Formulations for Training Two-Layer ReLU Neural Networks

A new formulation for feedforward neural networks

Strong mixed-integer programming formulations for trained neural networks

Barrier NLP Methods with Structured Regularization for Optimization of Degenerate Optimization Problems

Optimization Over Trained Neural Networks: Taking a Relaxing Walk

Structured Regularization In Barrier Nlp For Optimization Models With Dependent Constraints

Linear programming using diagonal linear networks

Tight Neural Network Verification via Semidefinite Relaxations and Linear Reformulations

An efficient nonconvex reformulation of stagewise convex optimization problems

Towards a mathematical framework to inform neural network modelling via polynomial regression

Linearization of ReLU Activation Function for Neural Network-Embedded Optimization:Optimal Day-Ahead Energy Scheduling

A Lifted Bregman Formulation for the Inversion of Deep Neural Networks

Bound Tightening using Rolling-Horizon Decomposition for Neural Network Verification

Neural Networks and (Virtual) Extended Formulations