Abstract:Neural networks have been widely applied in the power system area. They can be used for better predicting input information and modeling system performance with increased accuracy. In some applications such as battery degradation neural network-based microgrid day-ahead energy scheduling, the input features of the trained learning model are variables to be solved in optimization models that enforce limits on the output of the same learning model. This will create a neural network-embedded optimization problem; the use of nonlinear activation functions in the neural network will make such problems extremely hard to solve if not unsolvable. To address this emerging challenge, this paper investigated different methods for linearizing the nonlinear activation functions with a particular focus on the widely used rectified linear unit (ReLU) function. Four linearization methods tailored for the ReLU activation function are developed, analyzed and compared in this paper. Each method employs a set of linear constraints to replace the ReLU function, effectively linearizing the optimization problem, which can overcome the computational challenges associated with the nonlinearity of the neural network model. These proposed linearization methods provide valuable tools for effectively solving optimization problems that integrate neural network models with ReLU activation functions.

A One-Layer Dual Recurrent Neural Network with a Heaviside Step Activation Function for Linear Programming with Its Linear Assignment Application

A one-layer recurrent neural network for nonsmooth pseudoconvex optimization with quasiconvex inequality and affine equality constraints

A One-Layer Recurrent Neural Network for Pseudoconvex Optimization Problems with Equality and Inequality Constraints.

Neural Network Solution for General Nonlinear Optimization Problems

A discrete-time neural network for optimization problems with hybrid constraints.

A One-Layer Recurrent Neural Network for Constrained Complex-Variable Convex Optimization.

Deep Neural Networks for Linear Sum Assignment Problems.

A Recurrent Neural Network with Finite-Time Convergence for Convex Quadratic Bilevel Programming Problems

A new one-layer recurrent neural network for nonsmooth pseudoconvex optimization.

Another simple recurrent neural network for quadratic and linear programming

A Two-Layer Recurrent Neural Network for Nonsmooth Convex Optimization Problems

Recurrent Neural Network for Complex-Variable Pseudoconvex Optimization with Equality Constraints

Convergence analysis of discrete time recurrent neural networks for linear variational inequality problem

A Neural Network for Solving Nonlinear Convex Programming with Linear Equality and Bounded Constraints

A Recurrent Neural Network for Solving Nonconvex Optimization Problems

Linear programming using diagonal linear networks

A new recurrent neural network for solving convex quadratic programming problems with an application to the κ-winners-take-all problem

Neural Network for Nonsmooth Pseudoconvex Optimization with General Convex Constraints

A fast algorithm for solving large scale nonlinear optimization problems using RNN

A New Delayed Projection Neural Network for Solving Linear Variational Inequalities and Quadratic Optimization Problems

Linearization of ReLU Activation Function for Neural Network-Embedded Optimization:Optimal Day-Ahead Energy Scheduling