Linearization of ReLU Activation Function for Neural Network-Embedded Optimization:Optimal Day-Ahead Energy Scheduling

Cunzhi Zhao,Xingpeng Li

2023-10-03

Abstract:Neural networks have been widely applied in the power system area. They can be used for better predicting input information and modeling system performance with increased accuracy. In some applications such as battery degradation neural network-based microgrid day-ahead energy scheduling, the input features of the trained learning model are variables to be solved in optimization models that enforce limits on the output of the same learning model. This will create a neural network-embedded optimization problem; the use of nonlinear activation functions in the neural network will make such problems extremely hard to solve if not unsolvable. To address this emerging challenge, this paper investigated different methods for linearizing the nonlinear activation functions with a particular focus on the widely used rectified linear unit (ReLU) function. Four linearization methods tailored for the ReLU activation function are developed, analyzed and compared in this paper. Each method employs a set of linear constraints to replace the ReLU function, effectively linearizing the optimization problem, which can overcome the computational challenges associated with the nonlinearity of the neural network model. These proposed linearization methods provide valuable tools for effectively solving optimization problems that integrate neural network models with ReLU activation functions.

Machine Learning,Systems and Control

What problem does this paper attempt to address?

This paper attempts to address the issue in neural network embedding optimization problems, where the use of nonlinear activation functions (particularly the ReLU function) leads to problems that are difficult or even impossible to solve. Specifically, when neural network models are used to predict input information and model system performance, if the input features of these models are decision variables in the optimization model, the optimization problem becomes very complex, especially when these models use nonlinear activation functions such as ReLU. To tackle this challenge, the paper studies different linearization methods for the ReLU function and develops, analyzes, and compares four linearization methods for the ReLU activation function. These methods effectively linearize the optimization problem by introducing a set of linear constraints to replace the ReLU function, thereby overcoming the computational difficulties brought about by the nonlinearity of neural network models. These proposed linearization methods provide valuable tools for effectively solving optimization problems that integrate neural network models.