Morgan Jones,Matthew M. Peet
Abstract:We consider a general class of Dynamic Programming (DP) problems with non-separable objective functions. We show that for any problem in this class, there exists an augmented-state DP problem which satisfies the Principle of Optimality and the solutions to which yield solutions to the original problem. Furthermore, we identify a subclass of DP problems with Naturally Forward Separable (NFS) objective functions for which this state-augmentation scheme is tractable. We extend this framework to stochastic DP problems, proposing a suitable definition of the Principle of Optimality. We then apply the resulting algorithms to the problem of optimal battery scheduling with demand charges using a data-based stochastic model for electricity usage and solar generation by the consumer.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to handle optimization problems with non - additively separable objective functions in the Dynamic Programming (DP) framework. Specifically, the paper focuses on how to transform a DP problem that originally does not satisfy the Principle of Optimality into a new DP problem that satisfies the Principle of Optimality through the method of state augmentation, so that it can be solved using Bellman's Equation. This method is particularly applicable to problems in areas such as battery scheduling, demand charges, and renewable energy integration.
### Core Problems of the Paper
- **Non - additively Separable Objective Function**: In many practical applications, such as battery scheduling problems, the objective function may contain maximum - value terms (e.g., demand charges). Such objective functions are non - additively separable, that is, they cannot be simply expressed as the sum of costs at each stage. Therefore, traditional dynamic programming methods cannot be directly applied to these problems.
- **Lack of the Principle of Optimality**: Due to the existence of non - additively separable objective functions, these problems usually do not satisfy the Principle of Optimality, which means that the traditional Bellman's equation cannot directly solve these optimization problems.
### Solutions
- **State Augmentation**: The paper proposes a state augmentation method. By introducing additional state variables to expand the state space of the original problem, the new problem has an additively separable objective function and satisfies the Principle of Optimality. In this way, Bellman's equation can be used to solve the new problem, and then the optimal solution of the original problem can be obtained.
- **Naturally Forward Separable (NFS)**: To ensure that the DP problem after state augmentation is computationally feasible, the paper defines a class of objective functions called "Naturally Forward Separable (NFS)". The characteristic of these functions is that their representation dimension is independent of the number of time steps, thus avoiding the "curse of dimensionality".
### Application Examples
- **Battery Scheduling Problem**: The paper applies the above - mentioned methods to the battery scheduling problem, especially when considering demand charges and renewable energy fluctuations. By constructing a stochastic model based on the Gauss - Markov process, the paper shows how to use the state augmentation technique to optimize the battery scheduling strategy to minimize energy costs and maximum power consumption.
### Main Contributions
- **Theoretical Framework**: Proposes a general framework for dealing with non - additively separable objective functions, especially the concepts of the state augmentation method and NFS functions.
- **Algorithm Implementation**: Provides a specific algorithm implementation that can be used to solve battery scheduling problems in practice.
- **Practical Application**: Through specific case studies, verifies the effectiveness and practicality of the proposed method.
### Formula Examples
- **Non - additively Separable Objective Function**:
\[
J(u, x)=\sum_{t = t_0}^{T - 1}c_t(x(t), u(t))+\max_{t_0\leq k\leq T}d_k(x(k))
\]
- **Objective Function after State Augmentation**:
\[
L_t(u, z)=\phi_T(z_1(T), z_2(T))
\]
where
\[
\begin{bmatrix}
z_1(t + 1)\\
z_2(t + 1)
\end{bmatrix}=
\begin{bmatrix}
f(z_1(t), u(t), t)\\
\phi_t(z_1(t), u(t), z_2(t))
\end{bmatrix}
\]
Through these methods, the paper successfully addresses the challenges brought by non - additively separable objective functions and provides a powerful tool for practical applications in related fields.