Dynamic Programming Deconstructed: Transformations of the Bellman Equation and Computational Efficiency

Qingyin Ma,John Stachurski

DOI: https://doi.org/10.1287/opre.2020.2006

IF: 2.7

2021-09-01

Operations Research

Abstract:Transforming Dynamic Optimization Problems for Enhanced Computational Efficiency

management,operations research & management science

What problem does this paper attempt to address?

Dynamic Optimization for Chemical Process Based on Improved Iterative Dynamic Programming Algorithm

PAN Yan,FEI Zheng-shun,ZHAO Li,LIANG Jun

DOI: https://doi.org/10.3969/j.issn.1006-3080.2013.01.011

2013-01-01

Abstract:To get the global optimization of chemical engineering dynamic system rapidly,an improved iterative dynamic programming algorithm based on first order algorithm was proposed.This algorithm used both stochastic search and deterministic search,which combined the advantages of iterative dynamic programming algorithm and first order gradient algorithm.Then the algorithm was applied to two practical examples of chemical producing.The results proved that the algorithm had advantages of precision,high efficiency and global optimization.
An Improved Quasi-Sequential Approach To Large-Scale Dynamic Process Optimization

Weirong Hong,Pengcheng Tan,Quoc Dong Vu,Pu Li

DOI: https://doi.org/10.1016/S1570-7946(09)70262-7

2009-01-01

Abstract:We present an improved quasi-sequential approach to large-scale dynamic process optimization. This new approach incorporates the quasi-sequential approach with the interior point method to handle inequality constraints. In this way the eventual optimization problem to be solved becomes a NLP without constraints. Mathematical derivations and computation schemes are developed. We first take a two-dimensional constrained optimization problem as an example; the result is compared with the simultaneous and quasi-sequential approach in terms of path solution with a graphical interpretation. A highly nonlinear reactor optimal control problem is also taken to demonstrate the effectiveness of this approach.
A quasi-sequential approach to large-scale dynamic optimization problems

Weirong Hong,Shuqing Wang,Pu Li,Günter Wozny,Lorenz T. Biegler

DOI: https://doi.org/10.1002/aic.10625

IF: 4.167

2006-01-01

AIChE Journal

Abstract:A novel sequential approach for solving dynamic optimization problems containing path constraints on state variables is presented and its performance analyzed. As in the simultaneous approach, we discretize both state and control variables using collocation on finite elements, so that path constraints can be guaranteed inside each element. The state variables are solved in a manner similar to that in the sequential approach; this eliminates the discretized differential-algebraic equations and state variables, so that the problem is reduced to a smaller problem only with inequality constraints and control variables. Therefore, it possesses advantages of both the simultaneous and the sequential approach. Furthermore, the elimination of the equality constraints substantially simplifies the line search problem and thus larger steps can be taken by successive quadratic programming (SQP) toward the optimum. We call this dynamic optimization method a quasi-sequential approach. We compare this new approach with the simultaneous approach in terms of computational cost and by analyzing the solution path. A highly nonlinear reactor control and the optimal operation of a heat-integrated column system are used to demonstrate the effectiveness of this approach. As a result, it can be concluded that this quasi-sequential approach is well suited for solving highly nonlinear large-scale optimal control problems. (c) 2005 American Institute of Chemical Engineers.
A Comparison of Arithmetic Operations for Dynamic Process Optimization Approach

Hong Weirong,Tan Pengcheng,Wang Shuqing,Li Pu

DOI: https://doi.org/10.1016/s1004-9541(08)60326-2

2010-01-01

Abstract:A comparison of arithmetic operations of two dynamic process optimization approaches called quasi-sequential approach and reduced Sequential Quadratic Programming (rSQP) simultaneous approach with respect to equality constrained optimization problems is presented. Through the detail comparison of arithmetic operations, it is concluded that the average iteration number within differential algebraic equations (DAEs) integration of quasi-sequential approach could be regarded as a criterion. One formula is given to calculate the threshold value of average iteration number. If the average iteration number is less than the threshold value, quasi-sequential approach takes advantage of rSQP simultaneous approach which is more suitable contrarily. Two optimal control problems are given to demonstrate the usage of threshold value. For optimal control problems whose objective is to stay near desired operating point, the iteration number is usually small. Therefore, quasi-sequential approach seems more suitable for such problems.
Parallel and (Nearly) Work-Efficient Dynamic Programming

Xiangyun Ding,Yan Gu,Yihan Sun

DOI: https://doi.org/10.1145/3626183.3659958

2024-05-22

Abstract:The idea of dynamic programming (DP), proposed by Bellman in the 1950s, is one of the most important algorithmic techniques. However, in parallel, many fundamental and sequentially simple problems become more challenging, and open to a (nearly) work-efficient solution (i.e., the work is off by at most a polylogarithmic factor over the best sequential solution). In fact, sequential DP algorithms employ many advanced optimizations such as decision monotonicity or special data structures, and achieve better work than straightforward solutions. Many such optimizations are inherently sequential, which creates extra challenges for a parallel algorithm to achieve the same work bound.

Data Structures and Algorithms,Distributed, Parallel, and Cluster Computing
Speed-Up in Dynamic-Programming

FF YAO

DOI: https://doi.org/10.1137/0603055

1982-01-01

SIAM Journal on Algebraic and Discrete Methods

Abstract:Dynamic programming is a general problem-solving method that has been used widely in many disciplines, including computer science. In this paper we present some recent results in the design of efficient dynamic programming algorithms. These results illustrate two approaches for achieving efficiency: the first by developing general techniques that are applicable to a broad class of problems, and the second by inventing clever algorithms that take advantage of individual situations.
Efficient Dynamic Programming Using Quadrangle Inequalities

F. Frances Yao

DOI: https://doi.org/10.1145/800141.804691

1980-01-01

Abstract:ABSTRACTDynamic programming is one of several widely used problem-solving techniques in computer science and operation research. In applying this technique, one always seeks to find speed-up by taking advantage of special properties of the problem at hand. However, in the current state of art, ad hoc approaches for speeding up seem to be characteristic; few general criteria are known. In this paper we give a quadrangle inequality condition for rendering speed-up. This condition is easily checked, and can be applied to several apparently different problems. For example, it follows immediately from our general condition that the construction of optimal binary search trees may be speeded up from O(n3) steps to O(n2), a result that was first obtained by Knuth using a different and rather complicated argument.
Optimizing Operations Management and Business Analytics Strategies under Uncertainty: Dynamic Programming

Ran Yi

DOI: https://doi.org/10.54254/2754-1169/49/20230507

2023-12-01

Abstract:Dynamic programming is a method used in mathematics, management science, computer science, economics, and bioinformatics to solve complex problems by decomposing the original problem into relatively simple sub-problems. Dynamic programming is often applicable to problems with overlapping subproblems and optimal substructure properties. This paper employs a comprehensive research approach of literature review, as well as empirical analysis and case studies to investigate the topic and demonstrate the practicality and effectiveness of dynamic programming in solving complex decision-making problems. However, the curse of dimensionality poses challenges when dealing with high-dimensional decision spaces, requiring approximate dynamic programming methods, including reinforcement learning algorithms. Therefore, when dealing with more complex dynamic programming problems, it is also essential to use program construction tools such as Python to help design a program that can optimize the problem. Therefore, in the learning of dynamic programming, in addition to the correct understanding of basic concepts and methods, specific problems must be analyzed and dealt with in detail, models should be built with rich imagination, and solutions should be solved with creative skills.
The Principle of Optimality in Dynamic Programming: A Pedagogical Note

Bar Light

2024-08-13

Abstract:The principle of optimality is a fundamental aspect of dynamic programming, which states that the optimal solution to a dynamic optimization problem can be found by combining the optimal solutions to its sub-problems. While this principle is generally applicable, it is often only taught for problems with finite or countable state spaces in order to sidestep measure-theoretic complexities. Therefore, it cannot be applied to classic models such as inventory management and dynamic pricing models that have continuous state spaces, and students may not be aware of the possible challenges involved in studying dynamic programming models with general state spaces. To address this, we provide conditions and a self-contained simple proof that establish when the principle of optimality for discounted dynamic programming is valid. These conditions shed light on the difficulties that may arise in the general state space case. We provide examples from the literature that include the relatively involved case of universally measurable dynamic programming and the simple case of finite dynamic programming where our main result can be applied to show that the principle of optimality holds.

Optimization and Control,Probability
Gradient-Bounded Dynamic Programming with Submodular and Concave Extensible Value Functions

Denis Lebedev,Paul Goulart,Kostas Margellos

DOI: https://doi.org/10.48550/arXiv.2005.11213

2020-05-22

Abstract:We consider dynamic programming problems with finite, discrete-time horizons and prohibitively high-dimensional, discrete state-spaces for direct computation of the value function from the Bellman equation. For the case that the value function of the dynamic program is concave extensible and submodular in its state-space, we present a new algorithm that computes deterministic upper and stochastic lower bounds of the value function similar to dual dynamic programming. We then show that the proposed algorithm terminates after a finite number of iterations. Finally, we demonstrate the efficacy of our approach on a high-dimensional numerical example from delivery slot pricing in attended home delivery.

Optimization and Control
Optimal Control of Hybrid Systems Based on Dynamic Programming

ZS Yin,P Li

DOI: https://doi.org/10.1109/acc.2001.945828

2002-01-01

Abstract:The optimization of hybrid systems is studied in the paper, dynamic programming based on reachable net is proposed. This method can get the optimum and reduce the computing effectively.
Theory and applications of generalized dynamic programming: An overview

R.L. Carraway,T.L. Morin

DOI: https://doi.org/10.1016/0898-1221(88)90188-5

1988-01-01

Abstract:Extensions of dynamic programming (DP) into generalized preference structures, such as exist in multicriteria optimization, have invariably assumed sufficient structure to ensure the validity of Bellman's “principle of optimality” or the presence of monotonicity, hence ensuring the validity of the functional equations of DP. Often, however, the only reasonable formulation of problem (in terms of the size of the resulting state space) is one for which neither monotonicity nor the (strong) principle of optimality is satisfied. Generalized DP, based on a weaker principle of optimality, is a generalization of DP that produces optimal solutions, even in the absence of monotonicity. It is shown both to subsume conventional DP and to provide an alternative means of implementing Mitten's preference order DP. This paper provides an overview of generalized DP and its applications to date, summarizing our computational experiences in the process.

mathematics, applied
A Convex Optimization Approach to Dynamic Programming in Continuous State and Action Spaces

Insoon Yang

DOI: https://doi.org/10.1007/s10957-020-01747-1

2020-09-05

Abstract:In this paper, a convex optimization-based method is proposed for numerically solving dynamic programs in continuous state and action spaces. The key idea is to approximate the output of the Bellman operator at a particular state by the optimal value of a convex program. The approximate Bellman operator has a computational advantage because it involves a convex optimization problem in the case of control-affine systems and convex costs. Using this feature, we propose a simple dynamic programming algorithm to evaluate the approximate value function at pre-specified grid points by solving convex optimization problems in each iteration. We show that the proposed method approximates the optimal value function with a uniform convergence property in the case of convex optimal value functions. We also propose an interpolation-free design method for a control policy, of which performance converges uniformly to the optimum as the grid resolution becomes finer. When a nonlinear control-affine system is considered, the convex optimization approach provides an approximate policy with a provable suboptimality bound. For general cases, the proposed convex formulation of dynamic programming operators can be modified as a nonconvex bi-level program, in which the inner problem is a linear program, without losing uniform convergence properties.

Optimization and Control,Systems and Control
Approximate Dynamic Programming with Feasibility Guarantees

Alexander Engelmann,Maisa B. Bandeira,Timm Faulwasser

2024-01-29

Abstract:Safe and economic operation of networked systems is often challenging. Optimization-based schemes are frequently considered, since they achieve near-optimality while ensuring safety via the explicit consideration of constraints. In applications, these schemes, however, often require solving large-scale optimization problems. Iterative techniques from distributed optimization are frequently proposed for complexity reduction. Yet, they achieve feasibility only asymptotically, which induces a substantial computational burden. This work presents an approximate dynamic programming scheme, which is guaranteed to deliver a feasible solution in "one shot", i.e., in one backward-forward iteration over all subproblems provided they are coupled by a tree structure. Our proposed scheme generalizes methods from seemingly disconnected domains such as power systems and optimal control. We demonstrate its efficacy for problems with nonconvex constraints via numerical examples from both domains.

Optimization and Control,Systems and Control
Online Abstract Dynamic Programming with Contractive Models

Xiuxian Li,Lihua Xie

DOI: https://doi.org/10.48550/arXiv.2107.01383

2021-07-03

Abstract:This paper addresses the abstract dynamic programming (DP) in the online scenario, where the abstract DP mapping is time-varying, instead of static. In this case, optimal costs and policies at different time instants are not the same in general, and the problem amounts to tracking time-varying optimal costs and policies, which is of interest to many practical problems. It is thus necessary to analyze the performance of classical value iteration (VI) and policy iteration (PI) algorithms in the online case. In doing so, this paper develops and provides the theoretical analysis for several online algorithms, including approximate online VI, online PI, approximate online PI, online optimistic PI, approximate online optimistic PI, and asynchronous online PI and VI algorithms. It is proved that the tracking error bounds for all algorithms critically depend upon the largest difference between any two consecutive abstract mappings. Meanwhile, examples are presented to illustrate the theoretical results.

Optimization and Control
Automating Thinning Theorem: Synthesizing Efficient Dynamic Programming Algorithms

Ruyi Ji,Tianran Zhu,Yingfei Xiong,Zhenjiang Hu

2023-07-21

Abstract:Dynamic programming is an important optimization technique, but designing efficient dynamic programming algorithms can be difficult for even professional programmers. Thinning, a technique developed for systematically deriving efficient dynamic programming algorithms, has received much attention in studies because of its effectiveness for a large class of problems. Despite the success of thinning in theory, its practical usage is still limited because (1) applying thinning requires mathematical and algorithmic background, and (2) applying thinning solely may not be enough to generate algorithms as efficient as proposed by human experts. In this paper, we propose two approaches, MetHyl and MetHyl+, to resolve both problems. First, MetHyl automates the application of thinning via program synthesis and thus eliminates the burden to the user for applying thinning. Second, MetHyl+ integrates three rules into MetHyl that optimize three important factors on the time complexity of dynamic programming algorithms that are ignored by thinning, and thus make it able to automatically generate expert-level dynamic programming algorithms on many tasks. We evaluate our approaches on 37 tasks related to 16 optimization problems collected from Introduction to Algorithm, a popular textbook for algorithm courses. The results show that MetHyl+ achieves exponential speed-ups on 97.3% of tasks with an average time cost of less than one minute. Moreover, MetHyl+ generates algorithms that are as efficient as the reference programs provided by human experts on 70.3% of tasks.

Programming Languages
Improved Dynamic Programming Methods for Optimal Control of Lumped-Parameter Stochastic Systems

C. Russell Philbrick,Peter K. Kitanidis

DOI: https://doi.org/10.1287/opre.49.3.398.11219

IF: 2.7

2001-06-01

Operations Research

Abstract:New dynamic programming methods are developed to solve stochastic control problems with a larger number of state variables than previously possible. These methods apply accurate interpolation to numerical approximation of continuous cost-to-go functions, greatly reducing the number of discrete states that must be evaluated. By efficiently incorporating information on first and second derivatives, the approximation reduces computational effort by several orders of magnitude over traditional methods. Consequently, it is practical to apply dynamic programming to complex stochastic problems with a larger number of state variables than traditionally possible. Results are presented for hypothetical reservoir control problems with up to seven state variables and two random inputs.

management,operations research & management science
Exact Dynamic Programming for Positive Systems with Linear Optimal Cost

Yuchao Li,Anders Rantzer

2023-09-26

Abstract:Recent work [Ran22] formulated a class of optimal control problems involving positive linear systems, linear stage costs, and elementwise constraints on control. It was shown that the problem admits linear optimal cost and the associated Bellman's equation can be characterized by a finite-dimensional nonlinear equation, which is solved by linear programming. In this work, we report exact dynamic programming (DP) theories for the same class of problems. Moreover, we extend the results to a related class of problems where the norms of control are bounded while the optimal costs remain linear. In both cases, we provide conditions under which the solutions are unique, investigate properties of the optimal policies, study the convergence of value iteration, policy iteration, and optimistic policy iteration applied to such problems, and analyze the boundedness of the solution to the associated optimization programs. Apart from a form of the Frobenius-Perron theorem, the majority of our results are built upon generic DP theory applicable to problems involving nonnegative stage costs.

Optimization and Control,Systems and Control
A Course in Dynamic Optimization

Bar Light

2024-10-10

Abstract:These lecture notes are derived from a graduate-level course in dynamic optimization, offering an introduction to techniques and models extensively used in management science, economics, operations research, engineering, and computer science. The course emphasizes the theoretical underpinnings of discrete-time dynamic programming models and advanced algorithmic strategies for solving these models. Unlike typical treatments, it provides a proof for the principle of optimality for upper semi-continuous dynamic programming, a middle ground between the simpler countable state space case \cite{bertsekas2012dynamic}, and the involved universally measurable case \cite{bertsekas1996stochastic}. This approach is sufficiently rigorous to include important examples such as dynamic pricing, consumption-savings, and inventory management models. The course also delves into the properties of value and policy functions, leveraging classical results \cite{topkis1998supermodularity} and recent developments. Additionally, it offers an introduction to reinforcement learning, including a formal proof of the convergence of Q-learning algorithms. Furthermore, the notes delve into policy gradient methods for the average reward case, presenting a convergence result for the tabular case in this context. This result is simple and similar to the discounted case but appears to be new.

Optimization and Control,Theoretical Economics,Systems and Control
Dynamic Programming on Nominal Graphs

Nicklas Hoch,Ugo Montanari,Matteo Sammartino

DOI: https://doi.org/10.48550/arXiv.1504.02613

2015-04-10

Logic in Computer Science

Abstract:Many optimization problems can be naturally represented as (hyper) graphs, where vertices correspond to variables and edges to tasks, whose cost depends on the values of the adjacent variables. Capitalizing on the structure of the graph, suitable dynamic programming strategies can select certain orders of evaluation of the variables which guarantee to reach both an optimal solution and a minimal size of the tables computed in the optimization process. In this paper we introduce a simple algebraic specification with parallel composition and restriction whose terms up to structural axioms are the graphs mentioned above. In addition, free (unrestricted) vertices are labelled with variables, and the specification includes operations of name permutation with finite support. We show a correspondence between the well-known tree decompositions of graphs and our terms. If an axiom of scope extension is dropped, several (hierarchical) terms actually correspond to the same graph. A suitable graphical structure can be found, corresponding to every hierarchical term. Evaluating such a graphical structure in some target algebra yields a dynamic programming strategy. If the target algebra satisfies the scope extension axiom, then the result does not depend on the particular structure, but only on the original graph. We apply our approach to the parking optimization problem developed in the ASCENS e-mobility case study, in collaboration with Volkswagen. Dynamic programming evaluations are particularly interesting for autonomic systems, where actual behavior often consists of propagating local knowledge to obtain global knowledge and getting it back for local decisions.

Dynamic Programming Deconstructed: Transformations of the Bellman Equation and Computational Efficiency

Dynamic Optimization for Chemical Process Based on Improved Iterative Dynamic Programming Algorithm

An Improved Quasi-Sequential Approach To Large-Scale Dynamic Process Optimization

A quasi-sequential approach to large-scale dynamic optimization problems

A Comparison of Arithmetic Operations for Dynamic Process Optimization Approach

Parallel and (Nearly) Work-Efficient Dynamic Programming

Speed-Up in Dynamic-Programming

Efficient Dynamic Programming Using Quadrangle Inequalities

Optimizing Operations Management and Business Analytics Strategies under Uncertainty: Dynamic Programming

The Principle of Optimality in Dynamic Programming: A Pedagogical Note

Gradient-Bounded Dynamic Programming with Submodular and Concave Extensible Value Functions

Optimal Control of Hybrid Systems Based on Dynamic Programming

Theory and applications of generalized dynamic programming: An overview

A Convex Optimization Approach to Dynamic Programming in Continuous State and Action Spaces

Approximate Dynamic Programming with Feasibility Guarantees

Online Abstract Dynamic Programming with Contractive Models

Automating Thinning Theorem: Synthesizing Efficient Dynamic Programming Algorithms

Improved Dynamic Programming Methods for Optimal Control of Lumped-Parameter Stochastic Systems

Exact Dynamic Programming for Positive Systems with Linear Optimal Cost

A Course in Dynamic Optimization

Dynamic Programming on Nominal Graphs