Abstract:Dynamic programming (DP) is an algorithmic design paradigm for the efficient, exact solution of otherwise intractable, combinatorial problems. However, DP algorithm design is often presented in an ad-hoc manner. It is sometimes difficult to justify algorithm correctness. To address this issue, this paper presents a rigorous algebraic formalism for systematically deriving DP algorithms, based on semiring polymorphism. We start with a specification, construct an algorithm to compute the required solution which is self-evidently correct because it exhaustively generates and evaluates all possible solutions meeting the specification. We then derive, through the use of shortcut fusion, an implementation of this algorithm which is both efficient and correct. We also demonstrate how, with the use of semiring lifting, the specification can be augmented with combinatorial constraints, showing how these constraints can be fused with the algorithm. We furthermore demonstrate how existing DP algorithms for a given combinatorial problem can be abstracted from their original context and re-purposed. This approach can be applied to the full scope of combinatorial problems expressible in terms of semirings. This includes, for example: optimal probability and Viterbi decoding, probabilistic marginalization, logical inference, fuzzy sets, differentiable softmax, relational and provenance queries. The approach, building on ideas from the existing literature on constructive algorithmics, exploits generic properties of polymorphic functions, tupling and formal sums and algebraic simplifications arising from constraint algebras. We demonstrate the effectiveness of this formalism for some example applications arising in signal processing, bioinformatics and reliability engineering. Python software implementing these algorithms can be downloaded from: <a class="link-external link-http" href="http://www.maxlittle.net/software/dppolyalg.zip" rel="external noopener nofollow">this http URL</a>.

Safe Dynamic Programming

Safe Reinforcement Learning for Constrained Markov Decision Processes with Stochastic Stopping Time

Approximate Dynamic Programming with Feasibility Guarantees

Differential Dynamic Programming with Nonlinear Safety Constraints Under System Uncertainties

Unifying Safety Approaches for Stochastic Systems: From Barrier Functions to Uncertain Abstractions via Dynamic Programming

Controlled Markov Processes With Safety State Constraints

Dynamic Programming for Structured Continuous Markov Decision Problems

Bounding Procedures for Stochastic Dynamic Programs with Application to the Perimeter Patrol Problem

Safe Reinforcement Learning in Constrained Markov Decision Processes

Gradient-Bounded Dynamic Programming with Submodular and Concave Extensible Value Functions

Improved Exploration for Safety-Embedded Differential Dynamic Programming Using Tolerant Barrier States

Safe Exploration in Markov Decision Processes with Time-Variant Safety using Spatio-Temporal Gaussian Process

Safety of Dynamical Systems with Multiple Non-Convex Unsafe Sets Using Control Barrier Functions

Dynamic programming by polymorphic semiring algebraic shortcut fusion

Dynamic Maintenance of Monotone Dynamic Programs and Applications

Approximate dynamic programming for continuous state and control problems

Reach-Avoid Analysis for Polynomial Stochastic Differential Equations

Optimizing Operations Management and Business Analytics Strategies under Uncertainty: Dynamic Programming

Online Model-free Safety Verification for Markov Decision Processes Without Safety Violation

Fast, Smooth, and Safe: Implicit Control Barrier Functions through Reach-Avoid Differential Dynamic Programming

Safe Guaranteed Exploration for Non-linear Systems