Abstract:Planning under uncertainty is a central problem in the study of automated sequential decision making, and has been addressed by researchers in many different fields, including AI planning, decision analysis, operations research, control theory and economics. While the assumptions and perspectives adopted in these areas often differ in substantial ways, many planning problems of interest to researchers in these fields can be modeled as Markov decision processes (MDPs) and analyzed using the techniques of decision theory. This paper presents an overview and synthesis of MDP-related methods, showing how they provide a unifying framework for modeling many classes of planning problems studied in AI. It also describes structural properties of MDPs that, when exhibited by particular classes of problems, can be exploited in the construction of optimal or approximately optimal policies or plans. Planning problems commonly possess structure in the reward and value functions used to describe performance criteria, in the functions used to describe state transitions and observations, and in the relationships among features used to describe states, actions, rewards, and observations. Specialized representations, and algorithms employing these representations, can achieve computational leverage by exploiting these various forms of structure. Certain AI techniques -- in particular those based on the use of structured, intensional representations -- can be viewed in this way. This paper surveys several types of representations for both classical and decision-theoretic planning problems, and planning algorithms that exploit these representations in a number of different ways to ease the computational burden of constructing policies or plans. It focuses primarily on abstraction, aggregation and decomposition techniques based on AI-style representations.

Planning without state space explosion: Petri net to Markov decision process

Decision-Theoretic Planning with non-Markovian Rewards

Stochastic Decision Petri Nets

Occasionally Observed Piecewise-deterministic Markov Processes

Speeding Up Planning in Markov Decision Processes via Automatically Constructed Abstractions

Controlled Markov Processes With Safety State Constraints

A method for the online construction of the set of states of a Markov Decision Process using Answer Set Programming

Modeling for robot task planning based on light-weighted Markov decision process

Safe MDP Planning by Learning Temporal Patterns of Undesirable Trajectories and Averting Negative Side Effects

Dynamic Programming for Structured Continuous Markov Decision Problems

Reinforcement Learning of Risk-Constrained Policies in Markov Decision Processes

Monte Carlo Planning for Stochastic Control on Constrained Markov Decision Processes

Compositional Planning for Logically Constrained Multi-Agent Markov Decision Processes

A safe exploration approach to constrained Markov decision processes

Markov Decision Processes under External Temporal Processes

Planning with Partially Observable Markov Decision Processes: Advances in Exact Solution Method

Decision-Theoretic Planning: Structural Assumptions and Computational Leverage

Safe POMDP Online Planning among Dynamic Agents via Adaptive Conformal Prediction

Multi-agent Multi-target Path Planning in Markov Decision Processes

Hybrid Planning for Dynamic Multimodal Stochastic Shortest Paths

Robust Markov Decision Processes: A Place Where AI and Formal Methods Meet