Abstract:A decision process in which rewards depend on history rather than merely on the current state is called a decision process with non-Markovian rewards (NMRDP). In decision-theoretic planning, where many desirable behaviours are more naturally expressed as properties of execution sequences rather than as properties of states, NMRDPs form a more natural model than the commonly adopted fully Markovian decision process (MDP) model. While the more tractable solution methods developed for MDPs do not directly apply in the presence of non-Markovian rewards, a number of solution methods for NMRDPs have been proposed in the literature. These all exploit a compact specification of the non-Markovian reward function in temporal logic, to automatically translate the NMRDP into an equivalent MDP which is solved using efficient MDP solution methods. This paper presents NMRDPP (Non-Markovian Reward Decision Process Planner), a software platform for the development and experimentation of methods for decision-theoretic planning with non-Markovian rewards. The current version of NMRDPP implements, under a single interface, a family of methods based on existing as well as new approaches which we describe in detail. These include dynamic programming, heuristic search, and structured methods. Using NMRDPP, we compare the methods and identify certain problem features that affect their performance. NMRDPPs treatment of non-Markovian rewards is inspired by the treatment of domain-specific search control knowledge in the TLPlan planner, which it incorporates as a special case. In the First International Probabilistic Planning Competition, NMRDPP was able to compete and perform well in both the domain-independent and hand-coded tracks, using search control knowledge in the latter.

Fast Value Iteration for Goal-Directed Markov Decision Processes

Fast Planning in Stochastic Games

Incremental Value Iteration for Time-Aggregated Markov-Decision Processes

Efficient Strategy Iteration for Mean Payoff in Markov Decision Processes

An Accelerated Fitted Value Iteration Algorithm for MDPs with Finite and Vector-Valued Action Space

Asynchronous value iteration for markov decision processes with continuous state spaces

A Probabilistic Forward Search Value Iteration Algorithm for POMDP

Variational Planning for Graph-based MDPs.

A Probabilistic Greedy Search Value Iteration Algorithm For Pomdp

Value Iteration is Optic Composition

Acting in Delayed Environments with Non-Stationary Markov Policies

Solving Markov Decision Processes with Reachability Characterization from Mean First Passage Times

A Neighborhood-Based Value Iteration Algorithm For Pomdp Problems

Decision-Theoretic Planning with non-Markovian Rewards

Acceleration Operators in the Value Iteration Algorithms for Average Reward Markov Decision Processes

A Probability-Based Value Iteration on Optimal Policy Algorithm for POMDP

Speeding Up Planning in Markov Decision Processes via Automatically Constructed Abstractions

Adaptive Multi-Goal Exploration

Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning

Policy Graph Pruning And Optimization In Monte Carlo Value Iteration For Continuous-State Pomdps

Optimistic Planning by Regularized Dynamic Programming