Abstract:This work compares ways of extending Reinforcement Learning algorithms to Partially Observed Markov Decision Processes (POMDPs) with options. One view of options is as temporally extended action, which can be realized as a memory that allows the agent to retain historical information beyond the policy's context window. While option assignment could be handled using heuristics and hand-crafted objectives, learning temporally consistent options and associated sub-policies without explicit supervision is a challenge. Two algorithms, PPOEM and SOAP, are proposed and studied in depth to address this problem. PPOEM applies the forward-backward algorithm (for Hidden Markov Models) to optimize the expected returns for an option-augmented policy. However, this learning approach is unstable during on-policy rollouts. It is also unsuited for learning causal policies without the knowledge of future trajectories, since option assignments are optimized for offline sequences where the entire episode is available. As an alternative approach, SOAP evaluates the policy gradient for an optimal option assignment. It extends the concept of the generalized advantage estimation (GAE) to propagate option advantages through time, which is an analytical equivalent to performing temporal back-propagation of option policy gradients. This option policy is only conditional on the history of the agent, not future actions. Evaluated against competing baselines, SOAP exhibited the most robust performance, correctly discovering options for POMDP corridor environments, as well as on standard benchmarks including Atari and MuJoCo, outperforming PPOEM, as well as LSTM and Option-Critic baselines. The open-sourced code is available at <a class="link-external link-https" href="https://github.com/shuishida/SoapRL" rel="external noopener nofollow">this https URL</a>.

Real Options for Project Schedules (ROPS)

Finite Adaptability in Data-Driven Robust Optimization for Production Scheduling: A Case Study of the Ethylene Plant

Random Sampling Technique For Improving The Net Present Value Of Resource-Con-Strained Projects

A real-world way to manage real options

Application of Real Options in Project Portfolio Selection

Real Options in Operations Research: A Review

SOAP-RL: Sequential Option Advantage Propagation for Reinforcement Learning in POMDP Environments

Ant Colony Algorithm for Scheduling Resource Constrained Projects with Discounted Cash Flows

Robust Optimization-Based Genetic Algorithm for Project Scheduling with Stochastic Activity Durations

Bi-objective Optimization for Resource-constrained Robust Construction Project Scheduling

Real Options Technique as a Tool of Strategic Risk Management

A Real Options Design for Product Outsourcing

Ship Investment Valuation Using Real Option Analysis (ROA)

Real Option Method of Program Sequence Arrangement

Trading in Risk Dimensions (TRD)

Real Option Characters of Petroleum Exploration and Development Projects

Ordinal chemical reaction optimization for stochastic resource constrained project scheduling problem

Resource-Constrained Time-Cost Tradeoff for Repetitive Construction Projects

International Journal of Prognostics and Health Management, ISSN 2153-2648, 2016 008 PHM-Based Wind Turbine Maintenance Optimization Using Real Options

Real Options versus Traditional Methods to assess Renewable Energy Projects

A Real Options Design for Quality Control Charts