Abstract:A default assumption in the design of reinforcement-learning algorithms is that a decision-making agent always explores to learn optimal behavior. In sufficiently complex environments that approach the vastness and scale of the real world, however, attaining optimal performance may in fact be an entirely intractable endeavor and an agent may seldom find itself in a position to complete the requisite exploration for identifying an optimal policy. Recent work has leveraged tools from information theory to design agents that deliberately forgo optimal solutions in favor of sufficiently-satisfying or satisficing solutions, obtained through lossy compression. Notably, such agents may employ fundamentally different exploratory decisions to learn satisficing behaviors more efficiently than optimal ones that are more data intensive. While supported by a rigorous corroborating theory, the underlying algorithm relies on model-based planning, drastically limiting the compatibility of these ideas with function approximation and high-dimensional observations. In this work, we remedy this issue by extending an agent that directly represents uncertainty over the optimal value function allowing it to both bypass the need for model-based planning and to learn satisficing policies. We provide simple yet illustrative experiments that demonstrate how our algorithm enables deep reinforcement-learning agents to achieve satisficing behaviors. In keeping with previous work on this setting for multi-armed bandits, we additionally find that our algorithm is capable of synthesizing optimal behaviors, when feasible, more efficiently than its non-information-theoretic counterpart.

The Optimality of Satisficing Solutions

An enhanced heuristic searching algorithm for complicated constrained optimization problems

A New Heuristic Optimization Algorithm: Harmony Search

Surrogate Search As a Way to Combat Harmful Effects of Ill-behaved Evaluation Functions

Asymptotic Optimality of a Time Optimal Path Parametrization Algorithm

Satisficing as an alternative to optimality and suboptimality in perceptual decision making

Multi-Agent Systems with Quantitative Satisficing Goals

Space Net Optimization

Satisficing Equilibrium

Optimal-transport satisficing with applications to capacitated hub location

Modeling Managerial Search Behavior based on Simon's Concept of Satisficing

Heuristic search for one-to-many shortest path queries

Optimization's Neglected Normative Commitments

Automatic Algorithm Selection for Pseudo-Boolean Optimization with Given Computational Time Limits

No Panacea in Planning: Algorithm Selection for Suboptimal Multi-Agent Path Finding

An Effective Heuristic Algorithm for the Traveling-Salesman Problem

Nature's way of optimizing

Perfectionism Search Algorithm (PSA): An Efficient Meta-Heuristic Optimization Approach

Satisficing Exploration for Deep Reinforcement Learning

Enhanced Multi-Objective A* with Partial Expansion

A Pareto Optimal D* Search Algorithm for Multiobjective Path Planning