Abstract:A foundational machine-learning architecture is reinforcement learning, where an outstanding problem is achieving an optimal balance between exploration and exploitation. Specifically, exploration enables the agents to discover optimal policies in unknown domains of the environment for gaining potentially large future rewards, while exploitation relies on the already acquired knowledge to maximize the immediate rewards. We articulate an approach to this problem, treating the dynamical process of reinforcement learning as a Markov decision process that can be modeled as a nondeterministic finite automaton and defining a subset of states in the automaton to represent the preference for exploring unknown domains of the environment. Exploration is prioritized by assigning higher transition probabilities to these states. We derive a mathematical framework to systematically balance exploration and exploitation by formulating it as a mixed integer programming (MIP) problem to optimize the agent's actions and maximize the discovery of novel preferential states. Solving the MIP problem provides a trade-off point between exploiting known states and exploring unexplored regions. We validate the framework computationally with a benchmark system and argue that the articulated automaton is effectively an adaptive network with a time-varying connection matrix, where the states in the automaton are nodes and the transitions among the states represent the edges. The network is adaptive because the transition probabilities evolve over time. The established connection between the adaptive automaton arising from reinforcement learning and the adaptive network opens the door to applying theories of complex dynamical networks to address frontier problems in machine learning and artificial intelligence.

Reward-Based Exploration: Adaptive Control for Deep Reinforcement Learning

Optimal Exploration Algorithm of Multi-Agent Reinforcement Learning Methods (Student Abstract)

A Dynamic Adjusting Reward Function Method for Deep Reinforcement Learning with Adjustable Parameters

MADE: Exploration via Maximizing Deviation from Explored Regions

Exploration by Maximizing Renyi Entropy for Reward-Free RL Framework.

An Effective Maximum Entropy Exploration Approach for Deceptive Game in Reinforcement Learning.

Bounded Exploration with World Model Uncertainty in Soft Actor-Critic Reinforcement Learning Algorithm

Reward Space Noise for Exploration in Deep Reinforcement Learning

Random Latent Exploration for Deep Reinforcement Learning

Dynamic Subgoal-based Exploration via Bayesian Optimization

Leveraging Transition Exploratory Bonus for Efficient Exploration in Hard-Transiting Reinforcement Learning Problems.

Adaptive trajectory-constrained exploration strategy for deep reinforcement learning

BeBold: Exploration Beyond the Boundary of Explored Regions

Deep intrinsically motivated exploration in continuous control

Adaptive network approach to exploration-exploitation trade-off in reinforcement learning

LiFE:Deep Exploration Via Linear-Feature Bonus in Continuous Control

Influence-Based Multi-Agent Exploration

Subspace-Aware Exploration for Sparse-Reward Multi-Agent Tasks.

Reward Uncertainty for Exploration in Preference-based Reinforcement Learning

Careful at Estimation and Bold at Exploration

MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization