Abstract:When a reinforcement learning (RL) method has to decide between several optional policies by solely looking at the received reward, it has to implicitly optimize a Multi-Armed-Bandit (MAB) problem. This arises the question: are current RL algorithms capable of solving MAB problems? We claim that the surprising answer is no. In our experiments we show that in some situations they fail to solve a basic MAB problem, and in many common situations they have a hard time: They suffer from regression in results during training, sensitivity to initialization and high sample complexity. We claim that this stems from variance differences between policies, which causes two problems: The first problem is the "Boring Policy Trap" where each policy have a different implicit exploration depends on its rewards variance, and leaving a boring, or low variance, policy is less likely due to its low implicit exploration. The second problem is the "Manipulative Consultant" problem, where value-estimation functions used in deep RL algorithms such as DQN or deep Actor Critic methods, maximize estimation precision rather than mean rewards, and have a better loss in low-variance policies, which cause the network to converge to a sub-optimal policy. Cognitive experiments on humans showed that noised reward signals may paradoxically improve performance. We explain this using the aforementioned problems, claiming that both humans and algorithms may share similar challenges in decision making. Inspired by this result, we propose the Adaptive Symmetric Reward Noising (ASRN) method, by which we mean equalizing the rewards variance across different policies, thus avoiding the two problems without affecting the environment's mean rewards behavior. We demonstrate that the ASRN scheme can dramatically improve the results.

Quantum Reinforcement Learning for Multi-Armed Bandits

Quantum Reinforcement Learning Method and Application Based on Value Function

Quantum Multi-Armed Bandits and Stochastic Linear Bandits Enjoy Logarithmic Regrets

Quantum Computation For Action Selection Using Reinforcement Learning

Safe reinforcement learning under temporal logic with reward design and quantum action selection

Quantum exploration algorithms for multi-armed bandits

Multi-Armed Bandits and Quantum Channel Oracles

Online Restless Multi-Armed Bandits with Long-Term Fairness Constraints

An Introduction to Quantum Reinforcement Learning (QRL)

Deep Reinforcement Learning for Bandit Arm Localization.

The Bandit Whisperer: Communication Learning for Restless Bandits

Provably Efficient Reinforcement Learning for Adversarial Restless Multi-Armed Bandits with Unknown Transitions and Bandit Feedback

Provably Efficient Exploration in Quantum Reinforcement Learning with Logarithmic Worst-Case Regret

Can Q-learning solve Multi Armed Bantids?

Bandit Algorithm Driven by a Classical Random Walk and a Quantum Walk

A Quantum Reinforcement Learning Method For Repeated Game Theory

A reinforcement learning approach for quantum state engineering

Unentangled quantum reinforcement learning agents in the OpenAI Gym

Combinatorial Multivariant Multi-Armed Bandits with Applications to Episodic Reinforcement Learning and Beyond

Quantum Multi-Agent Reinforcement Learning for Aerial Ad-hoc Networks

Selective Reviews of Bandit Problems in AI via a Statistical View