Abstract:This work focuses on the sample collection in reinforcement learning (RL), where the interaction with the environment is typically time-consuming and extravagantly expensive. In order to collect samples in a more valuable way, we propose a confidence-based sampling strategy based on the optimal computing budget allocation algorithm (OCBA), which actively allocates the computing efforts to actions with different predictive uncertainties. We estimate the uncertainty with ensembles and generalize them from tabular representations to function approximations. The OCBA-based sampling strategy could be easily integrated into various off-policy RL algorithms, where we take Q-learning, DQN, and SAC as examples to show the incorporation. Besides, we provide the theoretical analysis towards convergence and evaluate the algorithms experimentally. According to the experiments, the incorporated algorithms obtain remarkable gains compared with modern ensemble-based RL algorithms. Note to Practitioners-Reinforcement learning is a powerful tool for handling sequential decision-making problems, e.g., autonomous driving and robotics control, where the behaviors typically have a long-term effect on future events. However, although RL achieves human-level control in some tasks, it severely suffers from low sample efficiency. Therefore, implementing RL in some practical areas, e.g., healthcare and rescue, is extremely hard due to the requirement of massive samples. This work aims to enhance the exploration of RL by incorporating OCBA, which provides an asymptotically optimal data-collection strategy for simulation-based optimization. Based on ensemble-based uncertainty estimation and OCBA-based action selection, the incorporated RL algorithms show competitive performance on many benchmarks and significantly reduce the sampling efforts during iterations.

A Kernel-Based Sarsa( \lambda ) Algorithm with Clustering-Based Sample Sparsification.

S2rl

An OCBA-Based Method for Efficient Sample Collection in Reinforcement Learning

Safe and Sample-efficient Reinforcement Learning for Clustered Dynamic Environments

SCGA: Controlling Genetic Algorithms with Sarsa(0)

CARE: Confidence-rich Autonomous Robot Exploration Using Bayesian Kernel Inference and Optimization

Semiconductor final test scheduling with Sarsa(λ, k) algorithm.

SALSA-CLRS: A Sparse and Scalable Benchmark for Algorithmic Reasoning

Robust Kernel Sparse Subspace Clustering

Large-Scale Subspace Clustering Based on Purity Kernel Tensor Learning

Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learning

State-Separated SARSA: A Practical Sequential Decision-Making Algorithm with Recovering Rewards

Contrastive Kernel Subspace Clustering.

Spaks: Self-paced multiple kernel subspace clustering with feature smoothing regularization

Robust subspace clustering based on automatic weighted multiple kernel learning

A Sparse Framework for Robust Possibilistic K-Subspace Clustering

MultiK-MHKS: a Novel Multiple Kernel Learning Algorithm.

SCA2: Novel Efficient Swarm Clustering Algorithm

Autonomous reinforcement of behavioral sequences in neural dynamics

Subspace Clustering Via Seeking Neighbors with Minimum Reconstruction Error

SSCNet: learning-based subspace clustering