Abstract:Contextual bandit learning is increasingly favored in modern large-scale recommendation systems. To better utlize the contextual information and available user or item features, the integration of neural networks have been introduced to enhance contextual bandit learning and has triggered significant interest from both academia and industry. However, a major challenge arises when implementing a disjoint neural contextual bandit solution in large-scale recommendation systems, where each item or user may correspond to a separate bandit arm. The huge number of items to recommend poses a significant hurdle for real world production deployment. This paper focuses on a joint neural contextual bandit solution which serves all recommending items in one single model. The output consists of a predicted reward $\mu$, an uncertainty $\sigma$ and a hyper-parameter $\alpha$ which balances exploitation and exploration, e.g., $\mu + \alpha \sigma$. The tuning of the parameter $\alpha$ is typically heuristic and complex in practice due to its stochastic nature. To address this challenge, we provide both theoretical analysis and experimental findings regarding the uncertainty $\sigma$ of the joint neural contextual bandit model. Our analysis reveals that $\alpha$ demonstrates an approximate square root relationship with the size of the last hidden layer $F$ and inverse square root relationship with the amount of training data $N$, i.e., $\sigma \propto \sqrt{\frac{F}{N}}$. The experiments, conducted with real industrial data, align with the theoretical analysis, help understanding model behaviors and assist the hyper-parameter tuning during both offline training and online deployment.

Neural Contextual Combinatorial Bandit under Non-stationary Environment

Learning Contextual Bandits in a Non-stationary Environment

Neural Exploitation and Exploration of Contextual Bandits

Deep Contextual Multi-armed Bandits

An Efficient Algorithm for Deep Stochastic Contextual Bandits

Batched Nonparametric Contextual Bandits

Contextual Bandits for Unbounded Context Distributions

Multiscale Non-stationary Stochastic Bandits

Contextual Combinatorial Conservative Bandits

Uncertainty of Joint Neural Contextual Bandit

Towards Domain Adaptive Neural Contextual Bandits

ContextWIN: Whittle Index Based Mixture-of-Experts Neural Model For Restless Bandits Via Deep RL

Stochastic Bandits with Context Distributions

Neural Dueling Bandits

Contextual Multi-armed Bandit Algorithm for Semiparametric Reward Model

Conservative Contextual Combinatorial Cascading Bandit

Stochastic Conservative Contextual Linear Bandits

Batched Neural Bandits

Neural Combinatorial Clustered Bandits for Recommendation Systems

Risk-Aware Continuous Control with Neural Contextual Bandits

Combinatorial Multi-Armed Bandit: General Framework and Applications.