Abstract:Reinforcement learning refers to a group of methods from artificial intelligence where an agent performs learning through trial and error. It differs from supervised learning, since reinforcement learning requires no explicit labels; instead, the agent interacts continuously with its environment. That is, the agent starts in a specific state and then performs an action, based on which it transitions to a new state and, depending on the outcome, receives a reward. Different strategies (e.g. Q-learning) have been proposed to maximize the overall reward, resulting in a so-called policy, which defines the best possible action in each state. Mathematically, this process can be formalized by a Markov decision process and it has been implemented by packages in R; however, there is currently no package available for reinforcement learning. As a remedy, this paper demonstrates how to perform reinforcement learning in R and, for this purpose, introduces the ReinforcementLearning package. The package provides a remarkably flexible framework and is easily applied to a wide range of different problems. We demonstrate its use by drawing upon common examples from the literature (e.g. finding optimal game strategies).

What problem does this paper attempt to address?

The problem this paper attempts to address is the current lack of a dedicated package in R for implementing model-free reinforcement learning. Specifically, while there are already some toolkits in R for solving Markov Decision Processes (MDP), these toolkits are primarily focused on model-based approaches, which require prior knowledge of the environment's state transition probabilities and reward functions. However, for many real-world problems, especially those with unknown or overly complex environment dynamics, model-free reinforcement learning methods are more suitable. Therefore, the paper introduces a new R package—`ReinforcementLearning`, aimed at filling this gap and enabling users to conveniently implement model-free reinforcement learning in R. ### Main Contributions of the Paper: 1. **Introduction of the `ReinforcementLearning` package**: This package allows users to train agents by sampling experiences (including states, actions, and rewards) to learn the optimal policy. 2. **Provision of a flexible framework**: The package can be applied to various problems, supports custom learning parameters, and includes some common performance optimization techniques such as experience replay. 3. **Examples and Applications**: The paper demonstrates how to use the package through some common examples (e.g., finding the optimal strategy in a game). ### Specific Problems Addressed: - **Lack of model-free reinforcement learning tools for R**: Existing R packages mainly focus on model-based approaches and cannot meet the needs of model-free reinforcement learning. - **Improving learning efficiency and stability**: Techniques such as batch reinforcement learning and experience replay improve the efficiency and stability of the learning process. - **Providing an easy-to-use interface**: This allows researchers and practitioners to conveniently implement and apply reinforcement learning algorithms in R. In summary, by introducing the `ReinforcementLearning` package, this paper addresses the lack of model-free reinforcement learning tools in R, providing researchers and practitioners with a powerful tool to more effectively solve various sequential decision-making problems.

Reinforcement Learning in R

Reinforcement Learning in Economics and Finance

Reinforcement Learning: An Introduction

Reinforcement Learning in Clinical Medicine: a Method to Optimize Dynamic Treatment Regime over Time.

Recursive Reinforcement Learning

Reinforcement Learning in Practice: Opportunities and Challenges

Decoding Reinforcement Learning for newcomers

Reinforcement learning for sequential decision making in population research

Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress

Automating Predictive Modeling Process using Reinforcement Learning

A Primer on Deep Reinforcement Learning for Finance

Recent Advances in Reinforcement Learning in Finance

RLIF: Interactive Imitation Learning as Reinforcement Learning

Reinforcement Learning and its Connections with Neuroscience and Psychology

Reinforcement Learning Algorithms: A brief survey

Challenges of real-world reinforcement learning: definitions, benchmarks and analysis

Robot See, Robot Do: Imitation Reward for Noisy Financial Environments

Reinforcement Learning Algorithms: An Overview and Classification

A Survey of Reinforcement Learning Techniques: Strategies, Recent Development, and Future Directions

Using Reinforcement Learning in the Algorithmic Trading Problem

Efficient Online Reinforcement Learning with Offline Data