Abstract:Reinforcement learning methods’ main idea is to train an agent to learn the optimal policy to get rewards from the environment through continuous interaction. The reward function is extremely important in the whole process, and its setting implies the researchers’ modeling of realistic task and guidance to the model, if it is not set properly, it may lead to problems such as the model failing to converge and being trapped in the local optimization. The design of reward function rely on the domain knowledge of experts and the experience and practice of researchers. We focus on the reward function setting problem in reinforcement learning, and combines the unique advantages of visual analytics to propose an interactive visual analytics method for reward function setting. On the basis of traditional reinforcement learning algorithms, the method helps researchers to explore model performance and model policy in depth by processing and mining data such as training process and trajectory of interaction between the agent and the environment, and combining visualization and interaction techniques, so as to enhance researchers’ specific understanding of the reinforcement learning environment and the model, and then assist them in the corresponding adjustment of the reward function and the subsequent training. For the theories and methods described, we carry out corresponding practice and exploration in a specific reinforcement learning soccer environment (Google Research Football). Based on this, we contributes a visual analysis system for interactive reinforcement learning, which realizes the visual view of training data and interaction trajectories, provides an interface for reward function modification, and subsequently illustrates the usability and practicability of our system with some examples.

Architecting and Visualizing Deep Reinforcement Learning Models

Learn to Interpret Atari Agents.

State of the Art Control of Atari Games Using Shallow Reinforcement Learning

Playing Atari with Deep Reinforcement Learning

A Survey of Deep Reinforcement Learning in Video Games

Model-Based Reinforcement Learning for Atari

A Deep Reinforcement Learning Agent for Geometry Online Tutoring

Toward a Psychology of Deep Reinforcement Learning Agents Using a Cognitive Architecture

Neural architecture impact on identifying temporally extended Reinforcement Learning tasks

Deep reinforcement learning from human preferences

High Performance Across Two Atari Paddle Games Using the Same Perceptual Control Architecture Without Training

A Human Mixed Strategy Approach to Deep Reinforcement Learning

Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environments

Emergent Solutions to High-Dimensional Multitask Reinforcement Learning

Learning to Play Air Hockey with Model-Based Deep Reinforcement Learning

Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals

Virtual Augmented Reality for Atari Reinforcement Learning

Interactive Visual Analytics for Reward Function Setting of Reinforcement Learning: A Case Study of Soccer Games

Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning

Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replay