Abstract:Deep reinforcement learning has proven to be effective in various video games, such as Atari games, StarCraft II, Google research football (GRF), and Dota II. We participated in the 2022 IEEE Conference on Games Football AI Competition and ranked in the top eight. Despite recent efforts, building agents for GRF still suffers from multi-agent coordination, sparse rewards, and stochastic environments. To address these issues and achieve good outcomes in the competition, we devised a reinforcement learning algorithm that uses deep reinforcement learning from demonstrations and policy distillation. In this study, we innovatively propose a two-stage algorithm named mimic-to-counteract reinforcement learning (MCRL) based on the historical game logs of opponents, we encountered during the warm-up session and formulated partner agents function similarly to human sparring partners, whereby they simulate opponents with diverse styles of play, enabling primary players to practice against a range of policies, they may encounter in real competitions. Additionally, we trained numerous mentor agents capable of restraining the sparring partners. We distilled their policies and amalgamated them to train a potent primary agent. Empirical results show that the proposed MCRL algorithm can efficiently search for valuable strategies with stable updates and balance the relationship between policy iteration and policy style deviation. Also, the primary agent can learn diverse but coordinated counteracting strategies and ranks in the top eight in the competition.

From mimic to counteract: a two-stage reinforcement learning algorithm for Google research football

S2rl

An Empirical Study on Google Research Football Multi-agent Scenarios

Boosting Studies of Multi-Agent Reinforcement Learning on Google Research Football Environment: the Past, Present, and Future

TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations

TiKick: Toward Playing Multi-agent Football Full Games from Single-agent Demonstrations

Diversity is Strength: Mastering Football Full Game with Interactive Reinforcement Learning of Multiple AIs

Hierarchical Deep Reinforcement Learning Agent with Counter Self-play on Competitive Games

Google Research Football: A Novel Reinforcement Learning Environment

Neural Auto-Curricula

Cooperative multi-agent game based on reinforcement learning

MARLadona -- Towards Cooperative Team Play Using Multi-Agent Reinforcement Learning

Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning

AlphaStar: an Integrated Application of Reinforcement Learning Algorithms

Neural Auto-Curricula in Two-Player Zero-Sum Games.

ACE: Cooperative Multi-agent Q-learning with Bidirectional Action-Dependency

Opponent Modeling in Deep Reinforcement Learning

A Deep Reinforcement Learning-Based Method Applied for Solving Multi-Agent Defense and Attack Problems.

Embedding Contextual Information through Reward Shaping in Multi-Agent Learning: A Case Study from Google Football

Deep Q-Network for AI Soccer

Deep reinforcement learning algorithm based on multi-agent parallelism and its application in game environment