Abstract:Cooperation as a self-organized collective behavior plays a significant role in the evolution of ecosystems and human society. Reinforcement learning (RL) offers a new perspective, distinct from imitation learning in evolutionary games, for exploring the mechanisms underlying its emergence. However, most existing studies with the public good game (PGG) employ a self-regarding setup or are on pairwise interaction networks. Players in the real world, however, optimize their policies based not only on their histories but also on the histories of their co-players, and the game is played in a group manner. In the work, we investigate the evolution of cooperation in the PGG under the other-regarding reinforcement learning evolutionary game (OR-RLEG) on hypergraph by combining the Q-learning algorithm and evolutionary game framework, where other players' action history is incorporated and the game is played on hypergraphs. Our results show that as the synergy factor increases, the parameter interval is divided into three distinct regions, the absence of cooperation (AC), medium cooperation (MC), and high cooperation (HC), accompanied by two abrupt transitions in the cooperation level near two transition points, respectively. Interestingly, we identify regular and anti-coordinated chessboard structures in the spatial pattern that positively contribute to the first cooperation transition but adversely affect the second. Furthermore, we provide a theoretical treatment for the first transition with an approximated first transition point and reveal that players with a long-sighted perspective and low exploration rate are more likely to reciprocate kindness with each other, thus facilitating the emergence of cooperation. Our findings contribute to understanding the evolution of human cooperation, where other-regarding information and group interactions are commonplace.
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is to explore how Other - Regarding Reinforcement Learning (OR - RL) from other perspectives promotes the evolution of cooperative behaviors on hypergraphs in the Public Goods Game (PGG). Specifically, the researchers want to understand how cooperation emerges and develops when individuals in group interactions consider not only their own historical decisions but also the action histories of their co - participants. In addition, the paper also explores the differences between this new form of cooperation and traditional Self - Regarding Reinforcement Learning (SR - RL) and Evolutionary Game Theory (EG) methods.
### Main Research Questions
1. **Emergence of Cooperation**: The researchers hope to observe the emergence mechanism of cooperative behaviors in the Public Goods Game by introducing Other - Regarding Reinforcement Learning (OR - RL). In particular, they want to know whether this new learning method can more effectively promote cooperation under the hypergraph structure.
2. **Changes in Cooperation Level**: The researchers focus on how the cooperation level ($f_c$) changes with the change of the synergy factor ($\hat{r}$), and identify different cooperation regions (non - cooperation region, medium - cooperation region, high - cooperation region) and the transition points ($\hat{r}^*_1$ and $\hat{r}^*_2$) between these regions.
3. **Spatial Patterns**: The researchers also analyze the spatial patterns at different cooperation levels, especially the influence of the checkerboard structure on the cooperation transition.
### Research Methods
- **Model Construction**: The researchers construct an Other - Regarding Reinforcement Learning Evolutionary Game (OR - RLEG) model and conduct simulations on the von Neumann hypergraph.
- **Q - Learning Algorithm**: Each agent uses the Q - Learning algorithm to select the optimal action according to its state information and historical experience. The state includes not only its own action history but also the action histories of other participants.
- **Simulation Analysis**: Through simulation, the researchers observe the changes in the cooperation level under different synergy factors and analyze the spatial distribution pattern of the cooperation level.
### Key Findings
- **Division of Cooperation Regions**: As the synergy factor $\hat{r}$ increases, the cooperation level $f_c$ can be divided into three regions: non - cooperation region (AC), medium - cooperation region (MC), and high - cooperation region (HC). There are two obvious transition points $\hat{r}^*_1$ and $\hat{r}^*_2$ between these three regions.
- **Checkerboard Structure**: The researchers find that under certain synergy factors, regular and anti - coordinated checkerboard structures will appear, and these structures have a significant impact on the cooperation transition.
- **Theoretical Analysis**: Through theoretical analysis, the researchers determine the first transition point $\hat{r}^*_1$ and reveal that agents with a long - term perspective and a low exploration rate are more inclined to reciprocal cooperation, thus promoting the emergence of cooperation.
### Conclusion
This research provides a new perspective for understanding cooperative behaviors in human society, especially in group interactions where information from other perspectives and group interactions are common. By introducing Other - Regarding Reinforcement Learning and hypergraph structures, the researchers show how cooperative behaviors emerge and develop in complex social networks. These findings have important application values in fields such as resource management, environmental protection, and disease prevention and control.