Abstract:Cooperation is the foundation of ecosystems and the human society, and the reinforcement learning provides crucial insight into the mechanism for its emergence. However, most previous work has mostly focused on the self-organization at the population level, the fundamental dynamics at the individual level remains unclear. Here, we investigate the evolution of cooperation in a two-agent system, where each agent pursues optimal policies according to the classical Q-learning algorithm in playing the strict prisoner's dilemma. We reveal that a strong memory and long-sighted expectation yield the emergence of Coordinated Optimal Policies (COPs), where both agents act like "Win-Stay, Lose-Shift" (WSLS) to maintain a high level of cooperation. Otherwise, players become tolerant toward their co-player's defection and the cooperation loses stability in the end where the policy "all Defection" (All-D) prevails. This suggests that tolerance could be a good precursor to a crisis in cooperation. Furthermore, our analysis shows that the Coordinated Optimal Modes (COMs) for different COPs gradually lose stability as memory weakens and expectation for the future decreases, where agents fail to predict co-player's action in games and defection dominates. As a result, we give the constraint to expectations of future and memory strength for maintaining cooperation. In contrast to the previous work, the impact of exploration on cooperation is found not be consistent, but depends on composition of COMs. By clarifying these fundamental issues in this two-player system, we hope that our work could be helpful for understanding the emergence and stability of cooperation in more complex scenarios in reality.

Community Enforcement with Endogenous Records

Endogenous monitoring through voluntary reporting in an infinitely repeated prisoner's dilemma game: experimental evidence

Equilibrium Behaviors in Repeated Games

Uniformly strict equilibrium for repeated games with private monitoring and communication

Cooperation Enforcement and Collusion Resistance in Repeated Public Goods Games

An Anti-Folk Theorem for Large Repeated Games with Imperfect Monitoring

Cooperation and self-regulation in a model of agents playing different games

Indirect reciprocity with simple records

Equilibria in Repeated Games under No-Regret with Dynamic Benchmarks

The regulation of good and evi promotes cooperation in public goods game

Privacy-Preserving Public Information for Sequential Games

Consequentialist conditional cooperation in social dilemmas with imperfect information

Emergence of cooperation in two-agent repeated games with reinforcement learning

Trust and Betrayals: Reputational Payoffs and Behaviors without Commitment

Simple Records Support Robust Indirect Reciprocity

Reputation in Repeated Global Games of Regime Change with Exit

Emergence of specialized third-party enforcement

Individual's expulsion to nasty environment promotes cooperation in public goods games

Replicator dynamics of public goods games with global exclusion.

State-clustering method of payoff computation in repeated multiplayer games