Abstract:The public goods game describes a social dilemma in which a large proportion of agents act as conditional cooperators (CC): they only act cooperatively if they see others acting cooperatively because they satisfice with the social norm to be in line with what others are doing instead of optimizing cooperation. CCs are guided by aspiration-based reinforcement learning guided by past experiences of interactions with others and satisficing aspirations. In many real-world settings, reinforcing social norms do not emerge. In this paper, we propose that an optimizing reinforcement agent can facilitate cooperation through nudges, i.e. indirect mechanisms for cooperation to happen. The agent's goal is to motivate CCs into cooperation through its own actions to create social norms that signal that others are cooperating. We introduce a multi-agent reinforcement learning model for public goods games, with 3 CC learning agents using aspirational reinforcement learning and 1 nudging agent using deep reinforcement learning to learn nudges that optimize cooperation. For our nudging agent, we model two distinct reward functions, one maximizing the total game return (sum DRL) and one maximizing the number of cooperative contributions contributions higher than a proportional threshold (prop DRL). Our results show that our aspiration-based RL model for CC agents is consistent with empirically observed CC behavior. Games combining 3 CC RL agents and one nudging RL agent outperform the baseline consisting of 4 CC RL agents only. The sum DRL nudging agent increases the total sum of contributions by 8.22% and the total proportion of cooperative contributions by 12.42%, while the prop nudging DRL increases the total sum of contributions by 8.85% and the total proportion of cooperative contributions by 14.87%. Our findings advance the literature on public goods games and reinforcement learning.

Matching Gains with Pays: Effective and Fair Learning in Multi-Agent Public Goods Dilemmas

Shapley Q-Value: A Local Reward Approach to Solve Global Reward Games

Synergistic effects of adaptive reward and reinforcement learning rules on cooperation

A Collaborative Multiagent Reinforcement Learning Method Based on Policy Gradient Potential

Aligning Individual and Collective Objectives in Multi-Agent Cooperation

Learning Fair Policies in Decentralized Cooperative Multi-Agent Reinforcement Learning

Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning

Towards Global Optimality in Cooperative MARL with Sequential Transformation

Dueling Network Architecture for Multi-Agent Deep Deterministic Policy Gradient

Learning Nudges for Conditional Cooperation: A Multi-Agent Reinforcement Learning Model

Cooperation in Public Goods Games: Leveraging Other-Regarding Reinforcement Learning on Hypergraphs

Toward Finding Strong Pareto Optimal Policies in Multi-Agent Reinforcement Learning

Egoism, utilitarianism and egalitarianism in multi-agent reinforcement learning

Achieving Collective Welfare in Multi-Agent Reinforcement Learning via Suggestion Sharing

Progressive Diversifying Policy for Multi-Agent Reinforcement Learning

Learning Optimal "Pigovian Tax" in Sequential Social Dilemmas

Friend-or-Foe Deep Deterministic Policy Gradient

A Cooperation Graph Approach for Multiagent Sparse Reward Reinforcement Learning

Priority over Quantity: A Self-Incentive Credit Assignment Scheme for Cooperative Multiagent Reinforcement Learning

Towards Global Optimality in Cooperative MARL with the Transformation And Distillation Framework