Abstract:Human culture relies on innovation: our ability to continuously explore how existing elements can be combined to create new ones. Innovation is not solitary, it relies on collective search and accumulation. Reinforcement learning (RL) approaches commonly assume that fully-connected groups are best suited for innovation. However, human laboratory and field studies have shown that hierarchical innovation is more robustly achieved by dynamic social network structures. In dynamic settings, humans oscillate between innovating individually or in small clusters, and then sharing outcomes with others. To our knowledge, the role of social network structure on innovation has not been systematically studied in RL. Here, we use a multi-level problem setting (WordCraft), with three different innovation tasks to test the hypothesis that the social network structure affects the performance of distributed RL algorithms. We systematically design networks of DQNs sharing experiences from their replay buffers in varying structures (fully-connected, small world, dynamic, ring) and introduce a set of behavioral and mnemonic metrics that extend the classical reward-focused evaluation framework of RL. Comparing the level of innovation achieved by different social network structures across different tasks shows that, first, consistent with human findings, experience sharing within a dynamic structure achieves the highest level of innovation in tasks with a deceptive nature and large search spaces. Second, experience sharing is not as helpful when there is a single clear path to innovation. Third, the metrics we propose, can help understand the success of different social network structures on different tasks, with the diversity of experiences on an individual and group level lending crucial insights.

Reward-Sharing Relational Networks in Multi-Agent Reinforcement Learning as a Framework for Emergent Behavior

S2rl

S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

Social Structure Emergence: A Multi-agent Reinforcement Learning Framework for Relationship Building.

Achieving Collective Welfare in Multi-Agent Reinforcement Learning via Suggestion Sharing

Learning to Share in Multi-Agent Reinforcement Learning

Attentive Relational State Representation in Decentralized Multiagent Reinforcement Learning.

Networked Multi-Agent Reinforcement Learning with Emergent Communication

Robust Multi-Agent Reinforcement Learning with Social Empowerment for Coordination and Communication

Leveraging Organizational Hierarchy to Simplify Reward Design in Cooperative Multi-agent Reinforcement Learning

Mediated Multi-Agent Reinforcement Learning

Reliably Re-Acting to Partner's Actions with the Social Intrinsic Motivation of Transfer Empowerment

Intrinsic Reward with Peer Incentives for Cooperative Multi-Agent Reinforcement Learning.

Investigating Relational State Abstraction in Collaborative MARL

Social Network Structure Shapes Innovation: Experience-sharing in RL with SAPIENS

Fully decentralized multi-agent reinforcement learning with networked agents

Emergent Dominance Hierarchies in Reinforcement Learning Agents

Decentralized Multi-Agent Reinforcement Learning with Networked Agents: Recent Advances

Learning to Incentivize Other Learning Agents

Learning Reward Machines in Cooperative Multi-Agent Tasks

SAEIR: Sequentially Accumulated Entropy Intrinsic Reward for Cooperative Multi-Agent Reinforcement Learning with Sparse Reward