Abstract:Human culture relies on innovation: our ability to continuously explore how existing elements can be combined to create new ones. Innovation is not solitary, it relies on collective search and accumulation. Reinforcement learning (RL) approaches commonly assume that fully-connected groups are best suited for innovation. However, human laboratory and field studies have shown that hierarchical innovation is more robustly achieved by dynamic social network structures. In dynamic settings, humans oscillate between innovating individually or in small clusters, and then sharing outcomes with others. To our knowledge, the role of social network structure on innovation has not been systematically studied in RL. Here, we use a multi-level problem setting (WordCraft), with three different innovation tasks to test the hypothesis that the social network structure affects the performance of distributed RL algorithms. We systematically design networks of DQNs sharing experiences from their replay buffers in varying structures (fully-connected, small world, dynamic, ring) and introduce a set of behavioral and mnemonic metrics that extend the classical reward-focused evaluation framework of RL. Comparing the level of innovation achieved by different social network structures across different tasks shows that, first, consistent with human findings, experience sharing within a dynamic structure achieves the highest level of innovation in tasks with a deceptive nature and large search spaces. Second, experience sharing is not as helpful when there is a single clear path to innovation. Third, the metrics we propose, can help understand the success of different social network structures on different tasks, with the diversity of experiences on an individual and group level lending crucial insights.

Replay shapes abstract cognitive maps for efficient social navigation

Replay shapes abstract cognitive maps for efficient social navigation

Reinforced communication and social navigation: remember your friends and remember yourself

A specific brain network for a social map in the human brain

Geometry of Spatial Memory Replay

Prioritizing replay when future goals are unknown

Predictable navigation through spontaneous brain states with cognitive-map-like representations

Cognitive maps of social features enable flexible inference in social networks

Human Navigation Network: the Intrinsic Functional Organization and Behavioral Relevance

A model of hippocampal replay driven by experience and environmental structure facilitates spatial learning

Mapping the social landscape: tracking patterns of interpersonal relationships

Robust spatial memory maps encoded in networks with transient connections

Forming cognitive maps for abstract spaces: the roles of the human hippocampus and orbitofrontal cortex

Social Network Structure Shapes Innovation: Experience-sharing in RL with SAPIENS

Map-based Experience Replay: A Memory-Efficient Solution to Catastrophic Forgetting in Reinforcement Learning

Social navigation: distance and grid-like codes support navigation of abstract social space in human brain

Replay-triggered brain-wide activation in humans

Latent learning and the formation of a spatiotemporal cognitive map of a road network

Distinct replay signatures for prospective decision-making and memory preservation

Structure and influence in an interconnected world: neurocomputational mechanism of real-time distributed learning on social networks

A Brain-Inspired Model of Hippocampal Spatial Cognition Based on a Memory-Replay Mechanism