Abstract:Humans have the ability to craft abstract, temporally extended and hierarchically organized plans. For instance, when considering how to make spaghetti for dinner, we typically concern ourselves with useful "subgoals" in the task, such as cutting onions, boiling pasta, and cooking a sauce, rather than particulars such as how many cuts to make to the onion, or exactly which muscles to contract. A core question is how such decomposition of a more abstract task into logical subtasks happens in the first place. Previous research has shown that humans are sensitive to a form of higher-order statistical learning named "community structure". Community structure is a common feature of abstract tasks characterized by a logical ordering of subtasks. This structure can be captured by a model where humans learn predictions of upcoming events multiple steps into the future, discounting predictions of events further away in time. One such model is the "successor representation", which has been argued to be useful for hierarchical abstraction. As of yet, no study has convincingly shown that this hierarchical abstraction can be put to use for goal-directed behavior. Here, we investigate whether participants utilize learned community structure to craft hierarchically informed action plans for goal-directed behavior. Participants were asked to search for paintings in a virtual museum, where the paintings were grouped together in "wings" representing community structure in the museum. We find that participants' choices accord with the hierarchical structure of the museum and that their response times are best predicted by a successor representation. The degree to which the response times reflect the community structure of the museum correlates with several measures of performance, including the ability to craft temporally abstract action plans. These results suggest that successor representation learning subserves hierarchical abstractions relevant for goal-directed behavior. Humans have the ability to achieve a diverse range of goals in a highly complex world. Classic theories of decision making focus on simple tasks involving single goals. In the current study, we test a recent theoretical proposal that aims to address the flexibility of hufigman decision making. By learning to predict the upcoming events, humans can acquire a 'model' of the world which they can then leverage to plan their behavior. However, given the complexity of the world, planning directly over all possible events can be overwhelming. We show that, by leveraging this predictive model, humans group similar events together into simpler "hierarchical" representations, which makes planning over these hierarchical representations markedly more efficient. Interestingly, humans seem to learn and remember both the complex predictive model and the simplified hierarchical model, using them for distinct purposes.

Hierarchical encoding of behavior: translating perception into action

Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective

HILONet: Hierarchical Imitation Learning from Non-Aligned Observations

Hierarchical control of goal-directed action in the cortical–basal ganglia network

Seeing "what" Through "why": Evidence from Probing the Causal Structure of Hierarchical Motion.

Neural mechanisms for learning hierarchical structures of information

Hierarchical categorization learning is associated with representational changes in the dorsal striatum and posterior frontal and parietal cortex

Hierarchical Reinforcement Learning, Sequential Behavior, and the Dorsal Frontostriatal System

Computational evidence for hierarchically structured reinforcement learning in humans

The successor representation subserves hierarchical abstraction for goal-directed behavior

Hierarchical reinforcement learning and decision making

Hierarchical reinforcement learning with natural language subgoals

Learning, memory and consolidation mechanisms for behavioral control in hierarchically organized cortico‐basal ganglia systems

Contributions of the basal ganglia to action sequence learning and performance

An algorithmic account for how humans efficiently learn, transfer, and compose hierarchically structured decision policies

Temporal hierarchy of observed goal-directed actions

Flexible constraint hierarchy during the visual encoding of tool-object interactions

Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?

Dynamics of Hierarchical Task Representations

Following instructions from working memory: Why does action at encoding and recall help?

Sociality and interaction envelope organize visual action representations