Abstract:Humans adaptively perform actions to achieve their goals. This flexible behaviour requires two core abilities: the ability to anticipate the outcomes of candidate actions and the ability to select and implement actions in a goal-directed manner. The ability to predict outcomes has been extensively researched in reinforcement learning paradigms, but this work has often focused on simple actions that are not embedded in hierarchical and sequential structures that are characteristic of goal-directed human behaviour. On the other hand, the ability to select actions in accordance with high-level task goals, particularly in the presence of alternative responses and salient distractors, has been widely researched in cognitive control paradigms. Cognitive control research, however, has often paid less attention to the role of action outcomes. The present review attempts to bridge these accounts by proposing an outcome-guided mechanism for selection of extended actions. Our proposal builds on constructs from the hierarchical reinforcement learning literature, which emphasises the concept of reaching and evaluating informative states, i.e., states that constitute subgoals in complex actions. We develop an account of the neural mechanisms that allow outcome-guided action selection to be achieved in a network that relies on projections from cortical areas to the basal ganglia and back-projections from the basal ganglia to the cortex. These cortico-basal ganglia-thalamo-cortical 'loops' allow convergence - and thus integration - of information from non-adjacent cortical areas (for example between sensory and motor representations). This integration is essential in action sequences, for which achieving an anticipated sensory state signals the successful completion of an action. We further describe how projection pathways within the basal ganglia allow selection between representations, which may pertain to movements, actions, or extended action plans. The model lastly envisages a role for hierarchical projections from the striatum to dopaminergic midbrain areas that enable more rostral frontal areas to bias the selection of inputs from more posterior frontal areas via their respective representations in the basal ganglia.

Adaptive control of human action: the role of outcome representations and reward signals

The role of prediction and outcomes in adaptive cognitive control

Reinforcement learning of adaptive control strategies

Intention Beyond Desire: Commitment in Human Action

Intention Beyond Desire: Humans Spontaneously Commit to Future Actions.

Reward actively engages both implicit and explicit components in dual force field adaptation

Reward Modulates Unconsciously Triggered Adaptive Control Processes

Neurophysiological mechanisms underlying the differential effect of reward prospect on response selection and inhibition

Motivation and Cognitive Control: From Behavior to Neural Mechanism

Reward boosts neural coding of task rules to optimise cognitive flexibility

The Role of the Medial Frontal Cortex in Cognitive Control

Balancing control: a Bayesian interpretation of habitual and goal-directed behavior

Dissociable influences of reward and punishment on adaptive cognitive control

On the Sensitivity of Reward Inference to Misspecified Human Models

The attention habit: how reward learning shapes attentional selection

Behavior Alignment via Reward Function Optimization

From conflict management to reward-based decision making: actors and critics in primate medial frontal cortex

Reward and expectancy effects on neural signals of motor preparation and execution

Computational and neural mechanisms underlying the influence of action affordances on value learning

Neural basis of self-control