Abstract:<p>This article aims to provide a synthesis on the question how brain structures cooperate to accomplish hierarchically organized behaviors, characterized by low‐level, habitual routines nested in larger sequences of planned, goal‐directed behavior. The functioning of a connected set of brain structures—prefrontal cortex, hippocampus, striatum, and dopaminergic mesencephalon—is reviewed in relation to two important distinctions: (a) goal‐directed as opposed to habitual behavior and (b) model‐based and model‐free learning. Recent evidence indicates that the orbitomedial prefrontal cortices not only subserve goal‐directed behavior and model‐based learning, but also code the "landscape" (task space) of behaviorally relevant variables. While the hippocampus stands out for its role in coding and memorizing world state representations, it is argued to function in model‐based learning but is not required for coding of action–outcome contingencies, illustrating that goal‐directed behavior is not congruent with model‐based learning. While the dorsolateral and dorsomedial striatum largely conform to the dichotomy between habitual versus goal‐directed behavior, ventral striatal functions go beyond this distinction. Next, we contextualize findings on coding of reward‐prediction errors by ventral tegmental dopamine neurons to suggest a broader role of mesencephalic dopamine cells, viz. in behavioral reactivity and signaling unexpected sensory changes. We hypothesize that goal‐directed behavior is hierarchically organized in interconnected cortico‐basal ganglia loops, where a limbic‐affective prefrontal‐ventral striatal loop controls action selection in a dorsomedial prefrontal–striatal loop, which in turn regulates activity in sensorimotor‐dorsolateral striatal circuits. This structure for behavioral organization requires alignment with mechanisms for memory formation and consolidation. We propose that frontal corticothalamic circuits form a high‐level loop for memory processing that initiates and temporally organizes nested activities in lower‐level loops, including the hippocampus and the ripple‐associated replay it generates. The evidence on hierarchically organized behavior converges with that on consolidation mechanisms in suggesting a frontal‐to‐caudal directionality in processing control.</p>

Working memory and response selection: a computational account of interactions among cortico-basalganglio-thalamic loops

Interactions between frontal cortex and basal ganglia in working memory: A computational model

A robot model of the basal ganglia: Behavior and intrinsic processing

Adaptive coordination of working-memory and reinforcement learning in non-human primates performing a trial-and-error problem solving task

A computational model of inhibitory control in frontal cortex and basal ganglia

Dynamical model of salience gated working memory, action selection and reinforcement based on basal ganglia and dopamine feedback

Attention and Working Memory: a Dynamical Model of Neuronal Activity in the Prefrontal Cortex

Biomimetic race model of the loop between the superior colliculus and the basal ganglia: Subcortical selection of saccade targets

A model for integrating elementary neural functions into delayed-response behavior

Neurodynamics of the Prefrontal Cortex during Conditional Visuomotor Associations

Neuronal Representation of a Working Memory-Based Decision Strategy in the Motor and Prefrontal Cortico-Basal Ganglia Loops

Functioning of Frontostriatal Anatomical "loops" in Mechanisms of Cognitive Control

Learning, memory and consolidation mechanisms for behavioral control in hierarchically organized cortico‐basal ganglia systems

Attention, Short-Term Memory, and Action Selection: A Unifying Theory

Cortico-subcortical dynamics in primate working memory

A ventral pallidal-thalamocortical circuit mediates the cognitive control of instrumental action

How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis

How cortico-basal ganglia-thalamic subnetworks can shift decision policies to maximize reward rate

Using TD learning to simulate working memory performance in a model of the prefrontal cortex and basal ganglia

Mechanisms of Hierarchical Reinforcement Learning in Corticostriatal Circuits 1: Computational Analysis