Abstract:The neural implementation of operant conditioning with few trials is unclear. We propose a Hippocampus-Inspired Cognitive Architecture (HICA) as a neural mechanism for operant conditioning. HICA explains a learning mechanism in which agents can learn a new behavior policy in a few trials, as mammals do in operant conditioning experiments. HICA is composed of two different types of modules. One is a universal learning module type that represents a cortical column in the neocortex gray matter. The working principle is modeled as Modulated Heterarchical Prediction Memory (mHPM). In mHPM, each module learns to predict a succeeding input vector given the sequence of the input vectors from lower layers and the context vectors from higher layers. The prediction is fed into the lower layers as a context signal (top-down feedback signaling), and into the higher layers as an input signal (bottom-up feedforward signaling). Rewards modulate the learning rate in those modules to memorize meaningful sequences effectively. In mHPM, each module updates in a local and distributed way compared to conventional end-to-end learning with backpropagation of the single objective loss. This local structure enables the heterarchical network of modules. The second type is an innate, special-purpose module representing various organs of the brain's subcortical system. Modules modeling organs such as the amygdala, hippocampus, and reward center are pre-programmed to enable instinctive behaviors. The hippocampus plays the role of the simulator. It is an autoregressive prediction model of the top-most level signal with a loop structure of memory, while cortical columns are lower layers that provide detailed information to the simulation. The simulation becomes the basis for learning with few trials and the deliberate planning required for operant conditioning.

A hippocampo-cortical pathway detects changes in the validity of an action as a predictor of reward

Partially dissociable roles of the Orbitofrontal cortex and dorsal Hippocampus in context-dependent (hierarchical) reward predictions and contextual inference in learning

Reward-Spatial View Representations and Learning in the Primate Hippocampus

Predictive Coding of Reward in the Hippocampus

Pathway specific interventions reveal the multiple roles of ventral hippocampus projections in cognitive functions

Ventral tegmental area dopamine neural activity switches simultaneously with rule representations in the prefrontal cortex and hippocampus

Representations of stimulus meaning in the hippocampus

A hippocampus-accumbens code guides goal-directed appetitive behavior

Neural correlates of object identity and reward outcome in the sensory cortical-hippocampal hierarchy: coding of motivational information in perirhinal cortex

Ventral hippocampal cholecystokinin interneurons gate contextual reward memory

Hidden state inference requires abstract contextual representations in ventral hippocampus

Ventrolateral prefrontal cortex in macaques guides decisions in different learning contexts

The Hippocampus and Appetitive Pavlovian Conditioning: Effects of Excitotoxic Hippocampal Lesions on Conditioned Locomotor Activity and Autoshaping

Reward Cues in Space: Commonalities and Differences in Neural Coding by Hippocampal and Ventral Striatal Ensembles

The hippocampal-striatal circuit for goal-directed and habitual choice

Hippocampus-Inspired Cognitive Architecture (HICA) for Operant Conditioning

Hippocampal place cell sequences during a visual discrimination task: recapitulation of paths near the chosen reward site and independence from perirhinal activity

Hippocampal sequences span experience relative to rewards

The hippocampal-striatal axis in learning, prediction and goal-directed behavior.

The Hippocampal–striatal Axis in Learning, Prediction and Goal-Directed Behavior

Dorsal-caudal and ventral hippocampus target different cell populations in the medial frontal cortex in rodents