Abstract:A key driver of mammalian intelligence is the ability to represent incoming sensory information across multiple abstraction levels. For example, in the visual ventral stream, incoming signals are first represented as low-level edge filters and then transformed into high-level object representations. Similar hierarchical structures routinely emerge in artificial neural networks (ANNs) trained for object recognition tasks, suggesting that similar structures may underlie biological neural networks. However, the classical ANN training algorithm, backpropagation, is considered biologically implausible, and thus alternative biologically plausible training methods have been developed such as Equilibrium Propagation, Deep Feedback Control, Supervised Predictive Coding, and Dendritic Error Backpropagation. Several of those models propose that local errors are calculated for each neuron by comparing apical and somatic activities. Notwithstanding, from a neuroscience perspective, it is not clear how a neuron could compare compartmental signals. Here, we propose a solution to this problem in that we let the apical feedback signal change the postsynaptic firing rate and combine this with a differential Hebbian update, a rate-based version of classical spiking time-dependent plasticity (STDP). We prove that weight updates of this form minimize two alternative loss functions that we prove to be equivalent to the error-based losses used in machine learning: the inference latency and the amount of top-down feedback necessary. Moreover, we show that the use of differential Hebbian updates works similarly well in other feedback-based deep learning frameworks such as Predictive Coding or Equilibrium Propagation. Finally, our work removes a key requirement of biologically plausible models for deep learning and proposes a learning mechanism that would explain how temporal Hebbian learning rules can implement supervised hierarchical learning.

Reinforcement Control with Hierarchical Backpropagated Adaptive Critics

Abstract Value Iteration for Hierarchical Reinforcement Learning

Hierarchical Deep Reinforcement Learning for Continuous Action Control

RLOC: Neurobiologically Inspired Hierarchical Reinforcement Learning Algorithm for Continuous Control of Nonlinear Dynamical Systems

Learning-Based Neural Dynamic Surface Predictive Control for MMC

A Dual Adaptive Control Theory Inspired by Hebbian Associative Learning

Intemittent Control with Reinforcement Leaning

Online Reinforcement Learning-based Neural Network Controller Design for Affine Nonlinear Discrete-time Systems.

Hierarchical learning control for autonomous robots inspired by central nervous system

Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning

Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Control.

Learning Hierarchical Control Systems for Autonomous Systems with Energy Constraints

Learning Hierarchical Control For Multi-Agent Capacity-Constrained Systems

Trajectory Tracking within a Hierarchical Primitive-Based Learning Approach

Learning cortical hierarchies with temporal Hebbian updates

Hierarchical Meta-learning-based Adaptive Controller

A Digital Receding-Horizon Learning Controller for Nonlinear Continuous-time Systems

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

Adaptive control of a mechatronic system using constrained residual reinforcement learning

A successive approximation method in functional spaces for hierarchical optimal control problems and its application to learning

Online adaptive learning of optimal control solutions using integral reinforcement learning