Abstract:A normative approach called Similarity Matching was recently introduced for deriving and understanding the algorithmic basis of neural computation focused on unsupervised problems. It involves deriving algorithms from computational objectives and evaluating their compatibility with anatomical and physiological observations. In particular, it introduces neural architectures by considering dual alternatives instead of primal formulations of popular models such as PCA. However, its connection to the Representer theorem remains unexplored. In this work, we propose to use teachings from this approach to explore supervised learning algorithms and clarify the notion of Hebbian learning. We examine regularized supervised learning and elucidate the emergence of neural architecture and additive versus multiplicative update rules. In this work, we focus not on developing new algorithms but on showing that the Representer theorem offers the perfect lens to study biologically plausible learning algorithms. We argue that many past and current advancements in the field rely on some form of dual formulation to introduce biological plausibility. In short, as long as a dual formulation exists, it is possible to derive biologically plausible algorithms. Our work sheds light on the pivotal role of the Representer theorem in advancing our comprehension of neural computation.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to explore the biological plausibility in supervised learning algorithms, especially by connecting the Representer theorem and Hebbian learning. Specifically, the author proposes using the duality principle to understand the basis of neural computation, especially in the context of supervised learning, how to derive biologically - plausible update rules and neural architectures from the primal and dual forms of regularized models. ### Main Research Questions 1. **Supervised Learning Algorithms with Biological Plausibility**: - The paper explores how to use the duality principle and the Representer theorem to derive biologically - plausible supervised learning algorithms. This includes explaining how to introduce biological plausibility through the dual form of regularized models in supervised learning. 2. **The Relationship between Hebbian Learning and the Representer Theorem**: - The author analyzes in detail the relationship between Hebbian learning rules and the Representer theorem, showing how to understand the learning mechanisms in neural networks through these theoretical frameworks. Specifically, the paper explores how to derive neural architectures and update rules from Hebbian learning rules in the dual space. 3. **The Role of the Duality Principle in Neural Computation**: - The paper emphasizes the importance of the duality principle in understanding neural computation, especially in connecting the neural dynamics and update rules of different learning algorithms (such as ridge regression, support vector machines, logistic regression, etc.). Through the duality principle, a deeper understanding of the biological plausibility of these algorithms can be achieved. ### Research Methods - **Duality Principle and Representer Theorem**: - The author uses the duality principle and the Representer theorem to derive the dual forms of supervised learning algorithms, thereby revealing the biological plausibility of these algorithms. Specifically, through dual transformation, the supervised learning problem can be transformed into an optimization problem, where the dual variables have clear neural - dynamics significance. - **Neural Dynamics and Update Rules**: - The paper discusses in detail the neural dynamics and update rules of different supervised learning algorithms (such as ridge regression, support vector machines, logistic regression, etc.). Through these analyses, the author shows how to derive biologically - plausible update rules, such as additive update rules and multiplicative update rules, from the dual forms. ### Conclusions - **Framework of Biological Plausibility**: - The paper proposes a framework based on the duality principle and the Representer theorem for understanding and designing biologically - plausible supervised learning algorithms. This framework not only helps in understanding existing neural - computation models but also can guide future research in developing learning algorithms that are more in line with the characteristics of biological systems. - **Practical Applications**: - Practical applications of this framework include developing local and biologically - plausible alternative algorithms, such as the neuromorphic implementation of back - propagation (BP). In this way, the efficiency and energy utilization of the algorithm can be improved, making it more suitable for online - learning and continuous - learning scenarios. In summary, this paper provides a new perspective for understanding the biological plausibility of supervised learning algorithms by connecting the Representer theorem and Hebbian learning rules, and also provides an important theoretical basis for future neural - computation research.

Duality Principle and Biologically Plausible Learning: Connecting the Representer Theorem and Hebbian Learning

Normative framework for deriving neural networks with multi-compartmental neurons and non-Hebbian plasticity

Two Tales of Single-Phase Contrastive Hebbian Learning

Towards Biologically Plausible Deep Learning

Unsupervised representation learning with Hebbian synaptic and structural plasticity in brain-like feedforward neural networks

Dual Propagation: Accelerating Contrastive Hebbian Learning with Dyadic Neurons

Coupled Hebbian learning and evolutionary dynamics in a formal model for structural synaptic plasticity

A Study of Biologically Plausible Neural Network: The Role and Interactions of Brain-Inspired Mechanisms in Continual Learning

Learning cortical hierarchies with temporal Hebbian updates

Towards Biologically Plausible Computing: A Comprehensive Comparison

Representation Learning in a Decomposed Encoder Design for Bio-inspired Hebbian Learning

Towards New Generation, Biologically Plausible Deep Neural Network Learning

Learning Sparse, Distributed Representations using the Hebbian Principle

Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry

Codimensional Matrix Pairing Perspective of BYY Harmony Learning: Hierarchy of Bilinear Systems, Joint Decomposition of Data-Covariance, and Applications of Network Biology

Learning in the Machine: the Symmetries of the Deep Learning Channel

Duality of Bures and Shape Distances with Implications for Comparing Neural Representations

A structural representing and learning model based on biological neural mechanism

Neocortical plasticity: an unsupervised cake but no free lunch

Unifying sparse coding, predictive coding, and divisive normalization