Sparse Coding by Spiking Neural Networks: Convergence Theory and Computational Results

Ping Tak Peter Tang,Tsung-Han Lin,Mike Davies

DOI: https://doi.org/10.48550/arXiv.1705.05475

2017-05-16

Abstract:In a spiking neural network (SNN), individual neurons operate autonomously and only communicate with other neurons sparingly and asynchronously via spike signals. These characteristics render a massively parallel hardware implementation of SNN a potentially powerful computer, albeit a non von Neumann one. But can one guarantee that a SNN computer solves some important problems reliably? In this paper, we formulate a mathematical model of one SNN that can be configured for a sparse coding problem for feature extraction. With a moderate but well-defined assumption, we prove that the SNN indeed solves sparse coding. To the best of our knowledge, this is the first rigorous result of this kind.

Machine Learning,Neural and Evolutionary Computing,Numerical Analysis,Neurons and Cognition

What problem does this paper attempt to address?

### The Problem Addressed by the Paper

This paper attempts to address how to reliably solve the sparse coding problem using Spiking Neural Networks (SNN). Specifically, the authors propose a mathematical model that describes an SNN configured for the sparse coding problem and prove that under certain assumptions, this SNN can indeed solve the sparse coding problem. This is the first study to provide a rigorous mathematical proof of the convergence of SNNs in sparse coding.

### Background and Motivation

1. **Computational Neuroscience**: Understanding how complex computations emerge from neural networks is a core issue in computational neuroscience. Neuroscientists typically focus on constructing neural network models that resemble actual neural activity and function, while precise mathematical definitions or analyses of these models are relatively secondary.
2. **Computer Science**: Computer scientists usually aim to design new solvers to address specific computational problems. While understanding neural activity can serve as inspiration, the models actually adopted need to be mathematically well-defined and possess provable properties such as stability and convergence.

### Research Objectives

The goal of this paper is to propose two neural network models from a computer science perspective that can be proven to solve the mixed \( \ell_2-\ell_1 \) optimization problem (commonly known as the LASSO problem). LASSO is a commonly used method for sparse coding and has wide applications in machine learning, signal processing, and statistics. The authors provide a framework that rigorously establishes the theoretical foundation for the convergence of firing rates in SNNs to the solution of the LASSO problem.

### Main Contributions

1. **Mathematical Model**: A Spiking LCA (S-LCA) model is proposed, and it is proven that under certain assumptions, it can converge to the solution of the LASSO problem.
2. **Convergence Proof**: Using a generalized version of LaSalle's invariance principle, it is proven that the instantaneous firing rate of S-LCA will converge to the solution of the CLASSO problem under appropriate configurations.
3. **Numerical Implementation**: The numerical implementation of S-LCA is designed, and experiments demonstrate its ability to converge to the CLASSO solution. Experimental results show that even when running on a traditional CPU, S-LCA can converge to the solution with high accuracy in a relatively short time, and its convergence speed even surpasses fast LASSO solvers like FISTA.

### Experimental Validation

1. **Small-Scale Network**: A simple example of a 3-neuron SNN is used to demonstrate how the network quickly converges after a few spike exchanges.
2. **Large-Scale Network**: A large-scale experiment with a 400-neuron SNN is conducted to verify the effectiveness and convergence of S-LCA in an image feature extraction task.
3. **Performance Comparison**: S-LCA is compared with traditional solvers like FISTA, and results show that S-LCA has a significant advantage in early convergence speed, although the final accuracy is slightly reduced due to spike timing errors.

### Conclusion

This paper rigorously proves and experimentally demonstrates the effectiveness of SNNs in solving the sparse coding problem. This research not only provides important support for the theoretical foundation of SNNs but also showcases their potential in practical applications, especially in scenarios requiring efficient communication and parallel computation.