Uncovering the Representation of Spiking Neural Networks Trained with Surrogate Gradient

Yuhang Li,Youngeun Kim,Hyoungseob Park,Priyadarshini Panda

2023-04-26

Abstract:Spiking Neural Networks (SNNs) are recognized as the candidate for the next-generation neural networks due to their bio-plausibility and energy efficiency. Recently, researchers have demonstrated that SNNs are able to achieve nearly state-of-the-art performance in image recognition tasks using surrogate gradient training. However, some essential questions exist pertaining to SNNs that are little studied: Do SNNs trained with surrogate gradient learn different representations from traditional Artificial Neural Networks (ANNs)? Does the time dimension in SNNs provide unique representation power? In this paper, we aim to answer these questions by conducting a representation similarity analysis between SNNs and ANNs using Centered Kernel Alignment (CKA). We start by analyzing the spatial dimension of the networks, including both the width and the depth. Furthermore, our analysis of residual connections shows that SNNs learn a periodic pattern, which rectifies the representations in SNNs to be ANN-like. We additionally investigate the effect of the time dimension on SNN representation, finding that deeper layers encourage more dynamics along the time dimension. We also investigate the impact of input data such as event-stream data and adversarial attacks. Our work uncovers a host of new findings of representations in SNNs. We hope this work will inspire future research to fully comprehend the representation power of SNNs. Code is released at <a class="link-external link-https" href="https://github.com/Intelligent-Computing-Lab-Yale/SNNCKA" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Neural and Evolutionary Computing

What problem does this paper attempt to address?

The paper attempts to address the following issues: 1. **Exploring the differences in representation learning between Spiking Neural Networks (SNN) and Artificial Neural Networks (ANN)**: Specifically, the study investigates whether SNNs trained with surrogate gradient methods can learn representations that are different from traditional ANNs. The surrogate gradient method is a technique that allows SNNs to be trained using backpropagation, but this technique makes SNNs mimic the learning style of ANNs to some extent. 2. **The role of the temporal dimension**: The paper also explores whether the temporal dimension in SNNs provides unique representational capabilities. Since SNNs process serialized spike information, the temporal dimension is an important consideration. 3. **The impact of residual connections**: The study examines how residual connections in SNNs affect representation learning and finds that residual connections tend to make the representations more similar to those in ANNs. Through the above research, the authors aim to reveal the representational characteristics of SNNs, providing a theoretical foundation for further designing architectures and optimization methods that are more suitable for SNNs.

Uncovering the Representation of Spiking Neural Networks Trained with Surrogate Gradient

CSNN: an Augmented Spiking Based Framework with Perceptron-Inception

Learnable Surrogate Gradient for Direct Training Spiking Neural Networks

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Spatial-Temporal Search for Spiking Neural Networks

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

Surrogate Gradient Scaling for Directly Training Spiking Neural Networks.

Directly Training Temporal Spiking Neural Network with Sparse Surrogate Gradient

Spiking Neural Networks -- Part I: Detecting Spatial Patterns

Surrogate Module Learning: Reduce the Gradient Error Accumulation in Training Spiking Neural Networks

Beyond Classification: Directly Training Spiking Neural Networks for Semantic Segmentation

Visual Explanations from Spiking Neural Networks using Interspike Intervals

Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting

Efficient Surrogate Gradients for Training Spiking Neural Networks

Spiking Neural Networks -- Part II: Detecting Spatio-Temporal Patterns

Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks

SNN Learning, Benchmarking, Programming, and Executing

Editorial: Spiking Neural Network Learning, Benchmarking, Programming and Executing

Exploring spiking neural networks: a comprehensive analysis of mathematical models and applications

A Parallel Spiking Neural Network Based on Adaptive Lateral Inhibition Mechanism for Objective Recognition

Unveiling the Potential of Spiking Dynamics in Graph Representation Learning through Spatial-Temporal Normalization and Coding Strategies