Twin Network Augmentation: A Novel Training Strategy for Improved Spiking Neural Networks and Efficient Weight Quantization

Lucas Deckers,Benjamin Vandersmissen,Ing Jyh Tsang,Werner Van Leekwijck,Steven Latré
2024-09-24
Abstract:The proliferation of Artificial Neural Networks (ANNs) has led to increased energy consumption, raising concerns about their sustainability. Spiking Neural Networks (SNNs), which are inspired by biological neural systems and operate using sparse, event-driven spikes to communicate information between neurons, offer a potential solution due to their lower energy requirements. An alternative technique for reducing a neural network's footprint is quantization, which compresses weight representations to decrease memory usage and energy consumption. In this study, we present Twin Network Augmentation (TNA), a novel training framework aimed at improving the performance of SNNs while also facilitating an enhanced compression through low-precision quantization of weights. TNA involves co-training an SNN with a twin network, optimizing both networks to minimize their cross-entropy losses and the mean squared error between their output logits. We demonstrate that TNA significantly enhances classification performance across various vision datasets and in addition is particularly effective when applied when reducing SNNs to ternary weight precision. Notably, during inference , only the ternary SNN is retained, significantly reducing the network in number of neurons, connectivity and weight size representation. Our results show that TNA outperforms traditional knowledge distillation methods and achieves state-of-the-art performance for the evaluated network architecture on benchmark datasets, including CIFAR-10, CIFAR-100, and CIFAR-10-DVS. This paper underscores the effectiveness of TNA in bridging the performance gap between SNNs and ANNs and suggests further exploration into the application of TNA in different network architectures and datasets.
Neural and Evolutionary Computing
What problem does this paper attempt to address?
The main problems this paper attempts to address are: 1. **Improving the performance of Spiking Neural Networks (SNNs)**: Although SNNs have significant advantages in energy efficiency over traditional Artificial Neural Networks (ANNs), their performance on various benchmark tasks is still inferior to ANNs. The paper proposes a new training method—Twin Network Augmentation (TNA), which aims to enhance the classification performance of SNNs by co-training a twin network with the same structure as the base SNN. 2. **Achieving efficient weight quantization**: The paper also explores how to maintain or even improve the performance of SNNs when compressed to low-precision weights (e.g., ternary weights). Through the TNA method, high classification accuracy can be maintained even after the network is compressed. Specifically, the paper addresses these problems through the following approaches: - **Introducing the TNA method**: During the training phase, a base SNN and a randomly initialized twin SNN are trained simultaneously. The optimization objectives include minimizing the cross-entropy loss of both networks and the mean squared error between their output logits (logit matching loss). This helps to enhance the model's generalization and representation capabilities. - **Application to ternary weight SNNs**: During training, full-precision weights are used initially, and after a certain number of training epochs, the network is compressed to ternary weights and training continues to fine-tune the model. Experimental results show that this method significantly improves classification performance on multiple datasets, particularly on more challenging datasets such as CIFAR-100 and CIFAR-10-DVS. In summary, by proposing the TNA method, the paper not only improves the classification performance of SNNs but also makes significant progress in weight quantization, providing a new solution for the application of low-power neuromorphic hardware.