Abstract:In recent years, deep learning has achieved breakthrough successes in various fields, such as image recognition, natural language processing, playing perfect-information game and so on. Unfortunately, the traditional deep neural networks, referred to as artificial neural networks (ANNs), suffer from high energy consumption. Therefore, spiking neural networks (SNNs) have been proposed, which can dramatically decrease the computational power during training and inference of machine learning models. However, SNNs are inferior to conventional neural networks in terms of performance even for the simplest task, image classification. The reason is that data handled in the real world often have been normalized into the range of [0, 1], which is not appropriate for SNNs processing spikes, sparse and binary signals. To cope with the aforementioned issues, we appeal to combine SNN and ANN, and propose hybrid neural networks (HNNs). An HNN consists of spiking layers (SLs, i.e., the neural layers of an SNN) and artificial layers (ALs, i.e., the neural layers of an ANN). Thanks to the coding methods, such as the existing ones (duplicate coding and Poisson coding) and our proposed differentiable Gaussian coding, we can bridge the two kinds of neural layers by converting the real-valued data into binary spike trains. Furthermore, we introduce two learning methods, which enables us to train SLs and ALs either separately or simultaneously. To demonstrate the effectiveness of the proposed framework, we conducted a series of experiments based on the widely used datasets, MNIST and CIFAR-10. The experimental results show that: (1) The performance of a network can generally be improved by increasing the ratio of ALs, which can surpass that of pure SNNs. (2) Gaussian coding is capable of achieving higher accuracy for complex datasets even when the percentage of ALs is small. The proposed HNNs can bridge the gap between the traits of ANNs and SNNs, which provides insights on well understanding the potential of SNN.

A Mixed-Integer Programming Approach to Training Dense Neural Networks

Strong mixed-integer programming formulations for trained neural networks

Feed-Forward Neural Networks as a Mixed-Integer Program

Class-Aware Pruning for Efficient Neural Networks

Efficient and Robust Mixed-Integer Optimization Methods for Training Binarized Deep Neural Networks

A Mixed Integer Linear Programming Formulation to Artificial Neural Networks

Multi-Objective Linear Ensembles for Robust and Sparse Training of Few-Bit Neural Networks

Deep learning enhanced mixed integer optimization: Learning to reduce model dimensionality

A discrete-time neural network for optimization problems with hybrid constraints.

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Securing Neural Networks with Knapsack Optimization

A Novel Conversion Method for Spiking Neural Network Using Median Quantization

Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorization

A New Likelihood Ratio Method for Training Artificial Neural Networks

Convex Formulations for Training Two-Layer ReLU Neural Networks

Reducing ANN-SNN Conversion Error Through Residual Membrane Potential.

Learning a Large Neighborhood Search Algorithm for Mixed Integer Programs

SNN Meets ANN: Combining Spiking Neural Network (SNN) and Artiﬁcial Neural Network (ANN) for Image Classiﬁcation

Learning to Optimize for Mixed-Integer Non-linear Programming