Abstract:In recent years, deep learning has achieved breakthrough successes in various fields, such as image recognition, natural language processing, playing perfect-information game and so on. Unfortunately, the traditional deep neural networks, referred to as artificial neural networks (ANNs), suffer from high energy consumption. Therefore, spiking neural networks (SNNs) have been proposed, which can dramatically decrease the computational power during training and inference of machine learning models. However, SNNs are inferior to conventional neural networks in terms of performance even for the simplest task, image classification. The reason is that data handled in the real world often have been normalized into the range of [0, 1], which is not appropriate for SNNs processing spikes, sparse and binary signals. To cope with the aforementioned issues, we appeal to combine SNN and ANN, and propose hybrid neural networks (HNNs). An HNN consists of spiking layers (SLs, i.e., the neural layers of an SNN) and artificial layers (ALs, i.e., the neural layers of an ANN). Thanks to the coding methods, such as the existing ones (duplicate coding and Poisson coding) and our proposed differentiable Gaussian coding, we can bridge the two kinds of neural layers by converting the real-valued data into binary spike trains. Furthermore, we introduce two learning methods, which enables us to train SLs and ALs either separately or simultaneously. To demonstrate the effectiveness of the proposed framework, we conducted a series of experiments based on the widely used datasets, MNIST and CIFAR-10. The experimental results show that: (1) The performance of a network can generally be improved by increasing the ratio of ALs, which can surpass that of pure SNNs. (2) Gaussian coding is capable of achieving higher accuracy for complex datasets even when the percentage of ALs is small. The proposed HNNs can bridge the gap between the traits of ANNs and SNNs, which provides insights on well understanding the potential of SNN.

Homogeneous Artificial Neural Network

A Gradient Free Neural Network Framework Based on Universal Approximation Theorem

A Gentle Introduction to Artificial Neural Networks

A Convex Relaxation Approach to Generalization Analysis for Parallel Positively Homogeneous Networks

A unified and constructive framework for the universality of neural networks

SNN Meets ANN: Combining Spiking Neural Network (SNN) and Artiﬁcial Neural Network (ANN) for Image Classiﬁcation

A Neural Network Approach for Homogenization of Multiscale Problems

A Neural Network of Smooth Hinge Functions.

An approximation by neural networkswith a fixed weight

HMN: Generalization of Heterogeneous and Multi-layered Network

Homological Neural Networks: A Sparse Architecture for Multivariate Complexity

Approximating Positive Homogeneous Functions with Scale Invariant Neural Networks

Approximation of Hybrid Systems by Neural Networks

Homogenization of the G-Equation: a metric approach

Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control

Generalized Homogeneity in Systems and Control

The Function Representation of Artificial Neural Network

Subhomogeneous Deep Equilibrium Models

Multivariate Smooth Symmetrized and Perturbed Hyperbolic Tangent Neural Network Approximation over Infinite Domains

Generalized hetero-associative neural networks

A Framework for the General Design and Computation of Hybrid Neural Networks