Abstract:Recently, brain-inspired computing models have shown great potential to outperform today's deep learning solutions in terms of robustness and energy efficiency. Particularly, Spiking Neural Networks (SNNs) and HyperDimensional Computing (HDC) have shown promising results in enabling efficient and robust cognitive learning. Despite the success, these two brain-inspired models have different strengths. While SNN mimics the physical properties of the human brain, HDC models the brain on a more abstract and functional level. Their design philosophies demonstrate complementary patterns that motivate their combination. With the help of the classical psychological model on memory, we propose SpikeHD, the first framework that fundamentally combines Spiking neural network and hyperdimensional computing. SpikeHD generates a scalable and strong cognitive learning system that better mimics brain functionality. SpikeHD exploits spiking neural networks to extract low-level features by preserving the spatial and temporal correlation of raw event-based spike data. Then, it utilizes HDC to operate over SNN output by mapping the signal into high-dimensional space, learning the abstract information, and classifying the data. Our extensive evaluation on a set of benchmark classification problems shows that SpikeHD provides the following benefit compared to SNN architecture: (1) significantly enhance learning capability by exploiting two-stage information processing, (2) enables substantial robustness to noise and failure, and (3) reduces the network size and required parameters to learn complex information.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the deficiencies in energy efficiency and robustness of existing deep - learning solutions, especially the application limitations on Internet of Things (IoT) devices. Specifically, existing deep neural networks (DNNs), due to their high computational complexity and memory requirements, are difficult to be widely used in many practical embedded applications, especially when device resources and power budgets are limited. Moreover, although bio - inspired learning methods such as spiking neural networks (SNNs) have some improvements in energy efficiency, they still have deficiencies in robustness and cognitive support, especially when dealing with complex cognitive tasks.
To overcome these challenges, the paper proposes the SpikeHD framework, which is a hybrid solution combining spiking neural networks (SNNs) and high - dimensional computing (HDC). SpikeHD aims to improve the learning ability in the following ways:
1. **Enhanced learning ability**: Significantly improve the learning ability through two - stage information processing (low - level feature extraction and high - level abstract information learning).
2. **Improved robustness**: Have stronger robustness against noise and faults.
3. **Reduced network size and parameters**: By combining the advantages of SNN and HDC, reduce the required learning parameters, so as to learn complex information more efficiently.
The specific implementation of SpikeHD includes the following steps:
- **SNN training**: First, use SNN to extract low - level spatio - temporal information, update the weights according to the gradient descent rule, and learn to extract information from noisy brain - like data.
- **HDC training**: After the SNN training is close to convergence, connect the output of SNN to the HDC module to conduct higher - level abstract information learning. The HDC encoder maps the output of SNN to a high - dimensional space and adopts a single - pass learning mode.
- **SNN and HDC co - training**: To further optimize the model, a new co - training method is proposed, which enables SNN to be trained based on the predictions of the HDC model, ensuring that the output of the SNN layer is more suitable for HDC learning.
- **Online learning**: For embedded devices with limited resources, SpikeHD supports online learning, that is, updating the model in real - time in the data stream without the need to store a large amount of training data.
Through these innovations, SpikeHD provides a more efficient, more robust brain - like computing framework that is more suitable for embedded devices.