Abstract:Simulation of large scale biologically plausible spiking neural networks, e.g., Bayesian Confidence Propagation Neural Network (BCPNN), usually requires high-performance supercomputers with dedicated accelerators, such as GPUs, FPGAs, or even Application-Specific Integrated Circuits (ASICs). Almost all of these computers are based on the von Neumann architecture that separates storage and computation. In all these solutions, memory access is the dominant cost even for highly customized computation and memory architecture, such as ASICs. In this paper, we propose an optimization technique that can make the BCPNN simulation memory access friendly by avoiding a dual-access pattern. The BCPNN synaptic traces and weights are organized as matrices accessed both row-wise and column-wise. Accessing data stored in DRAM with a dual-access pattern is extremely expensive. A post-synaptic history buffer and an approximation function thus are introduced to eliminate the troublesome column update. The error analysis combining theoretical analysis and experiments suggests that the probability of introducing intolerable errors by such optimization can be bounded to a very small number, which makes it almost negligible. Derivation and validation of such a bound is the core contribution of this paper. Experiments on a GPU platform shows that compared to the previously reported baseline simulation strategy, the proposed optimization technique reduces the storage requirement by 33%, the global memory access demand by more than 27% and DRAM access rate by more than 5%; the latency of updating synaptic traces decreases by roughly 50%. Compared with the other similar optimization technique reported in the literature, our method clearly shows considerably better results. Although the BCPNN is used as the targeted neural network model, the proposed optimization method can be applied to other artificial neural network models based on a Hebbian learning rule.

Learning Memory Access Patterns

DaDianNao: A Machine-Learning Supercomputer

Highly Efficient Neuromorphic Computing Systems with Emerging Nonvolatile Memories

Emerging Nonvolatile Memories for Machine Learning

Memristors -- from In-memory computing, Deep Learning Acceleration, Spiking Neural Networks, to the Future of Neuromorphic and Bio-inspired Computing

In-memory Computing with Emerging Nonvolatile Memory Devices

Predicting Memory Compiler Performance Outputs using Feed-Forward Neural Networks

Learning-to-learn enables rapid learning with phase-change memory-based in-memory computing

An Efficient Data Prefetch Strategy for Deep Learning Based on Non-volatile Memory

Data and Power Efficient Intelligence with Neuromorphic Learning Machines

Optimizing for In-memory Deep Learning with Emerging Memory Technology

Compute-in-Memory Technologies and Architectures for Deep Learning Workloads

Low-Rank Training of Deep Neural Networks for Emerging Memory Technology

Flash Memory Array for Efficient Implementation of Deep Neural Networks

A Co-design view of Compute in-Memory with Non-Volatile Elements for Neural Networks

Optimizing BCPNN Learning Rule for Memory Access

Computing In-Memory, Revisited

In-memory Implementation of On-chip Trainable and Scalable ANN for AI/ML Applications

Multi-Strided Access Patterns to Boost Hardware Prefetching

Data Cache Prefetching with Perceptron Learning

Emerging Memristor Technology Enabled Next Generation Cortical Processor