Abstract:Deep learning has been one of the trendiest research topics. However, as data quantities rise exponentially, training large neural networks can become prohibitively expensive with billions of parameters. Fortunately, recent research has discovered that not all of the computations in traditional network training are necessary. By selectively sparsifying the majority of the neurons during training, we can still obtain acceptable accuracy. SLIDE, a C++ OpenMP-based sub-linear deep learning engine, has been developed in this situation. SLIDE uses the algorithm of locality sensitive hashing (LSH) to query neurons with high activation in sub-linear time. It achieves a remarkable speedup in training large fully-connected networks by making use of the network sparsity as well as multi-core parallelism. However, SLIDE is limited to CPUs, ignoring the popular GPU devices with greater parallel potential and computational capability. In this article, we propose G-SLIDE, a GPU-based sub-linear deep learning engine, which combines the benefits of SLIDE’s adaptive sparsification algorithms with GPUs’ high performance. The main challenges in developing G-SLIDE are efficiently using LSH to sparsify networks and training the special sparse neural networks on the GPU. To address these challenges, we propose several novel solutions, such as specific data formats and appropriate workload partitioning for threads to fully utilize the GPU resources. We evaluate G-SLIDE on two extremely sparse datasets with a 2080 Ti GPU, and the results demonstrate that for the time of one training epoch, G-SLIDE can achieve more than 16.4× speedup over SLIDE on a 32-core/64-thread CPU. Furthermore, on the same platform, G-SLIDE can earn an average of 16.2× speedup over TensorFlow-GPU and 30.8× speedup over TensorFlow-CPU.

Performance optimization of sparse deep neural network based on GPU

Deep Neural Network Acceleration with Sparse Prediction Layers

At-Scale Sparse Deep Neural Network Inference With Efficient GPU Implementation

Performance of Training Sparse Deep Neural Networks on GPUs

Joint Sparsity with Mixed Granularity for Efficient GPU Implementation

Fast Sparse Deep Neural Network Inference with Flexible SpMM Optimization Space Exploration

Accelerating DNN Inference with GraphBLAS and the GPU

Balanced Sparsity for Efficient DNN Inference on GPU

SparseNN: A Performance-Efficient Accelerator for Large-Scale Sparse Neural Networks

A Computing Efficient Hardware Architecture for Sparse Deep Neural Network Computing

Shfl-BW: Accelerating Deep Neural Network Inference with Tensor-Core Aware Weight Pruning

Fast Sparse Deep Neural Networks: Theory and Performance Analysis

Performance Modeling and Optimization of Sparse Matrix-Vector Multiplication on NVIDIA CUDA Platform

Escoin: Efficient Sparse Convolutional Neural Network Inference on GPUs

Accelerating Sparse DNN Models Without Hardware-Support Via Tile-Wise Sparsity

G-SLIDE: A GPU-Based Sub-Linear Deep Learning Engine via LSH Sparsification

Optimizing sparse matrix-vector multiplication based on gpu

High Performance Simulation of Spiking Neural Network on GPGPUs

DNN Training Acceleration Via Exploring GPGPU Friendly Sparsity

Accelerating Large Sparse Neural Network Inference using GPU Task Graph Parallelism

Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes