Abstract:Large scale artificial neural networks (ANNs) have been widely used in data processing applications. The recurrent neural network (RNN) is a special type of neural network equipped with additional recurrent connections. Such a unique architecture enables the recurrent neural network to remember the past processed information and makes it an expressive model for nonlinear sequence processing tasks. However, the large computation complexity makes it difficult to effectively train a recurrent neural network and therefore significantly limits the research on the recurrent neural network in the last 20 years. In recent years, the use of graphics processing units (GPUs) becomes a significant advance to speed up the training process of large scale neural networks by taking advantage of the massive parallelism capabilities of GPUs. In this paper, we propose an efficient GPU implementation of the large scale recurrent neural network and demonstrate the power of scaling up the recurrent neural network with GPUs. We first explore the potential parallelism of the recurrent neural network and propose a fine-grained two-stage pipeline implementation. Experiment results show that the proposed GPU implementation can achieve 2 ~ 11 x speed-up compared with the basic CPU implementation with the Intel Math Kernel Library. We then use the proposed GPU implementation to scale up the recurrent neural network and improve its performance. The experiment results of the Microsoft Research Sentence Completion Challenge demonstrate that the large scale recurrent network without class layer is able to beat the traditional class-based modest-size recurrent network and achieve an accuracy of 47%, the best result achieved by a single recurrent neural network on the same dataset.

Research on Classification of Large-Scale Text on GPU Platform

Large-Scale Graph Label Propagation on GPUs

Research of parallel global sea surface temperature contours extraction algorithm on CUDA platform

Enabling Efficient Random Access to Hierarchically Compressed Text Data on Diverse GPU Platforms

A Parallel Transient Stability Algorithm for Large-Scale Power System Based on GPU Platform

Robust and Parallel Uyghur Text Localization in Complex Background Images

A Parallel Approach for Contour Extraction Based on CUDA Platform

GPU-Accelerated Graph Label Propagation for Real-Time Fraud Detection

A Real-Time Spike Sorting Method Based on the Embedded GPU

Large Scale Recurrent Neural Network on GPU

A Scalable Attention Mechanism Based Neural Network for Text Classification

Large-Scale Training System for 100-Million Classification at Alibaba

CuLDA_CGS: Solving Large-scale LDA Problems on GPUs

Boosting Large-scale Parallel Training Efficiency with C4: A Communication-Driven Approach

Applying GPU and POSIX thread technologies in massive remote sensing image data processing

CuLDA: Solving Large-scale LDA Problems on GPUs

Parallel L-BFGS-B Algorithm on GPU.

Accelerating String Matching Using Multi-Threaded Algorithm on GPU

Efficient Large-Scale Language Model Training on GPU Clusters

Analysis of GPU Parallel Computing based on Matlab

Parallel Inference for Latent Dirichlet Allocation on Graphics Processing Units.