Abstract:The stacked autoencoder is a deep learning model that consists of multiple autoencoders. This model has been widely applied in numerous machine learning applications. A significant amount of effort has been made to increase the size of the deep learning model with respect to the size of the training dataset and the parameter of the model to improve performance. However, training a large deep learning model is highly time consuming. Recent studies have applied the CPU cluster with thousands of machines as well as the single GPU or the GPU cluster, to train large scale deep learning models. As a high-performance coprocessor like GPU, the Xeon Phi can be an alternative tool for training large scale deep learning models on a single machine. The Xeon Phi can be recognized as a small cluster which features about 60 cores, and each core supports four hardware threads. Massive parallelism offsets the low computing capacity of every core, but challenges an efficient parallel autoencoders design.In this paper, we analyze the training algorithm of autoencoders based on the matrix operation and point out the thread oversubscription problem, which results in performance degradation. Based on the observation, we propose our map-reduce implementation of autoencoders on the Xeon Phi coprocessor. Our basic idea is to parallelize multiple autoencoder model replicas with bulk synchronous parallel (BSP) communication model where the parameters are updated after the computations of all replicas are completed. Each thread is responsible for one model replica, and all replicas work together on the same mini-batch. This data parallelism method is suitable for training autoencoders on the Xeon Phi, and can extend to asynchronous parallel training method without thread oversubscription. In our experiment the speedup is four times higher than that of sequential implementation. Enlarging the size of the autoencoder model, our method still gets stable speedup.

CHAOS: A Parallelization Scheme for Training Convolutional Neural Networks on Intel Xeon Phi

Parallelizing Convolutional Neural Networks On Intel (R) Many Integrated Core Architecture

Training Large Scale Deep Neural Networks on the Intel Xeon Phi Many-Core Coprocessor

A Map-Reduce Method for Training Autoencoders on Xeon Phi.

A Parallel Non-Local Means Denoising Algorithm Implementation with OpenMP and OpenCL on Intel Xeon Phi Coprocessor

Test-driving Intel Xeon Phi

Parallel Computing In Dnns Using Cpu And Mic

Parallelizing convolutional neural network for the handwriting recognition problems with different architectures

Investigating Large-Scale Feature Matching Using The Intel (R) Xeon Phi (Tm) Coprocessor

Single binding of data and model parallelisms to parallelize convolutional neural networks through multiple machines.

An Early Performance Evaluation Of Opencl On Intel Xeon Phi

Performance Modelling of Deep Learning on Intel Many Integrated Core Architectures

Weak Execution Ordering - Exploiting Iterative Methods on Many-Core GPUs.

Utilizing Multiple Xeon Phi Coprocessors on One Compute Node.

Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform

CNNLab: a Novel Parallel Framework for Neural Networks using GPU and FPGA-a Practical Study with Trade-off Analysis

NUMERICAL SIMULATION OF PLANETARY FLUID DYNAMICS ON CPU-MIC HETEROGENEOUS MANY-CORE SYSTEMS

Integrated Model, Batch and Domain Parallelism in Training Neural Networks

Model Parallelism Optimization for CNN FPGA Accelerator

Synergy Exploration in Deploying Convolutional Neural Networks Across Distributed Neuromorphic System

Accelerating Large-Scale Biological Database Search on Xeon Phi-based Neo-Heterogeneous Architectures