Abstract:The accuracy of the neural networks can usually be improved by increasing the size of the dataset and the layers or operators of the network, as it has strong composability. But, it makes a challenge to train these models efficiently, due to the limited resources of a single device, such as memory, computational resource and so on. For this challenge, it has become an inevitable trend to automatically implement model parallelism across multiple devices. This paper proposes Aware, an adaptive distributed parallel training method, to search distributed parallel strategy automatically for deep learning model. It firstly proposes an operator fusion strategy with computation and communication awareness to simplify the computational graph of deep learning model. And then, it introduces position-aware graph embedding algorithm to extract the structural features of models, which can make the searched parallel strategies to transplant to other networks with similar structures. On this basis, we use reinforcement learning algorithm to search distributed parallel strategy automatically. This paper makes experiments with neural networks such as Inception V3, NMT, GNM, NasNet and ResNet, and uses CIFAR10 and PTB datasets to compare the two methods of Hierarchical and Placeto. The results show that compared with Placeto, Aware achieves up to 5% reductions in runtime. Aware achieves more than 8% reductions in runtime compared with Hierarchical. Moreover, it has better transplantation and generalization capabilities, and supports pre-training for large-scale network parallel strategy search and accelerates convergence.

MapReduce Based Parallel Neural Networks in Enabling Large Scale Machine Learning

Parallelizing Backpropagation Neural Network Using MapReduce and Cascading Model.

A parallel computing platform for training large scale neural networks

The Parallelization of Back Propagation Neural Network in MapReduce and Spark

Parallel Graph Pattern Matching in Massive Networks Based on MapReduce

Parallel Implementation of Multilayered Neural Networks Based on Map-Reduce on Cloud Computing Clusters.

Implementation of Artificial Neural Networks in MapReduce Optimization

Research on parallel strategy of convolution neural network in distributed environment

Large-scale Artificial Neural Network: MapReduce-based Deep Learning.

Fuzzy Method and Neural Network Model Parallel Implementation of Multi-Layer Neural Network Based on Cloud Computing for Real Time Data Transmission in Large Offshore Platform

Big Data: A Parallel Particle Swarm Optimization-Back-Propagation Neural Network Algorithm Based on MapReduce

Parallel Implementation Of Classification Algorithms Based On Mapreduce

Evaluating Large Graph Processing in MapReduce Based on Message Passing

A MapReduce Based High Performance Neural Network in Enabling Fast Stability Assessment of Power Systems

Mapping Very Large Scale Spiking Neuron Network to Neuromorphic Hardware.

Vhadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration

Parallel implementation of multilayered neural networks based on Map-Reduce on cloud computing clusters

Aware: Adaptive Distributed Training with Computation, Communication and Position Awareness for Deep Learning Model.

MapReduce-based Backpropagation Neural Network over large scale mobile data

Distributed Framework of Artificial Neural Network for Big Data Analysis

A Hybrid Parallelization Approach for Distributed and Scalable Deep Learning