A Survey on Auto-Parallelism of Neural Networks Training

Peng Liang,Yu Tang,Xiaoda Zhang,Youhui Bai,Teng Su,linbo qiao,Zhiquan Lai,Dongsheng Li

DOI: https://doi.org/10.36227/techrxiv.19522414

2022-01-01

Abstract:The datas in this survey were collected from reference papers.

What problem does this paper attempt to address?

A Survey on Auto-Parallelism of Large-Scale Deep Learning Training

Peng Liang,Yu Tang,Xiaoda Zhang,Youhui Bai,Teng Su,Zhiquan Lai,Linbo Qiao,Dongsheng Li

DOI: https://doi.org/10.1109/tpds.2023.3281931

IF: 5.3

2023-07-04

IEEE Transactions on Parallel and Distributed Systems

Abstract:Deep learning (DL) has gained great success in recent years, leading to state-of-the-art performance in research community and industrial fields like computer vision and natural language processing. One of the reasons for this success is the huge amount parameters adopted in DL models. However, it is impractical to train a moderately large model with a large number of parameters on a typical single device. Thus, It is necessary to train DL models in clusters with distributed training algorithms. However, traditional distributed training algorithms are usually sub-optimal and highly customized, which owns the drawbacks to train large-scale DL models in varying computing clusters. To handle the above problem, researchers propose auto-parallelism, which is promising to train large-scale DL models efficiently and practically in various computing clusters. In this survey, we perform a broad and thorough investigation on challenges, basis, and strategy searching methods of auto-parallelism in DL training. First, we abstract basic parallelism schemes with their communication cost and memory consumption in DL training. Further, we analyze and compare a series of current auto-parallelism works and investigate strategies and searching methods which are commonly used in practice. At last, we discuss several trends in auto-parallelism which are promising in further research.

computer science, theory & methods,engineering, electrical & electronic
An Empirical Study on Improving the Speed and Generalization of Neural Networks Using a Parallel Circuit Approach

Kien Tuong Phan,Tomas Henrique Maul,Tuong Thuy Vu

DOI: https://doi.org/10.1007/s10766-016-0435-4

2016-05-12

International Journal of Parallel Programming

Abstract:One of the common problems of neural networks, especially those with many layers, consists of their lengthy training time. We attempt to solve this problem at the algorithmic level, proposing a simple parallel design which is inspired by the parallel circuits found in the human retina. To avoid large matrix calculations, we split the original network vertically into parallel circuits and let the backpropagation algorithm flow in each subnetwork independently. Experimental results have shown the speed advantage of the proposed approach but also point out that this advantage is affected by multiple dependencies. The results also suggest that parallel circuits improve the generalization ability of neural networks presumably due to automatic problem decomposition. By studying network sparsity, we partly justified this theory and proposed a potential method for improving the design.

computer science, theory & methods
Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis

Tal Ben-Nun,Torsten Hoefler

DOI: https://doi.org/10.48550/arXiv.1802.09941

2018-09-15

Abstract:Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this survey, we describe the problem from a theoretical perspective, followed by approaches for its parallelization. We present trends in DNN architectures and the resulting implications on parallelization strategies. We then review and model the different types of concurrency in DNNs: from the single operator, through parallelism in network inference and training, to distributed deep learning. We discuss asynchronous stochastic optimization, distributed system architectures, communication schemes, and neural architecture search. Based on those approaches, we extrapolate potential directions for parallelism in deep learning.

Machine Learning,Computer Vision and Pattern Recognition,Distributed, Parallel, and Cluster Computing,Neural and Evolutionary Computing
Auto-GNAS: A Parallel Graph Neural Architecture Search Framework

Jiamin Chen,Jianliang Gao,Yibo Chen,Babatounde Moctard Oloulade,Tengfei Lyu,Zhao Li

DOI: https://doi.org/10.1109/TPDS.2022.3151895

IF: 5.3

2022-01-01

IEEE Transactions on Parallel and Distributed Systems

Abstract:Graph neural networks (GNNs) have received much attention as GNNs have recently been successfully applied on non-euclidean data. However, artificially designed graph neural networks often fail to get satisfactory model performance for a given graph data. Graph neural architecture search effectively constructs the GNNs that achieve the expected model performance with the rise of automatic machine learning. The challenge is efficiently and automatically getting the optimal GNN architecture in a vast search space. Existing search methods serially evaluate the GNN architectures, severely limiting system efficiency. To solve these problems, we develop an Automatic Graph Neural Architecture Search framework (Auto-GNAS) with parallel estimation to implement an automatic graph neural search process that requires almost no manual intervention. In Auto-GNAS, we design the search algorithm with multiple genetic searchers. Each searcher can simultaneously use evaluation feedback information, information entropy, and search results from other searchers based on sharing mechanism to improve the search efficiency. As far as we know, this is the first work using parallel computing to improve the system efficiency of graph neural architecture search. According to the experiment on the real datasets, Auto-GNAS obtain competitive model performance and better search efficiency than other search algorithms. Since the parallel estimation ability of Auto-GNAS is independent of search algorithms, we expand different search algorithms based on Auto-GNAS for scalability experiments. The results show that Auto-GNAS with varying search algorithms can achieve nearly linear acceleration with the increase of computing resources.
Model Parallelism on Distributed Infrastructure: A Literature Review from Theory to LLM Case-Studies

Felix Brakel,Uraz Odyurt,Ana-Lucia Varbanescu

2024-03-06

Abstract:Neural networks have become a cornerstone of machine learning. As the trend for these to get more and more complex continues, so does the underlying hardware and software infrastructure for training and deployment. In this survey we answer three research questions: "What types of model parallelism exist?", "What are the challenges of model parallelism?", and "What is a modern use-case of model parallelism?" We answer the first question by looking at how neural networks can be parallelised and expressing these as operator graphs while exploring the available dimensions. The dimensions along which neural networks can be parallelised are intra-operator and inter-operator. We answer the second question by collecting and listing both implementation challenges for the types of parallelism, as well as the problem of optimally partitioning the operator graph. We answer the last question by collecting and listing how parallelism is applied in modern multi-billion parameter transformer networks, to the extend that this is possible with the limited information shared about these networks.

Distributed, Parallel, and Cluster Computing,Machine Learning
Dynamic Universal Approximation Theory: Foundations for Parallelism in Neural Networks

Wei Wang,Qing Li

2024-11-29

Abstract:Neural networks are increasingly evolving towards training large models with big data, a method that has demonstrated superior performance across many tasks. However, this approach introduces an urgent problem: current deep learning models are predominantly serial, meaning that as the number of network layers increases, so do the training and inference times. This is unacceptable if deep learning is to continue advancing. Therefore, this paper proposes a deep learning parallelization strategy based on the Universal Approximation Theorem (UAT). From this foundation, we designed a parallel network called Para-Former to test our theory. Unlike traditional serial models, the inference time of Para-Former does not increase with the number of layers, significantly accelerating the inference speed of multi-layer networks. Experimental results validate the effectiveness of this network.

Machine Learning,Artificial Intelligence
A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions

Pengzhen Ren,Yun Xiao,Xiaojun Chang,Po-Yao Huang,Zhihui Li,Xiaojiang Chen,Xin Wang

DOI: https://doi.org/10.48550/arXiv.2006.02903

IF: 5.414

2020-06-01

Machine Learning

Abstract:Deep learning has made breakthroughs and substantial in many fields due to its powerful automatic representation capabilities. It has been proven that neural architecture design is crucial to the feature representation of data and the final performance. However, the design of the neural architecture heavily relies on the researchers' prior knowledge and experience. And due to the limitations of human' inherent knowledge, it is difficult for people to jump out of their original thinking paradigm and design an optimal model. Therefore, an intuitive idea would be to reduce human intervention as much as possible and let the algorithm automatically design the neural architecture. Neural Architecture Search (NAS) is just such a revolutionary algorithm, and the related research work is complicated and rich. Therefore, a comprehensive and systematic survey on the NAS is essential. Previously related surveys have begun to classify existing work mainly based on the key components of NAS: search space, search strategy, and evaluation strategy. While this classification method is more intuitive, it is difficult for readers to grasp the challenges and the landmark work involved. Therefore, in this survey, we provide a new perspective: beginning with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these early NAS algorithms, and then providing solutions for subsequent related research work. Besides, we conduct a detailed and comprehensive analysis, comparison, and summary of these works. Finally, we provide some possible future research directions.
Parallel Learning: Overview and Perspective for Computational Learning Across Syn2Real and Sim2Real

Qinghai Miao,Yisheng Lv,Min Huang,Xiao Wang,Fei-Yue Wang

DOI: https://doi.org/10.1109/jas.2023.123375

2023-03-04

IEEE/CAA Journal of Automatica Sinica

Abstract:The virtual-to-real paradigm, i.e., training models on virtual data and then applying them to solve real-world problems, has attracted more and more attention from various domains by successfully alleviating the data shortage problem in machine learning. To summarize the advances in recent years, this survey comprehensively reviews the literature, from the viewport of parallel intelligence. First, an extended parallel learning framework is proposed to cover main domains including computer vision, natural language processing, robotics, and autonomous driving. Second, a multi-dimensional taxonomy is designed to organize the literature in a hierarchical structure. Third, the related virtual-to-real works are analyzed and compared according to the three principles of parallel learning known as description, prediction, and prescription, which cover the methods for constructing virtual worlds, generating labeled data, domain transferring, model training and testing, as well as optimizing the strategies to guide the task-oriented data generator for better learning performance. Key issues remained in virtual-to-real are discussed. Furthermore, the future research directions from the viewpoint of parallel learning are suggested.

automation & control systems
A Survey of Neural Trees

Haoling Li,Jie Song,Mengqi Xue,Haofei Zhang,Jingwen Ye,Lechao Cheng,Mingli Song

DOI: https://doi.org/10.48550/arXiv.2209.03415

2022-09-08

Abstract:Neural networks (NNs) and decision trees (DTs) are both popular models of machine learning, yet coming with mutually exclusive advantages and limitations. To bring the best of the two worlds, a variety of approaches are proposed to integrate NNs and DTs explicitly or implicitly. In this survey, these approaches are organized in a school which we term as neural trees (NTs). This survey aims to present a comprehensive review of NTs and attempts to identify how they enhance the model interpretability. We first propose a thorough taxonomy of NTs that expresses the gradual integration and co-evolution of NNs and DTs. Afterward, we analyze NTs in terms of their interpretability and performance, and suggest possible solutions to the remaining challenges. Finally, this survey concludes with a discussion about other considerations like conditional computation and promising directions towards this field. A list of papers reviewed in this survey, along with their corresponding codes, is available at: <a class="link-external link-https" href="https://github.com/zju-vipa/awesome-neural-trees" rel="external noopener nofollow">this https URL</a>

Machine Learning,Artificial Intelligence
Parallel Algorithms Align with Neural Execution

Valerie Engelmayer,Dobrik Georgiev,Petar Veličković

2024-01-03

Abstract:Neural algorithmic reasoners are parallel processors. Teaching them sequential algorithms contradicts this nature, rendering a significant share of their computations redundant. Parallel algorithms however may exploit their full computational power, therefore requiring fewer layers to be executed. This drastically reduces training times, as we observe when comparing parallel implementations of searching, sorting and finding strongly connected components to their sequential counterparts on the CLRS framework. Additionally, parallel versions achieve (often strongly) superior predictive performance.

Machine Learning
A Survey of Neural Trees: Co-Evolving Neural Networks and Decision Trees

Haoling Li,Jie Song,Mengqi Xue,Haofei Zhang,Mingli Song

DOI: https://doi.org/10.1109/TNNLS.2024.3446891

2024-10-07

Abstract:Neural networks (NNs) and decision trees (DTs) are both popular models of machine learning, yet coming with mutually exclusive advantages and limitations. To bring the best of the two worlds, a variety of approaches are proposed to integrate NNs and DTs explicitly or implicitly. In this survey, these approaches are organized in a school which we term neural trees (NTs). This survey aims to present a comprehensive review of NTs and explore in detail how they enhance the model interpretability. Our first contribution is a detailed taxonomy of NTs, which characterizes the seamless integration and co-evolution of NNs and DTs. Subsequently, we analyze NTs in terms of their interpretability and performance and suggest potential solutions to the remaining challenges. Finally, this survey concludes with a discussion about other considerations like conditional computation and promising directions toward this field. A list of papers reviewed in this survey, along with their corresponding codes, is available at: https://github.com/ zju-vipa/awesome-neural-trees.
Research and development of neural network ensembles: a survey

Hui Li,Xuesong Wang,Shifei Ding

DOI: https://doi.org/10.1007/s10462-016-9535-1

IF: 9.588

2017-01-18

Artificial Intelligence Review

Abstract:A Neural Network Ensemble (NNE) combines the outputs of several individually trained neural networks in order to improve generalization performance. This article summarizes different approaches on the development and the latest studies on NNE. The introduction of the basic principles of NNE is followed by detailed descriptions of individual neural network generation method, conclusion generation method and fusion based on granular computing and NNE. In addition, for each of these methods we provide a short taxonomy in terms of their relevant characteristics, and analyze several of NNE applications, classic algorithms and contributions on various fields.

computer science, artificial intelligence
A parallel computing platform for training large scale neural networks

Rong Gu,Furao Shen,Yihua Huang

DOI: https://doi.org/10.1109/BigData.2013.6691598

2013-01-01

Abstract:Artificial neural networks (ANNs) have been proved to be successfully used in a variety of pattern recognition and data mining applications. However, training ANNs on large scale datasets are both data-intensive and computation-intensive. Therefore, large scale ANNs are used with reservation for their time-consuming training to get high precision. In this paper, we present cNeural, a customized parallel computing platform to accelerate training large scale neural networks with the backpropagation algorithm. Unlike many existing parallel neural network training systems working on thousands of training samples, cNeural is designed for fast training large scale datasets with millions of training samples. To achieve this goal, firstly, cNeural adopts HBase for large scale training dataset storage and parallel loading. Secondly, it provides a parallel in-memory computing framework for fast iterative training. Third, we choose a compact, event-driven messaging communication model instead of the heartbeat polling model for instant messaging delivery. Experimental results show that the overhead time cost by data loading and messaging communication is very low in cNeural and cNeural is around 50 times faster than the solution based on Hadoop MapReduce. It also achieves nearly linear scalability and excellent load balancing.
Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey

Zhihong Liu,Xin Xu,Peng Qiao,Dongsheng Li

2024-11-08

Abstract:Deep reinforcement learning has led to dramatic breakthroughs in the field of artificial intelligence for the past few years. As the amount of rollout experience data and the size of neural networks for deep reinforcement learning have grown continuously, handling the training process and reducing the time consumption using parallel and distributed computing is becoming an urgent and essential desire. In this paper, we perform a broad and thorough investigation on training acceleration methodologies for deep reinforcement learning based on parallel and distributed computing, providing a comprehensive survey in this field with state-of-the-art methods and pointers to core references. In particular, a taxonomy of literature is provided, along with a discussion of emerging topics and open issues. This incorporates learning system architectures, simulation parallelism, computing parallelism, distributed synchronization mechanisms, and deep evolutionary reinforcement learning. Further, we compare 16 current open-source libraries and platforms with criteria of facilitating rapid development. Finally, we extrapolate future directions that deserve further research.

Machine Learning,Artificial Intelligence,Distributed, Parallel, and Cluster Computing
Deep Learning and Its Parallelization

X. Li,G. Zhang,K. Li,W. Zheng

DOI: https://doi.org/10.1016/b978-0-12-805394-2.00004-0

IF: 4.426

2016-01-01

Big Data

Abstract:......................................................................................................................... 3 1 Introduction ............................................................................................................. 3 1.1 Application Background ............................................................................... 4 1.2 Performance Demands for Deep Learning ................................................... 4 1.3 Existing Parallel Frameworks of Deep Learning ......................................... 4 1.4 Chapter Organization ................................................................................... 5 2 Concepts and Categories of Deep Learning ............................................................ 5 2.1 Deep Learning ............................................................................................. 5 2.1.1 Artificial Neural Networks ................................................................ 5 2.1.2 Concept of Deep Learning ................................................................. 7 2.2 Mainstream Deep Learning Models ............................................................. 8 2.2.1 Autoencoders ..................................................................................... 8 2.2.2 Back Propagation ............................................................................... 9 2.2.3 Convolutional Neural Network ........................................................ 11 3 Parallel Optimization for Deep Learning .............................................................. 13 3.1 Convolutional Architecture for Fast Feature Embedding ......................... 13 3.1.1 Introduction ...................................................................................... 13 3.1.2 CUDA Programming ....................................................................... 14 3.1.3 Architecture of Caffe ....................................................................... 17 3.1.4 Parallel Implementation of Convolution in Caffe ............................ 18 3.2 DistBelief .................................................................................................. 20 3.2.1 Introduction of DistBelief ................................................................ 20 3.2.2 Downpour SGD ............................................................................... 20 3.2.4 Sandblaster L-BFGS ........................................................................ 21 3.3 Deep Learning Based-on Multi-GPUs ...................................................... 22 3.3.1 Data Parallelism ............................................................................... 22 3.3.2 Model Parallelism ............................................................................ 23 3.3.3 Data-Model Parallelism ................................................................... 24 3.3.4 Example System of Multi-GPUs ..................................................... 25 4 Discussions ........................................................................................................... 26 4.1 Grand Challenges of Deep Learning with Big Data ................................. 26 4.1.1 Massive Amounts of Training Sample ............................................ 26 4.1.2 Incremental Streaming Data ............................................................ 26 4.1.3 Learning Speed with Big Data ......................................................... 26 4.1.4 Scalability of Deep Models .............................................................. 27 4.2 Future Work .............................................................................................. 27 References .................................................................................................................... 28 Deep Learning and Its Parallelization: Concepts and Instances Xiaqing Li, Guangyan Zhang, Keqin Li, and Weimin Zheng Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism

Zhenkun Cai,Xiao Yan,Kaihao Ma,Yidi Wu,Yuzhen Huang,James Cheng,Teng Su,Fan Yu

DOI: https://doi.org/10.1109/tpds.2021.3132413

IF: 5.3

2022-08-01

IEEE Transactions on Parallel and Distributed Systems

Abstract:Effective parallelization strategies are crucial for the performance of distributed deep neural network (DNN) training. Recently, several methods have been proposed to search parallelization strategies but they all optimize a single objective (e.g., execution time, memory consumption) and produce only one strategy. We propose Frontier Tracking (FT), an efficient algorithm that finds a set of Pareto-optimal parallelization strategies to explore the best trade-off among different objectives. FT can minimize the memory consumption when the number of devices is limited and fully utilize additional resources to reduce the execution time. Based on FT, we develop a user-friendly system, called TensorOpt, which allows users to run their distributed DNN training jobs without caring the details about searching and coding parallelization strategies. Experimental results show that TensorOpt is more flexible in adapting to resource availability compared with existing frameworks.

computer science, theory & methods,engineering, electrical & electronic
Parallel Learning - A New Framework for Machine Learning

Li LI,Yi-Lun LIN,Dong-Pu CAO,Nan-Ning ZHENG,Fei-Yue WANG

DOI: https://doi.org/10.16383/j.aas.2017.y000001

2017-01-01

Abstract:In this paper, we propose a new framework of machine learning theory, parallel learning,which incorporates and inherits many elements from various existing machine learning theories. Special designs are also presented to deal with some important problems in the machine learning research field, e.g., useful data retrieval from big data using software defined artificial systems, combination of predictive learning and ensemble learning, application of Merton0s law to prescriptive learning.
TAP: Accelerating Large-Scale DNN Training Through Tensor Automatic Parallelisation

Ziji Shi,Le Jiang,Ang Wang,Jie Zhang,Xianyan Jia,Yong Li,Chencan Wu,Jialin Li,Wei Lin

DOI: https://doi.org/10.48550/arXiv.2302.00247

2023-02-01

Abstract:Model parallelism has become necessary to train large neural networks. However, finding a suitable model parallel schedule for an arbitrary neural network is a non-trivial task due to the exploding search space. In this work, we present a model parallelism framework TAP that automatically searches for the best data and tensor parallel schedules. Leveraging the key insight that a neural network can be represented as a directed acyclic graph, within which may only exist a limited set of frequent subgraphs, we design a graph pruning algorithm to fold the search space efficiently. TAP runs at sub-linear complexity concerning the neural network size. Experiments show that TAP is $20\times- 160\times$ faster than the state-of-the-art automatic parallelism framework, and the performance of its discovered schedules is competitive with the expert-engineered ones.

Machine Learning,Artificial Intelligence,Distributed, Parallel, and Cluster Computing
Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependencies

Wei Fang,Zhaofei Yu,Zhaokun Zhou,Ding Chen,Yanqi Chen,Zhengyu Ma,Timothée Masquelier,Yonghong Tian

2024-01-09

Abstract:Vanilla spiking neurons in Spiking Neural Networks (SNNs) use charge-fire-reset neuronal dynamics, which can only be simulated serially and can hardly learn long-time dependencies. We find that when removing reset, the neuronal dynamics can be reformulated in a non-iterative form and parallelized. By rewriting neuronal dynamics without reset to a general formulation, we propose the Parallel Spiking Neuron (PSN), which generates hidden states that are independent of their predecessors, resulting in parallelizable neuronal dynamics and extremely high simulation speed. The weights of inputs in the PSN are fully connected, which maximizes the utilization of temporal information. To avoid the use of future inputs for step-by-step inference, the weights of the PSN can be masked, resulting in the masked PSN. By sharing weights across time-steps based on the masked PSN, the sliding PSN is proposed to handle sequences of varying lengths. We evaluate the PSN family on simulation speed and temporal/static data classification, and the results show the overwhelming advantage of the PSN family in efficiency and accuracy. To the best of our knowledge, this is the first study about parallelizing spiking neurons and can be a cornerstone for the spiking deep learning research. Our codes are available at \url{<a class="link-external link-https" href="https://github.com/fangwei123456/Parallel-Spiking-Neuron" rel="external noopener nofollow">this https URL</a>}.

Neural and Evolutionary Computing,Computer Vision and Pattern Recognition
PaSE: Parallelization Strategies for Efficient DNN Training

Venmugil Elango

DOI: https://doi.org/10.1109/IPDPS49936.2021.00111

2024-07-04

Abstract:Training a deep neural network (DNN) requires substantial computational and memory requirements. It is common to use multiple devices to train a DNN to reduce the overall training time. There are several choices to parallelize each layer in a DNN. Exhaustively searching this list to find an optimal parallelization strategy is prohibitively time consuming and impractical. The standard practice is to use data parallelism because of its simplicity. However, data parallelism is often sub-optimal, and suffers from poor performance and high memory requirement. Expert-designed strategies have been proposed on a case-by-case basis using domain specific knowledge. These expert-designed strategies do not generalize well to DNNs other than the ones for which they were designed, and are not always necessarily the best choice. In this paper, we propose an approach to automatically find efficient parallelization strategies for DNNs from their computation graphs. We present an efficient algorithm to compute these strategies within a reasonable time in practice. We evaluate the effectiveness of our approach on various DNNs. We also compare the performance of the strategies identified by our approach against data parallelism, expert-designed strategies, and the state-of-the-art approaches. Our results show that the strategies found using our approach outperform the baseline data parallelism strategy in all the cases. In addition, our strategies achieve better performance than the expert-designed strategies and the state-of-the-art approaches.

Machine Learning,Distributed, Parallel, and Cluster Computing

A Survey on Auto-Parallelism of Neural Networks Training

A Survey on Auto-Parallelism of Large-Scale Deep Learning Training

An Empirical Study on Improving the Speed and Generalization of Neural Networks Using a Parallel Circuit Approach

Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis

Auto-GNAS: A Parallel Graph Neural Architecture Search Framework

Model Parallelism on Distributed Infrastructure: A Literature Review from Theory to LLM Case-Studies

Dynamic Universal Approximation Theory: Foundations for Parallelism in Neural Networks

A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions

Parallel Learning: Overview and Perspective for Computational Learning Across Syn2Real and Sim2Real

A Survey of Neural Trees

Parallel Algorithms Align with Neural Execution

A Survey of Neural Trees: Co-Evolving Neural Networks and Decision Trees

Research and development of neural network ensembles: a survey

A parallel computing platform for training large scale neural networks

Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey

Deep Learning and Its Parallelization

TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism

Parallel Learning - A New Framework for Machine Learning

TAP: Accelerating Large-Scale DNN Training Through Tensor Automatic Parallelisation

Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependencies

PaSE: Parallelization Strategies for Efficient DNN Training