Abstract:Distributed Machine learning has delivered considerable advances in training neural networks by leveraging parallel processing, scalability, and fault tolerance to accelerate the process and improve model performance. However, training of large-size models has exhibited numerous challenges, due to the gradient dependence that conventional approaches integrate. To improve the training efficiency of such models, gradient-free distributed methodologies have emerged fostering the gradient-independent parallel processing and efficient utilization of resources across multiple devices or nodes. However, such approaches, are usually restricted to specific applications, due to their conceptual limitations: computational and communicational requirements between partitions, limited partitioning solely into layers, limited sequential learning between the different layers, as well as training a potential model in solely synchronous mode. In this paper, we propose and evaluate, the Neuro-Distributed Cognitive Adaptive Optimization (ND-CAO) methodology, a novel gradient-free algorithm that enables the efficient distributed training of arbitrary types of neural networks, in both synchronous and asynchronous manner. Contrary to the majority of existing methodologies, ND-CAO is applicable to any possible splitting of a potential neural network, into blocks (partitions), with each of the blocks allowed to update its parameters fully asynchronously and independently of the rest of the blocks. Most importantly, no data exchange is required between the different blocks during training with the only information each block requires is the global performance of the model. Convergence of ND-CAO is mathematically established for generic neural network architectures, independently of the particular choices made, while four comprehensive experimental cases, considering different model architectures and image classification tasks, validate the algorithms' robustness and effectiveness in both synchronous and asynchronous training modes. Moreover, by conducting a thorough comparison between synchronous and asynchronous ND-CAO training, the algorithm is identified as an efficient scheme to train neural networks in a novel gradient-independent, distributed, and asynchronous manner, delivering similar – or even improved results in Loss and Accuracy measures.

A parallel computing platform for training large scale neural networks

Distributed Framework of Artificial Neural Network for Big Data Analysis

Fuzzy Method and Neural Network Model Parallel Implementation of Multi-Layer Neural Network Based on Cloud Computing for Real Time Data Transmission in Large Offshore Platform

Mapping Very Large Scale Spiking Neuron Network to Neuromorphic Hardware.

Neuro-distributed cognitive adaptive optimization for training neural networks in a parallel and asynchronous manner

Large Scale Artificial Neural Network Training Using Multi-GPUs

Coded Parallelism for Distributed Deep Learning.

Training Large Scale Deep Neural Networks on the Intel Xeon Phi Many-Core Coprocessor

A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs

Integrated Model, Batch and Domain Parallelism in Training Neural Networks

Artificial Neural Network Computation On Graphic Process Unit

HPH: Hybrid Parallelism on Heterogeneous Clusters for Accelerating Large-scale DNNs Training.

A High Energy-Efficiency Multi-core Neuromorphic Architecture for Deep SNN Training

ByteGNN: Efficient Graph Neural Network Training at Large Scale

A Novel Co-design Peta-scale Heterogeneous Cluster for Deep Learning Training

A novel neural network model with distributed evolutionary approach for big data classification

Parallelizing Training of Deep Generative Models on Massive Scientific Datasets

Using GPI-2 for Distributed Memory Paralleliziation of the Caffe Toolbox to Speed up Deep Neural Network Training

Towards Efficient Large-Scale Graph Neural Network Computing.

A Hybrid Parallelization Approach for Distributed and Scalable Deep Learning

HierTrain: Fast Hierarchical Edge AI Learning with Hybrid Parallelism in Mobile-Edge-Cloud Computing