Abstract:Distributed Machine learning has delivered considerable advances in training neural networks by leveraging parallel processing, scalability, and fault tolerance to accelerate the process and improve model performance. However, training of large-size models has exhibited numerous challenges, due to the gradient dependence that conventional approaches integrate. To improve the training efficiency of such models, gradient-free distributed methodologies have emerged fostering the gradient-independent parallel processing and efficient utilization of resources across multiple devices or nodes. However, such approaches, are usually restricted to specific applications, due to their conceptual limitations: computational and communicational requirements between partitions, limited partitioning solely into layers, limited sequential learning between the different layers, as well as training a potential model in solely synchronous mode. In this paper, we propose and evaluate, the Neuro-Distributed Cognitive Adaptive Optimization (ND-CAO) methodology, a novel gradient-free algorithm that enables the efficient distributed training of arbitrary types of neural networks, in both synchronous and asynchronous manner. Contrary to the majority of existing methodologies, ND-CAO is applicable to any possible splitting of a potential neural network, into blocks (partitions), with each of the blocks allowed to update its parameters fully asynchronously and independently of the rest of the blocks. Most importantly, no data exchange is required between the different blocks during training with the only information each block requires is the global performance of the model. Convergence of ND-CAO is mathematically established for generic neural network architectures, independently of the particular choices made, while four comprehensive experimental cases, considering different model architectures and image classification tasks, validate the algorithms' robustness and effectiveness in both synchronous and asynchronous training modes. Moreover, by conducting a thorough comparison between synchronous and asynchronous ND-CAO training, the algorithm is identified as an efficient scheme to train neural networks in a novel gradient-independent, distributed, and asynchronous manner, delivering similar – or even improved results in Loss and Accuracy measures.

Towards a Flexible and High-Fidelity Approach to Distributed DNN Training Emulation

Proteus: Simulating the Performance of Distributed DNN Training

EFLOPS: Algorithm and System Co-Design for a High Performance Distributed Training Platform

DistSim: A Performance Model of Large-Scale Hybrid Distributed DNN Training

Distributed SLIDE: Enabling Training Large Neural Networks on Low Bandwidth and Simple CPU-Clusters via Model Parallelism and Sparsity

Automated Deep Neural Network Inference Partitioning for Distributed Embedded Systems

BFTrainer: Low-Cost Training of Neural Networks on Unfillable Supercomputer Nodes

Joint Dynamic Data and Model Parallelism for Distributed Training of DNNs over Heterogeneous Infrastructure

Performance Modeling of Distributed Deep Neural Networks

MalleTrain: Deep Neural Network Training on Unfillable Supercomputer Nodes

TBD: Benchmarking and Analyzing Deep Neural Network Training

Scalable Network Emulation on Analog Neuromorphic Hardware

DynaHB: A Communication-Avoiding Asynchronous Distributed Framework with Hybrid Batches for Dynamic GNN Training

ONNXim: A Fast, Cycle-level Multi-core NPU Simulator

Network Emulation as a Service (NEaaS): Towards a Cloud-Based Network Emulation Platform

Rationing Bandwidth Resources for Mitigating Network Resource Contention in Distributed DNN Training Clusters.

Neuro-distributed cognitive adaptive optimization for training neural networks in a parallel and asynchronous manner

Learning the Optimal Partition for Collaborative DNN Training with Privacy Requirements

ByteGNN: Efficient Graph Neural Network Training at Large Scale

DropCompute: simple and more robust distributed synchronous training via compute variance reduction

Optimum splitting computing for DNN training through next generation smart networks: a multi-tier deep reinforcement learning approach