Abstract:Most research on novel techniques for 3D Medical Image Segmentation (MIS) is currently done using Deep Learning with GPU accelerators. The principal challenge of such technique is that a single input can easily cope computing resources, and require prohibitive amounts of time to be processed. Distribution of deep learning and scalability over computing devices is an actual need for progressing on such research field. Conventional distribution of neural networks consist in data parallelism, where data is scattered over resources (e.g., GPUs) to parallelize the training of the model. However, experiment parallelism is also an option, where different training processes are parallelized across resources. While the first option is much more common on 3D image segmentation, the second provides a pipeline design with less dependence among parallelized processes, allowing overhead reduction and more potential scalability. In this work we present a design for distributed deep learning training pipelines, focusing on multi-node and multi-GPU environments, where the two different distribution approaches are deployed and benchmarked. We take as proof of concept the 3D U-Net architecture, using the MSD Brain Tumor Segmentation dataset, a state-of-art problem in medical image segmentation with high computing and space requirements. Using the BSC MareNostrum supercomputer as benchmarking environment, we use TensorFlow and Ray as neural network training and experiment distribution platforms. We evaluate the experiment speed-up, showing the potential for scaling out on GPUs and nodes. Also comparing the different parallelism techniques, showing how experiment distribution leverages better such resources through scaling. Finally, we provide the implementation of the design open to the community, and the non-trivial steps and methodology for adapting and deploying a MIS case as the here presented.

Improving Barnes-Hut t-SNE Algorithm in Modern GPU Architectures with Random Forest KNN and Simulated Wide-Warp

Efficient Algorithms for t-distributed Stochastic Neighborhood Embedding

CUDA Optimization Strategies for Compute- and Memory-Bound Neuroimaging Algorithms

Accelerating hyperbolic t-SNE

Accelerating Fast Fourier Transforms Using Hadoop and CUDA

AtSNE: Efficient and Robust Visualization on GPU through Hierarchical Optimization

Accelerating Spatiotemporal Supervised Training of Large-Scale Spiking Neural Networks on GPU

Accelerating Genome-Wide Association Studies Using CUDA Compatible Graphics Processing Units

GPU accelerated matrix factorization of large scale data using block based approach

cuFasterTucker: A Stochastic Optimization Strategy for Parallel Sparse FastTucker Decomposition on GPU Platform

Performance Evaluation and Optimization of HBM-Enabled GPU for Data-Intensive Applications.

Accelerating Geospatial Analysis on GPUs Using CUDA

Parallel Randomized Tucker Decomposition Algorithms

Sparse MTTKRP Acceleration for Tensor Decomposition on GPU

Accelerating BIRCH for Clustering Large Scale Streaming Data Using CUDA Dynamic Parallelism.

Distributing Deep Learning Hyperparameter Tuning for 3D Medical Image Segmentation

Parallel Inference for Latent Dirichlet Allocation on Graphics Processing Units.

Distributed Out-of-Memory NMF on CPU/GPU Architectures

A Novel Parallel Algorithm for Sparse Tensor Matrix Chain Multiplication via TCU-Acceleration

A Parallel Implementation of the 2D Wavelet Transform Using CUDA