Abstract:In recent years, machine learning and deep learning techniques such as deep neural networks and recurrent neural networks have found uses in diverse fields including computer vision, speech recognition, natural language processing, social network analysis, bioinformatics and medicine, where they have produced results comparable to and in some cases surpassing human experts. Machine learning requires large amount of data for training its models with much of this data residing in object storage, an inexpensive and scalable data store. Also, deep learning make use of state of the art processing capabilities from high-end GPUs and accelerators, such as Google Tensor Processing Units (TPUs), which enable parallel and efficient execution. The throughput that such GPUs can support is very high. This however constitutes an impedance mismatch as the object storage is not designed for high performance data transfers and standard practices for feeding deep learning models from the object storage can result in poor training performance. Furthermore, the typical deep learning framework uses a file access interface, and object storage support a REST based interface with different APIs and semantics than a file system [2]. To fully take advantage of these GPUs and operate at full utilization, frameworks, such as TensorFlow, Cafe, and Torch, needs to deliver data as fast as possible to keep the GPUs busy. This becomes a significant challenge when the training data does not reside in the same machine as the GPUs, as is the case when using object storage, resulting in a utilization challenge for the expensive processing units. To solve the impedance mismatch and keep the processing units fully utilized, we have added a FUSE based file system, S3fs [1], to our deep learning stack. S3fs translates POSIX file API requests into REST API against the object storage. It is an open source project which, as part of this work, we optimized so that read requests are performed using new innovative logic that translates the requests into multiple concurrent range reads requests against the object storage. This enables us to obtain higher throughput from the object storage than is possible using the naive approach. Reads are cached in memory and are served back to the deep learning framework asynchronously. Since deep learning frameworks often run their training in multiple epochs the in memory cache speed is highly beneficial. Our FUSE based architecture has been implemented in the Deep Learning as a Service offering on the IBM Cloud, and our S3fs enhancements have been contributed to the S3fs project repository. Using our architecture we are able to speed up deep learning performance many folds and keep expensive GPUs fully utilized.

I/O Characterization and Performance Evaluation of BeeGFS for Deep Learning

P F ] 1 3 A ug 2 01 9 HPC AI 500 : A Benchmark Suite for HPC AI Systems

High Performance I/O For Large Scale Deep Learning

FanStore: Enabling Efficient and Scalable I/O for Distributed Deep Learning

I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey

Analysis of Performance and Optimization in MindSpore on Ascend NPUs

Keeping deep learning GPUs well fed using object storage

Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning

DFS-Perf : A Scalable and Unified Benchmarking Framework for Distributed File Systems

Understanding I/O Performance of IPFS Storage

CoFB: latency-constrained co-scheduling of flows and batches for deep learning inference service on the CPU–GPU system

Black-hole threshold solutions in stiff fluid collapse

FPDeep: Scalable Acceleration of CNN Training on Deeply-Pipelined FPGA Clusters

Toward Comprehensive and Accurate Simulation Performance Prediction of Parallel File Systems

A Case Study on the Efficiency of User-Level Parallel File Systems

Octopus: A Heterogeneous In-network Computing Accelerator Enabling Deep Learning for network

HadaFS: A File System Bridging the Local and Shared Burst Buffer for Exascale Supercomputers

HARL: Optimizing Parallel File Systems with Heterogeneity-Aware Region-Level Data Layout

24/7 Characterization of petascale I/O workloads

tf-Darshan: Understanding Fine-grained I/O Performance in Machine Learning Workloads

Performance-Aware Data Placement in Hybrid Parallel File Systems.