Abstract:In the big data era, the distributed file system is getting more and more significant due to the characteristics of its scale-out capability, high availability, and high performance. Different distributed file systems may have different design goals. For example, some of them are designed to have good performance for small file operations, such as GlusterFS, while some of them are designed for large file operations, such as Hadoop distributed file system. With the divergence of big data applications, a distributed file system may provide good performance for some applications but fails for some other applications, that is, there has no universal distributed file system that can produce good performance for all applications. In this paper, we propose a hybrid file system framework, HybridFS, which can deliver satisfactory performance for all applications. HybridFS is composed of multiple distributed file systems with the integration of advantages of these distributed file systems. In HybridFS, on top of multiple distributed file systems, we have designed a metadata management server to perform three functions: file placement, partial metadata store, and dynamic file migration. The file placement is performed based on a decision tree. The partial metadata store is performed for files whose size is less than a few hundred Bytes to increase throughput. The dynamic file migration is performed to balance the storage usage of distributed file systems without throttling performance. We have implemented HybridFS in java on eight nodes and choose Ceph, HDFS, and GlusterFS as designated distributed file systems. The experimental results show that, in the best case, HybridFS can have up to 30% performance improvement of read/write operations over a single distributed file system. In addition, if the difference of storage usage among multiple distributed file systems is less than 40%, the performance of HybridFS is guaranteed, that is, no performance degradation.

HadaFS: A File System Bridging the Local and Shared Burst Buffer for Exascale Supercomputers

DHFS: A High-Throughput Heterogeneous File System Based on Mainframe for Cloud Storage

Triple-H: A Hybrid Approach to Accelerate HDFS on HPC Clusters with Heterogeneous Storage Architecture

A High-Speed Heterogeneous Cluster File System for Network-Attached Storage

HSAStore - A Hierarchical Storage Architecture for Computing Systems Containing Large-Scale Intermediate Data.

Hybrid Hierarchy Storage System in MilkyWay-2 Supercomputer

HybridFS - A High Performance and Balanced File System Framework with Multiple Distributed File Systems

SQL-DFS: A Massive Small File Storage System Based on HDFS

ConeSSD: A Novel Policy to Optimize the Performance of HDFS Heterogeneous Storage.

HyF2FS: A Filesystem to Fully Exploit the Parallelism of Hybrid Storage

Nlss: A Near-Line Storage System Design Based on the Combination of Hdfs and Zfs

Boosting Parallel File System Performance Via Heterogeneity-Aware Selective Data Layout.

Big Data Analytics on Traditional HPC Infrastructure Using Two-Level Storage

Imfssc: An In-Memory Distributed File System Framework For Super Computing

Hysf: A Striped File Assignment Strategy for Parallel File System with Hybrid Storage

HAS: Heterogeneity-Aware Selective Data Layout Scheme for Parallel File Systems on Hybrid Servers

A Migratory Heterogeneity-Aware Data Layout Scheme for Parallel File Systems.

PeakFS: an Ultra-high Performance Parallel File System Via Computing-Network-Storage Co-optimization for HPC Applications

SpanFS: A Scalable File System on Fast Storage Devices.

Fully Decentralized Data Distribution for Exascale-HPC: End of the Provider-Demander Matching Puzzle

MHDFS: A Memory-Based Hadoop Framework for Large Data Storage.