Abstract:Hadoop Distributed File System (HDFS) is designed to reliably storage and manage large-scale files. All the files in HDFS are managed by a single server, the NameNode. The NameNode stores metadata, in its main memory, for each file stored into HDFS. HDFS suffers the penalty of performance with increased number of small files. It imposes a heavy burden to the NameNode to store and manage a mass of small files. The number of files that can be stored into HDFS is constrained by the size of NameNode's main memory. In order to improve the efficiency of storing and accessing the small files on HDFS, we propose Small Hadoop Distributed File System (SHDFS), which bases on original HDFS. Compared to original HDFS, we add two novel modules in the proposed SHDFS: merging module and caching module. In merging module, the correlated files model is proposed, which is used to find out the correlated files by user-based collaborative filtering and then merge correlated files into a single large file to reduce the total number of files. In caching module, we use Log - linear model to dig out some hot-spot data that user frequently access to, and then design a special memory subsystem to cache these hot-spot data. Caching mechanism speeds up access to hot-spot data. The experimental results indicate that SHDFS is able to reduce the metadata footprint on NameNode's main memory and also improve the efficiency of storing and accessing large number of small files.

THE Optimization of HDFS Based on Small Files

An Optimized Approach for Storing and Accessing Small Files on Cloud Storage

A Strategy for Small Files Processing in HDFS.

A Novel Scalable Architecture of Cloud Storage System for Small Files Based on P2P

Storage Optimization Method of Small Files Based on Hadoop

Research on Small Files Optimized Storage Strategy in Hadoop System

An Optimized Strategy for Small Files Storing and Accessing in HDFS

Efficient File Accessing Techniques on Hadoop Distributed File Systems.

An Optimized Method of HDFS for Massive Small Files Storage

Optimization of small files accessed base on MapFile in HDFS

An Optimized Approach for Storing Small Files on HDFS-based on Dynamic Queue

Optimization strategy of Hadoop small file storage for big data in healthcare

A Strategy to Deal with Mass Small Files in HDFS

Improving the Storage Efficiency of Small Files in Cloud Storage

Sfs: A Massive Small File Processing Middleware in Hadoop

Optimization of Cloud Computing Storage Based on Stackable File System

Hadoop Massive Small File Merging Technology Based on Visiting Hot-Spot and Associated File Optimization

Research on Optimization Method of Merging and Prefetching for Massive Small Files in HDFS

Storage-Optimization Method for Massive Small Files of Agricultural Resources Based on Hadoop

A Method to Improve the Performance for Storing Massive Small Files in Hadoop

A Novel Approach in Improving I/O Performance of Small Meteorological Files on HDFS