An Efficient Big Data Storage Service Architecture
Xiang Li,Pei-Xiang Bai,Yan Wang,Jianmin Dong,Haibo Wang
DOI: https://doi.org/10.1109/CSCWD54268.2022.9776303
2022-05-04
Abstract:Due to the limitations of the distributed file storage system, when data storage is surging, the depth and width of file directory will continue to increase, which results in reduced I/O efficiency of metadata management. Rapidly growing data can also cause system congestion or data loss. In today's environment where information and data are linked and collaborative in all walks of life, this phenomenon is absolutely unacceptable. Object storage is an efficient way to store massive data through separating metadata. However, the original data storage system cluster has been configured with non-object-oriented storage data types. If the underlying storage system is replaced by object storage, it will not be compatible with the upper file parallel processing model. To solve this problem, this paper proposes a big data storage service architecture based on object storage. In the architecture, a transparent access method from file system to object storage cluster is designed to support efficient access to object storage system in the form of file. For improving the reliability of data storage service, a high fault tolerance mechanism based on multi-gateway is presented. Experimental results show that the architecture can effectively solve the problem of data access compatibility from file system to object storage system and significantly improve storage efficiency.
Computer Science,Engineering