Research on data storage structure and management model in hierarchical storage

Chun Yan,Sheng Xi Li,Ya Zhou Li

DOI: https://doi.org/10.4028/www.scientific.net/AMR.121-122.198

2010-01-01

Abstract:As hierarchical storage is one of the highlights in researching and realizing of mass data storage, this paper mainly describes a study on data storage structure, evaluation of data importance and data management model of hierarchical storage.

What problem does this paper attempt to address?

A Tree-Based Hierarchy Data Storage Framework in A Pervasive Space

Zhuzhong Qian,Ilsun You,Youyou Lu,Sanglu Lu

2011-01-01

Abstract:Context data is important information for catching the behaviors of applications in a pervasive space. To effectively store huge amount of data, treelike layered storage architecture is proposed, where the leaf nodes collect data from sensing devices. In order to integrate data from mobile devices, the related leaf nodes that get data from the same device should upload and store the data to the host node. This paper presents a deep study of the data storage problem and proposes a global algorithm GHS and an online algorithm DHS to dynamically select the host node, which reduces the communication cost significantly. This paper also gives the theoretical and experimental analysis of these algorithms, which shows both GHS and DHS are correct and effective.
Scientific data organization and management in numerical simulations

Chen Hong,Xia Fang,Song Lei,Zheng Weimin

DOI: https://doi.org/10.3321/j.issn:1671-4512.2005.z1.007

2005-01-01

Journal of Huazhong University of Science and Technology

Abstract:It is a crucial bottleneck of scientific computing performance to organize and manage the large scale scientific data in numerical simulations.Considering the complicated data characters such as multi-dimension,multivariable and time-space correlative,in this paper,a hierarchical data model for numerical computation is presented for scientific data organization so as to share and exchange data better between applications and platforms.By adding the data semantic information,flexible high-level application programming interfaces of accessing data are provided to facilitate ease of use for applications,like numerical computing,visualization and data analysis.The validity of the hierarchical data model for numerical computation is verified by some typical programs applying.Meanwhile,a scientific data management prototype system is designed,in which metadata technique is used for balancing efficiency and easyto-use through integrating high-performance file system and database.
A Study on Data Storage and Management for Massive Remote Sensing Data Based on Multi-level Grid Model

LI Shuang,CHENG Chengqi,TONG Xiaochong,CHEN Bo,ZHAI Weixin

DOI: https://doi.org/10.11947/j.agcs.2016.f013

2016-01-01

Abstract:With the rapid development of remote sensing technology, spatial information is exploding. For current remote sensing data storage management system, their data volume, rich data sources, query retrieves slow and other issues are problems to be solved. This paper then proposed a remote sensing data organization scheme based on GeoSOT. By firstly adding a GeoSOT code column which is array format in relational database, spatial information in the metadata can be stored and logically subdivided, in order to achieve unified storage and retrieval of image data space area. We compare our method with Oracle platform by simulating worldwide image data. Experimental results show that the retrieval efficiency of this article has obvious advantages and can effectively improve the integration of remote sensing data, retrieval efficiency. Our approach also offers a more effective storage management program for existing storage center or management system.
Study of Seamless Organization and Storage Structure for Massive Spatio-Temporal Data

Jiong Xie,Renyi Liu,Nan Liu

DOI: https://doi.org/10.1117/12.651254

2005-01-01

Abstract:Integrative management of massive spatio-temporal data and the relevant capabilities for time series analysis are important research targets of new generation GIS software architecture. At present, solid developments are imperatively needed in technology for effective organization and storage of massive spatio-temporal data. Oriented to spatio-temporal multi-dimension structure, a massive spatio-temporal data seamless organization and storage model based on spatio-temporal partition (STP), spaito-temporal clustering (STC) is presented, and the test on single table of 2GB to 60 GB shows that integration of STP and STC can improve performance of spaito-temporal search effectively.
Massive Data HBase Storage Method for Electronic Archive Management

Huaquan Su,Junwei Li,Li Guo,Wanshuo Wang,Yongjiao Yang,You Wen,Kai Li,Pingyan Mo

DOI: https://doi.org/10.1002/nem.2308

2024-10-28

International Journal of Network Management

Abstract:A study has proposed a massive data storage method for electronic archive management based on HBase, aiming to improve the intelligence, efficiency and retrieval performance of data storage. The results show that this method is superior to traditional database systems in terms of write speed and query latency and is suitable for efficient storage and management of massive electronic archives. The acceleration of the digitalization process in enterprise and university education management has generated a massive amount of electronic archive data. In order to improve the intelligence, storage quality, and efficiency of electronic records management and achieve efficient storage and fast retrieval of data storage models, this study proposes a massive data storage model based on HBase and its retrieval optimization scheme design. In addition, HDFS is introduced to construct a two‐level storage structure and optimize values to improve the scalability and load balancing of HBase, and the retrieval efficiency of the HBase storage model is improved through SL‐TCR and BF filters. The results indicated that HDFS could automatically recover data after node, network partition, and NameNode failures. The write time of HBase was 56 s, which was 132 and 246 s less than Cassandra and CockroachDB. The query latency was reduced by 23% and 32%, and the query time was reduced by 9988.51 ms, demonstrating high reliability and efficiency. The delay of BF‐SL‐TCL was 1379.28 s after 1000 searches, which was 224.78 and 212.74 s less than SL‐TCL and Blockchain Retrieval Acceleration and reduced the delay under high search times. In summary, this storage model has obvious advantages in storing massive amounts of electronic archive data and has high security and retrieval efficiency, which provides important reference for the design of storage models for future electronic archive management. The storage model designed by the research institute has obvious advantages in storing massive electronic archive data, solving the problem of lack of scalability in electronic archive management when facing massive data, and has high security and retrieval efficiency. It has important reference for the design of storage models for future electronic archive management.

computer science, information systems,telecommunications
Hierarchical Storage for Massive Social Network Data Based on Improved Decision Tree

Yanning Zhang,Guanghao Jin,Jingyu Li,Taizhong Zhang

DOI: https://doi.org/10.1007/s11036-024-02426-5

2024-10-01

Mobile Networks and Applications

Abstract:In order to reasonably store and optimize the read/write (w/r) efficiency of massive social network data, a hierarchical storage method for massive social network data based on improved decision tree is proposed. Improved decision tree is used to classify the social network data into multi-value level. Then the high-level, intermediate-level and low-level data are stored in three different storage containers, namely, memory, SSD and mechanical hard disk. The migration storage strategy with adaptive downgrading and upgrading based on the value-classified data is used to migrate and store the social network data reasonably by combining the data value level and the memory status of the storage containers. In addition, the CFS lock model is used to maintain the w/r integrity of the hierarchical storage. In the experiment, this method can reasonably store data of different value levels. The w/r latency of migration and storage for the social network data is in the range of [1.5ms, 2.1ms], [1.6ms, 2.0ms] respectively, which is extremely short. The packet loss rate is only 0.01%, and the data integrity is high. Meanwhile, it has high throughput and low variance that below 0.15, which can effectively ensure the global consistency of data.

computer science, information systems,telecommunications, hardware & architecture
Efficient Hierarchical Storage Management Framework Empowered by Reinforcement Learning

Tianru Zhang,Salman Toor,Andreas Hellander

DOI: https://doi.org/10.48550/arXiv.2201.11668

2022-01-12

Abstract:With the rapid development of big data and cloud computing, data management has become increasingly challenging. Over the years, a number of frameworks for data management and storage with various characteristics and features have become available. Most of these are highly efficient, but ultimately create data silos. It becomes difficult to move and work coherently with data as new requirements emerge as no single framework can efficiently fulfill the data management needs of diverse applications. A possible solution is to design smart and efficient hierarchical (multi-tier) storage solutions. A hierarchical storage system (HSS) is a meta solution that consists of different storage frameworks organized as a jointly constructed large storage pool. It brings a number of benefits including better utilization of the storage, cost-efficiency, and use of different features provided by the underlying storage frameworks. In order to maximize the gains of hierarchical storage solutions, it is important that they include intelligent and autonomous mechanisms for data management grounded in the features of the different underlying frameworks. These decisions should be made according to the characteristics of the dataset, tier status, and access patterns. These are highly dynamic parameters and defining a policy based on the mentioned parameters is a non-trivial task. This paper presents an open-source hierarchical storage framework with a dynamic migration policy based on reinforcement learning (RL). We present a mathematical model, a software architecture, and an implementation based on both simulations and a live cloud-based environment. We compare the proposed RL-based strategy to a baseline of three rule-based policies, showing that the RL-based policy achieves significantly higher efficiency and optimal data distribution in different scenarios compared to the dynamic rule-based policies.

Distributed, Parallel, and Cluster Computing
Design and Implementation of Hierarchical Storage System FQ-HSM Based on File Queue

Rui-jie ZHANG,Wen-sheng ZHANG,Zhan-huai LI

DOI: https://doi.org/10.3969/j.issn.1006-2475.2017.02.014

2017-01-01

Abstract:Through simplifying the architecture of hierarchical storage system, this paper puts forward a hierarchical storage sys-tem FQ-HSM based on file queue. Based on the FVAP( File Value Accurate Prediction) , FQ-HSM calculates the system relative values of files and builds file queue, then it decides the data migration plan through storage early warning mechanism. After tes-ting FQ-HSM, the results show that it has had due functions: data classifying, data automatically migrating, user transparently accessing and so on. FQ-HSM shows a good performance compared with present data classifying methods in hit rate and back-mi-gration rate.
The Stndy of Massive lnformation Management System 13ased on Multilevel Storage Architecture

石晶,周立柱

DOI: https://doi.org/10.3969/j.issn.1002-137X.2000.09.003

2000-01-01

Computer Science

Abstract:It is very important for such large-scale application systems as global information systems, GIS,digital libraries, data warehousing to effectively manage and manipulate the large volumes of infor marion(more than terabytes of data). One of challenge problems is to study massive information management system and related techniques. Since the large volumes of information are mainly stored on ter tiary storage devices,the study of massive information management system should focus on the following aspects. (1)the multilevel storage architecture in which the tertiary storage devices are main data media and the secondary storage devices are data cache; (2)scalable system architecture of massive in formation management system; (3)retrieve process techniques. This paper wilt deeply study and discuss the above problems.
Hierarchical Architecture Design of Computer System

Chuang LIN,Chao XUE,Jie HU,Wen-Zhuo LI

DOI: https://doi.org/10.11897/SP.J.1016.2017.01996

2017-01-01

Chinese Journal of Computers

Abstract:Computer system is composed by several layers and sub-systems which link with each other with complex interactions between them.The overall architecture design of computer system is very important.Many design principles were proposed during the evolution and development of computer system architecture,among which hierarchical architecture design is an efficient way for large system design.Since the literature elaborated by Dijkstra,hierarchical architecture design principle of computer system architecture has been proposed for many years,and it is an important part of computer system architecture design.Hierarchy design exists widely in computer science and system,such as computer architecture design,network architecture design,cloud computing,network virtualization,software engineering and many other branches.Computer science and technology is evolving and renovating with a high frequency,and it is extremely important to propose a general model framework and design mechanism for hierarchy design.Though some literatures have been explored on hierarchy design of system architecture,few works explored the essence of the principles and the methods of hierarchy design,let alone general model framework and evaluation metric.The main insufficiencies appear as follows:(1) the descriptions of hierarchy design lack of formal descriptions;(2) the analyses of hierarchy design mainly focus on specific system and application lacking of understanding on the essence of the principles and the methods;(3) the models of hierarchy design are also limited to the target system lack of general model framework and evaluation metrics.This paper shed lights on these mentioned issues of hierarchical architecture design of computer systems in the following procedures.Firstly,the basic concepts and preliminary formal definitions related to hierarchy design are given.After that,this paper surveys the state of the arts of hierarchy design through four categories named as hierarchy model design,hierarchy component design,hierarchy cross-layer design,and hierarchy overlay design respectively.Hierarchy model design includes description model and quantification model giving mathematical description and deduction to specific characteristics of the target system;Hierarchy component design abstractsa given substructure of the target system into a simple one with its core function;Hierarchy cross-layer design generates new virtual links and interactions beside the original hierarchical structure for specific demands;Hierarchy overlay design uses virtual nodes and logical links to form a flexible virtual plane.On that basis,the essence of basic concepts,design principles,main mechanisms,and implementation method of hierarchy design are explored.We summarize simplification and efficiency as two essential principles.They also imply the objectives of hierarchy design at the same time.These two principles are achieved by abstraction mechanism and virtualization mechanism which are formally defined by plane mappings.Accordingly,we establish an evaluation framework for complexity and performance of hierarchical structure,give their formal expressions,and obtain some theorems.This paper also makes essential introduction and discussion on the hierarchical structure of three typical systems,i.e.super computer system,software defined network,and cloud computing system with complexity and performance evaluation methodology.Conclusion and prospective future research challenges are summarized at the end of this paper.
Study on System Architecture of Subdivision Storage Cluster for Global Spatial Data

CHENG Chengqi,GUAN Li

DOI: https://doi.org/10.13209/j.0479-8023.2011.015

2011-01-01

Abstract:In order to solve the unified reference and multi-scale problems and orderly store and manage massive spatial data,a new storage and management architecture based on global subdivision theory is presented by taking deficiencies of organization and storage methods for global spatial data.The system architecture is integrated with the framework of global subdivision grid(GSG),the computer cluster theory,and the space-temporal integration technology.The main characteristics of this storage technology are the storage based on space and each storage node has an accordingly geospatial meaning.The concepts of global subdivision storage cluster,general design ideal and key design of the system architecture are described.The study on cluster storage theory based on the framework of global subdivision will make a solid theoretical foundation for global spatial data storage systems.
A High-Dimensional Indexing Model for Multi-Source Remote Sensing Big Data

Lilu Zhu,Xiaolu Su,Xianqing Tai

DOI: https://doi.org/10.3390/rs13071314

IF: 5

2021-03-30

Remote Sensing

Abstract:With continuous improvement of earth observation technology, source, and volume of remote sensing data are gradually enriched. It is critical to realize unified organization and to form data sharing service capabilities for massive remote sensing data effectively. We design a hierarchical multi-dimensional hybrid indexing model (HMDH), to address the problems in underlying organization and management, and improve query efficiency. Firstly, we establish remote sensing data grid as the smallest unit carrying and processing spatio-temporal information. We implement the construction of the HMDH in two steps, data classification based on fuzzy clustering algorithm, and classification optimization based on recursive neighborhood search algorithm. Then, we construct a hierarchical “cube” structure, filled with continuous space filling curves, to complete the coding of the HMDH. The HMDH reduces the amount of data to 6–17% and improves the accuracy to more than eight times than traditional grid model. Moreover, it can reduce the query time to 25% in some query scenarios than algorithms selected as the baseline in this paper. The HMDH model proposed can be used to solve the efficiency problems of fast and joint retrieval of remote sensing data. It extends the pattens of data sharing service and has a high application value.

environmental sciences,imaging science & photographic technology,remote sensing,geosciences, multidisciplinary
RESEARCH ON HIERARCHICAL STORAGE STRATEGY ON P2P HYBRID CLUSTER

Chaofan Tu,Qifei Zhang,Yingzhuang Chen,Yuchang Zhang,Erteng Liu,Honghua Gan,Weidong Zhang,Fan Bai,Xianrong Zheng

DOI: https://doi.org/10.3969/j.issn.1000-386x.2017.04.002

2017-01-01

Abstract:With the scale of distributed storage systems growing rapidly, the problem of energy consumption has become increasingly prominent, and storage cluster hybrid low-power node has become one of the important ways to solve this problem.To face the problem of how to use different types of storage nodes discriminatively, in this paper, a hierarchical storage strategy called vnode hierarchical remapping (VHR) is designed and implemented based on a P2P distributed system of ZDFS.The strategy didn't affect the high scalability of P2P, so that the system can automatically distinguish between the use of different types of storage nodes with strong operability.Experiments are carried out on a real X86 and ARM hybrid cluster, and experiments show that the VHR works well and plays to the advantages of different types of storage nodes.Under the condition of the performance degradation is not obvious, the overall level of energy consumption was reduced by 44.8%.
A Survey of Index Structure in Big Data Era

YAN Zhaofeng,ZHANG Weihua

DOI: https://doi.org/10.11959/j.issn.2096-0271.2019028

IF: 3.3

2019-01-01

Big Data Research

Abstract:With the advent of the big data era,the volume of data storage has exploded.In order to ensure the performance of system,the concurrent index structure becomes more and more important.However,the increase of storage volume and the increase of user performance requirements have made the index structure face many challenges.Designing an efficient and easy-to-use index structure has become an important issue in the system field.Based on current research status,firstly,the research status of designing more optimized and easy-to-use concurrency control strategies were discussed,and then the researches based on the new hardware structure features and possible directions were discussed.
A Kind of Cloud Storage Model Research Based on Massive Information Processing

Zhang Guigang,Li Chao,Zhang Yong,Xing Chunxiao

2012-01-01

Journal of Computer Research and Development

Abstract:This paper presents a kind of cloud storage model based on massive information processing. The model includes: 1)storage methods of structured, semi-structured and unstructured data and the corresponding resources; 2)THDFS (Tsing Hua Distributed File System) which will take into account all size of the massive data files(big files and small files); 3)a cloud database system design model THCloudDB (Tsing Hua Cloud Database); 4)a internet-base massive data processing computation model THMapReduce (Tsing Hua MapReduce) computation model based on hadoop+ + which is stronger than MapReduce and weaker than SQL, especially it have merged into the semantic computing technology; 5)a intelligent mining tool THBI (Tsing Hua Business Intelligent) based on the THDFS and THCloudDB, and it have the semantic functions. Finally, paperend takes "Tsinghua Tong" as a case and make some analysis about the model's applications.
Towards understanding hierarchical clustering

WuJunjie,XiongHui,ChenJian

IF: 6

2009-01-01

Neurocomputing

Abstract:A very important category of clustering methods is hierarchical clustering. There are considerable research efforts which have been focused on algorithm-level improvements of the hierarchical clust...
Big Data Storage Architecture Design in Cloud Computing

Xuebin Chen,Shi Wang,Yanyan Dong,Xu Wang

DOI: https://doi.org/10.1007/978-981-10-0457-5_2

2016-01-01

Abstract:To solve the lag problem of the traditional storage technology in mass data storage and management, the application platform is designed and built for big data on Hadoop and data warehouse integration platform, which ensured the convenience for the management and usage of data. In order to break through the master node system bottlenecks, a storage system with better performance is designed through introduction of cloud computing technology, which adopts the design of master-slave distribution patterns by the network access according to the recent principle. Thus the burden of single access the master node is reduced. Also file block update strategy and fault recovery mechanism are provided to solve the management bottleneck problem of traditional storage system on the data update and fault recovery and offer feasible technical solutions to storage management for big data.
Research on Distributed Storage Technology of Database Big Data Based on Cloud Computing

Duan Hongwei,Bi Ligetu

DOI: https://doi.org/10.1088/1742-6596/1982/1/012195

2021-07-01

Journal of Physics: Conference Series

Abstract:Abstract With China entering the era of big data, the storage and management of big data has become a hot issue. The proposal and use of distributed storage technology has achieved good results. The continuous development of Internet communication technology and the popularization of computer information technology promote the evolution of traditional data model. The distributed storage system for big data has strong expansibility. It can optimize the existing storage space, realize the optimal allocation of space resources, and reduce the cost of data storage. Cloud computing as a resource sharing business computing model, it receives a huge amount of data every day, and the amount of data is still expanding. This has caused great pressure on distributed storage in cloud computing environment. This paper studies the distributed storage technology of database big data based on cloud computing, in order to strengthen the data management in the era of big data and improve the security and utilization of data.

English Else
Design and Implementation of Information Lifecycle Management System on Hierarchical Storage Management

付先平,孙林春,熊伟伟,刘志宽,方扬,汪东升

2009-01-01

Abstract:Information lifecycle management is a sustainable development storage policy,which balance the storage cost and manage depending on its characters.This thesis presents the design and implementation of the Tsinghua ILM based on hierarchical storage in SAN and NAS environment.A new implementation of file-reparse is presented to resolve the user directly access file system problem and consolidate the data in different hierarchies.It brings little overhead to the system and could be ignored after performance evaluation.The optimization of the management of application-level metadata and policy cache technology is presented at the end of the paper.It is demonstrated from the performance evaluation that policy cache technology improved the performance of migration policy execution by an order of magnitude.Besides,Policy Metadata Container is designed to improve the poor performance in the initialization process of the policy cache.The time spent on initialization decreased by 16 to 20 times due to the performance evaluation.
NSSSD: A New Semantic Hierarchical Storage for Sensor Data

Mehdi Gheisari,Ali Akbar Movassagh,Yongrui Qin,Jianming Yong,Xiaohui Tao,Ji Zhang,Haifeng Shen

DOI: https://doi.org/10.1109/cscwd.2016.7565984

2016-01-01

Abstract:Sensor networks usually generate mass of data, which if not structured for future applications, will require much effort on analytical processing and interpretations. Thus, storing sensor data in an effective and structured format is a key issue in the area of sensor networks. In the meantime, even a little improvement on data storing structure may lead to a significant effect on the lifetime and performance of the sensor network. This paper describes a new method for sensor storage that combines semantic web concepts, a data aggregation method along with aligning sensors in hierarchical form. This solution is able to reduce the amount of data stored at the sink nodes significantly. At the same time, the method structures sensed data in a way that we can respond to semantic web-based queries with less consumption of energy compared to previous conventional methods. Results show that, in some situations especially when the diversity of query responses and life of network are vital, the efficiency of our new solution is much better.

Research on data storage structure and management model in hierarchical storage

A Tree-Based Hierarchy Data Storage Framework in A Pervasive Space

Scientific data organization and management in numerical simulations

A Study on Data Storage and Management for Massive Remote Sensing Data Based on Multi-level Grid Model

Study of Seamless Organization and Storage Structure for Massive Spatio-Temporal Data

Massive Data HBase Storage Method for Electronic Archive Management

Hierarchical Storage for Massive Social Network Data Based on Improved Decision Tree

Efficient Hierarchical Storage Management Framework Empowered by Reinforcement Learning

Design and Implementation of Hierarchical Storage System FQ-HSM Based on File Queue

The Stndy of Massive lnformation Management System 13ased on Multilevel Storage Architecture

Hierarchical Architecture Design of Computer System

Study on System Architecture of Subdivision Storage Cluster for Global Spatial Data

A High-Dimensional Indexing Model for Multi-Source Remote Sensing Big Data

RESEARCH ON HIERARCHICAL STORAGE STRATEGY ON P2P HYBRID CLUSTER

A Survey of Index Structure in Big Data Era

A Kind of Cloud Storage Model Research Based on Massive Information Processing

Towards understanding hierarchical clustering

Big Data Storage Architecture Design in Cloud Computing

Research on Distributed Storage Technology of Database Big Data Based on Cloud Computing

Design and Implementation of Information Lifecycle Management System on Hierarchical Storage Management

NSSSD: A New Semantic Hierarchical Storage for Sensor Data